Umeå University's logo

Background: Despite the evident trend in health research to emphasise co-production approaches, there is a lack of established, comprehensive and concrete strategies and evaluation methods to effectively guide and assess them. This project aimed to develop, validate, and test a toolkit designed to enhance and evaluate co-productions in mental health research. The toolkit includes practical evaluation tools, such as a structured questionnaire and tailored interview guides, to support the initiation of research projects and assess the involvement, process and impact of co-production efforts.

Methods: This project used a co-production approach with formative research design to develop a comprehensive toolkit for evaluating the process and impact of co-production in mental health research. Conducted between 2022 and 2024, the project involved iterative engagement with diverse stakeholder groups, providing a dynamic testbed for developing, validating, and field-testing the instruments. The paper outlines the four-phase process: (1) toolkit generation, (2) validation, (3) field-testing, and (4) completion, detailing how the co-production approach shaped the toolkit’s design, relevance, usability, and rigor.

Results: The result of this project is a structured, practical, and comprehensive co-production evaluation toolkit designed specifically for mental health research, potentially involving a wide range of partnerships. The toolkit includes a project initiation guide, a process-oriented survey and interview-guide for mid- and post-project evaluations, and an impact-focused post-project group interview guide.

Conclusions: The findings address a critical gap in mental health research by developing a structured, practical, and comprehensive co-production evaluation toolkit. The toolkit offers comprehensive strategies for evaluating involvement and both the processes and impacts of co-production throughout a project's lifecycle.

User-focused monitoring (UFM) is a method for service evaluation led by service users, aimed at enhancing quality and service user involvement. Our study examines the experienced outcomes following the completion of five UFM projects within Swedish mental health service programs. Specifically, we focus on the barriers and facilitators to integrating user-led evaluations into service program development. Through interviews with user monitors, managers, staff, and service users involved in these evaluation projects, we conducted a directed content analysis focused on preconditions, processes, and outcomes. Our findings indicate that UFM projects contribute to specific development measures and improved relationships within service programs but also to broader cultural change. However, a lack of trust among service users in actual change poses a significant obstacle to their involvement. Additionally, the absence of feedback regarding the implementation of change measures by service programs to service user groups risks reinforcing this lack of trust. To enhance the integration of evaluation results into quality development, early engagement with staff groups and clear implementation plans are recommended.

User-focused monitoring (UFM) is a method for fostering user involvement at organizational levels in the evaluation and quality development of mental health service and support programs. To enhance understanding of factors that promote or hinder the implementation of UFM, we have purposely sampled and followed five Swedish UFM projects through a case study approach, paying specific attention to key components for its sustainable implementation. We collected empirical data through interviews with involved actors during a start-up phase and shortly after the UFM projects ended, and examination of UFM reports. The acquired data were subjected to directed content analysis, guided by normalization process theory. The results highlight the importance of coherence in the goals and methodological procedures for promoting implementation of UFM projects, as well as activation of the engagement of both service users and staff in initial stages.

User-focused monitoring (UFM) is an emerging method for user involvement and quality development in the Swedish mental health sector. It involves people with lived experience conducting a systematized evaluation of a mental health service or support programme. The aim of this study is to analyse motives and central concerns when initiating a UFM project. Five UFM projects in Sweden were sampled, 16 semi-structured interviews were performed with user monitors and commissioners and analysed using conventional content analysis. Our results showcase a range of motives for conducting UFM, illustrate the centrality of trust in the relationships of the UFM practice and the varying degrees of user autonomy in UFM projects. We further discuss our results in relation to epistemic trust and organizational learning theories. Accountability processes are discussed as important for securing continuous trust between the stakeholders. Furthermore, trust is required for UFM being applied to engage in continuous learning and review of organizational norms and goals together with service users.

Background: The purpose of this paper is to outline the protocol for the research program "UserInvolve", with the aim of developing sustainable, service user involvement practices in mental health services in Sweden.

Methods: This protocol outlines the knowledge gap and aim of the UserInvolve-program. It further provides an overview of the research infrastructure, with specific focus on the organization and management of the program as well as the design of the six underlying research projects. These six research projects form the core of the UserInvolve-program and will be carried out during a six-year period (2022–2027). The projects are focused on examining articulations of experiential knowledge in user collectives, on four specific user involvement interventions (shared decision-making, peer support, user-focused monitoring, and systemic involvement methods) and on developing theory and method on co-production in mental health research and practice.

Results or conclusion: The knowledge gained through the co-production approach will be disseminated throughout the program years, targeting service users, welfare actors and the research community. Based on these research activities, our impact goals relate to strengthening the legitimacy of and methods for co-production in the mental health research and practice field.

User-focused monitoring (UFM) is a method of user-led mental health service evaluation that focuses on strengthening user involvement and developing the quality of services. Despite an increased emphasis on user involvement and the recovery orientation of services, scientific knowledge remains limited regarding how such goals can be realised. In this study, our aim is to explore UFM with a specific focus on how recovery processes are examined through the method in order to discuss how UFM can be developed in order to support a recovery orientation in mental health service evaluation. We sampled 20 Swedish UFM reports for qualitative analysis, and we found that UFM is a promising method for integrating a personal recovery perspective in service evaluations. By being performed peer-to-peer, the method has the unique ability to gather experiential knowledge regarding the situation of service users. UFM especially contributes to exploring service users' experiences related to social connectedness and user involvement in services. We also discuss how the method can be developed to further support a recovery orientation in UFM. This might be achieved by integrating a process-oriented approach in the evaluations and by including the user informants' own goals and views on what constitutes meaningful support in UFM. Suggestions for future developments concern incorporating personal recovery perspectives in the training of user monitors and creating structures for aggregating the knowledge produced through UFM. 

Previous research has shown that experiential knowledge plays an important role for service user movements and has also discussed ambiguities in the definition of this type of knowledge. This study contributes to clarifying how experiential knowledge is understood within the service user movement. Through semi-structured interviews, it explores how Swedish mental health service user organisations (MHSUOs) and service user entrepreneurs (SUEs) regard the role of experiential knowledge for their endeavours. The study shows that SUEs and MHSUOs view experiential knowledge as complementary to established medical and expert knowledge. Both groups further reflect on the functions of individual and collective dimensions of experiential knowledge. Emotions are important in conveying experiential knowledge, although emotional commitment can be short-lived. The study discusses challenges for SUEs and MHSUOs in developing strategies to manage risks tied to strategic essentialism, and also in maintaining a critical standpoint towards established service providers.

Including the voices and knowledge of service users is essential for developing recovery-oriented and evidence-based mental health services. Recent studies have however, suggested that challenges remain to the legitimization of user knowledge in practice. To further explore such challenges, a co-production study was conducted by a team of researchers and representatives from user organizations in Sweden. The aim of the study was to explore the barriers and facilitators to the legitimacy of user knowledge, as a central factor in sustainably implementing user influence in mental health practice. A series of workshops, with representatives of mental health services and user organizations were conducted by the research team to explore these issues. The analysis built on the theoretical framework of epistemic injustice, and the underlying aspects, testimonial, hermeneutic and participation-based injustice, were utilized as a framework for a deductive analysis. Results suggest that this is a useful model for exploring the complex dynamics related to the legitimacy of user knowledge in mental health systems. The analysis suggests that the legitimacy of user knowledge is related to the representativeness of the knowledge base, the systematic formulation of this knowledge in applicable methods, access to resources and positions within the mental health system and participation in the process of integrating this knowledge-base in mental health contexts. Legitimizing user knowledge in practice additionally challenges mental health systems to support readiness for change in working environments and to address the power and role issues that these changes involve.

User-focused monitoring (UFM) is a method of evaluating mental health services, conducted by people with lived experience of mental ill health. Research on UFM and on user involvement focused on service monitoring and evaluation is lacking. This study addresses this knowledge gap by examining UFM as a strategy for user involvement. More specifically, this study aims to synthesize patterns in UFM reports to characterize the phenomenon, as well as to further discuss negotiation processes and political opportunities in UFM. The empirical material consists of 136 Swedish UFM reports that have been analyzed in two steps: All reports were mapped according to general characteristics and a sample of 20 reports were selected to provide additional information on the method. This study has been conducted in collaboration with actors representing the user movement and municipality-based mental health services. Our analysis shows that long-term contracts between user organizations and service providers are important to create a sustainable implementation of UFM. However, strategies to protect user autonomy must be carefully considered and employed in relation to such collaborations. We further highlight the risks of a restricted focus on consumer satisfaction, and discuss the current development towards including follow-ups in the UFM process as a strategy for counteracting tokenism.