Implementing a mental health app intervention in a university setting: multi-methods evaluation study

Abstract

Purpose

The number of university students seeking mental health support is rapidly increasing. To provide additional psychological support to students accessing well-being services, this study aimed to pilot a mental health Web application (app) called Orpheus®.

Design/methodology/approach

Guided by student consultations, a multi-methods approach was adopted, including an examination of in-app data, chart reviews of routinely collected student information and interviews with mental health practitioners. Usage data were analysed descriptively. Changes in mental health outcomes were examined using means, standard deviations and reliable change indices for anxiety and depression scores. Inductive and deductive thematic analysis was conducted on qualitative data from staff interviews and student feedback.

Findings

A total of 26 students registered an account with 39 completed app visits. On 37 of the 39 (94.9%) occasions, students reported reductions in the intensity of unwanted negative emotions. Statistically significant reductions in the average pooled anxiety and depression scores were observed. Of the 15 students who completed pre- and post-routine outcome measures, between 20% and 60% showed reliable and meaningful symptom improvements. Students reported that the app was helpful and easy to set up and use, with no adverse events. Practitioners highlighted barriers and facilitators related to the technology features, situational contexts and individual differences.

Originality/value

Integrating Orpheus in real-world settings resulted in promising implementation processes, potential for future uptake and positive outcomes. However, ongoing research, staff training and app testing are needed to further improve the implementation processes for digital mental health interventions.

Citation

Liverpool, S., Fletcher, K., Chopra, T.K., Jay, D., Walters, F. and Kaye, L.K. (2024), "Implementing a mental health app intervention in a university setting: multi-methods evaluation study", Mental Health and Digital Technologies, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/MHDT-07-2024-0015

Abbreviations

GAD-7

= Generalised Anxiety Disorder-7;

PHQ-9

= Patient Health Questionnaire;

TIM

= Technology Integration Model; and

TAM

= Technology Acceptance Model.

Introduction

Mental health problems among university students

Recent findings indicate that the mental health and well-being of university students have been on the decline, especially since the COVID-19 pandemic (Office for National Statistics, 2022; Liverpool et al., 2023; Sheldon et al., 2021). This has impacted students’ academic performance and hindered positive student experiences (Duffy et al., 2020). Research has highlighted that dropping out from university is significantly associated with poor mental health (Thorley, 2017). Statistics from the UK have shown that 1 in 5 students have had a diagnosed mental health problem and at least 50% of students have reported that they have experienced psychological distress and, therefore, needed professional support. These students commonly reported symptoms of mental health problems related to anxiety and depression including low mood and feelings of being overwhelmed (Ladejo, 2023; Akram et al., 2023).

Availability of interventions and services

With the increasing awareness of mental health problems and the demand for student support, several government and university policies have called for additional psychological resources (Bennett et al., 2024). Although there is a wealth of evidence for the effectiveness of face-to-face mindfulness-based and cognitive behavioural therapy interventions in university settings (Worsley, Pennington and Corcoran, 2022), the demand for support exceeds the availability of qualified practitioners (Blane et al., 2013). This has resulted in longer waiting times and delayed access to care, leaving students at risk of developing severe mental health problems (Punton, Dodd and McNeill, 2022; Office for Students, 2019). To address this crisis, student support services have started to offer digital or online interventions, which have shown some evidence of promise (Harith et al., 2022).

Therapeutic support that is underpinned by desensitisation and reprocessing theories is also increasing as they can be effective, evidence-based treatments for several mental health issues including post-traumatic stress disorder (PTSD), depression and anxiety (Shapiro, 2001). For example, Eye Movement Desensitization and Reprocessing (EMDR) is recommended as a first-line intervention for symptoms associated with PTSD by the World Health Organisation (2013) and the UK National Institute of Health and Care Excellence (National Institute for Health and Care Excellence, 2018). With the continued advancements in technology, there is an increasing interest in using mobile health applications (apps) to support this process (Marotta-Walters et al., 2018). However, more information about its feasibility, outcomes, uptake and safety is still needed.

Accessibility and acceptability of digital interventions

Although there is a wealth of knowledge on the effectiveness of digital mental health interventions (Ferrari et al., 2022; Harith et al., 2022), less is known about the general acceptability and integration of these interventions during implementation into existing student support services (Pankow et al., 2024). The Technology Acceptance Model (TAM) highlights that key factors such as perceived ease of use and perceived usefulness of digital interventions can predict potential uptake and usage (Marangunic and Granic, 2015). Some pilot studies have revealed that from the students' perspectives, they view digital interventions as useful resources to improve emotional self-awareness and knowledge (Pankow et al., 2024). Other studies also highlighted that students are satisfied with digital interventions like apps owing to their ease of access (Bear et al., 2022). Yet, several studies within this population have resulted in a substantial drop in adherence and high attrition rates from as early as one week (Linardon, 2023). Consequently, end users, experts and policymakers have called for better integration of digital interventions into routine clinical practice to promote adoption (Pankow et al., 2024).

Pilot and feasibility studies are useful for testing important characteristics before implementing new interventions (Teresi et al., 2022). In light of some limitations noted about the TAM, Shaw, Ellis and Ziegler (2018) developed the Technology Integration Model (TIM) to help predict the continued use of technology-enabled interventions. This model builds on the TAM by considering additional features influencing an individual’s behaviour, such as agency, individual differences and intrinsic and extrinsic motivations, allowing researchers to capture the views of end users and product providers.

Aims and research questions

This study aimed to evaluate the process, outcomes and uptake of a mobile mental health Web-based app intervention called Orpheus®, offered to university students through student well-being support services. The following questions were addressed:

Q1.

To what extent is the mental health of students impacted by the implementation of a new app called Orpheus®?

Q2.

What factors influence usage, uptake and integration from both the student and practitioner perspectives?

Methods

Ethical considerations

The study was reviewed and ethically approved by the University’s Health Research Ethics Committee (ETH2223-0003).

Student consultations

During the study conception phase, the research team consulted a sample of second-year undergraduate students (n = 7, 1 male and 6 females) who volunteered to be part of a research advisory and involvement session. This was an important step to ensure our study adhered to international recommendations for improving the quality of research (Bensenor et al., 2022). Five of the seven students had previous contact with student well-being services. Students endorsed the aim of the study and confirmed the design and data collection plans were appropriate. Students indicated that the use of routinely collected data also minimised the potential burden, as they were not being asked to complete additional questionnaires or participate in lengthy interviews.

Design and settings

This study used a single site one-group multi-methods research design (Becker, 2023). The data was collected during the feasibility testing stage of a Web-based application called Orpheus (see description below). We conducted an examination of in-app data, chart reviews of routinely collected student information, including service feedback and interviews with mental health and well-being practitioners. The study site was the student Wellbeing Services at a modern university in the northwest of England.

Intervention: brief description of Orpheus

For this study, the Orpheus app (version 01, Negative Emotion Destroyer Track) was evaluated (Figure 1). Orpheus is an mHealth app designed to transform limiting emotions, patterns and beliefs. The app aims to reduce the intensity of unwanted feelings associated with internalised emotional problems such as stress, anxiety and depression (Morales‐Rivero et al., 2021). The system works at the subconscious level where automatic patterns and emotional responses originate (Reyes-Santos et al., 2022). The Negative Emotion Destroyer track is an audio intervention that lasts up to 12 min. The tracks were developed based on well-known psychological mechanisms such as neuroplasticity and memory reconsolidation that focus on rapidly reducing the emotional load carried by intrusive thoughts or memories (Ecker and Bridges, 2020). The Orpheus app also aligns with techniques from EMDR that involve focusing on a target problem whilst trying to keep hold of the emotion (Shapiro, 2001). Thus, the intervention disrupts the reconsolidation process that the brain uses to interpret thoughts, thereby reducing the emotional significance attributed to the target.

During a face-to-face intervention delivery session, each participating student was asked by a mental health practitioner to think about any past, present or future situation(s) that caused distress (e.g. irrational fear of failing all exams). Next, the student pressed play on the Negative Emotion Destroyer track. While listening to the track, the student was asked to bring up the intense feelings associated with their negative experiences. If something was too painful to think about directly, they were advised to imagine the overwhelming feeling leaving the body as they went through the process. They were then given instructions to perform a hand-tapping task. Depending on which of the three sounds they heard on the track, students tapped along with either their left, right, or both hands. Whilst performing the tasks, the audio track provided a positive narrative aimed at convincing the subconscious mind that they were safe and that the unpleasant feeling was fading away.

Participants and procedures

A total of 26 students were invited to pilot the app between February 2023 and August 2023. All mental health practitioners within the student well-being service received training. Six practitioners then used purposive sampling techniques to identify eligible students to be part of the study. The app was introduced to students during intake or drop-in sessions. Students were selected to use the app if they were assessed as experiencing mild or moderate symptoms of anxiety and depression. These symptoms included any unwanted negative emotions such as low mood and feelings of being overwhelmed. As this was a preliminary evaluation, students with severe symptoms such as suicidal thoughts and those with a previous clinical diagnosis of anxiety or depression were excluded from taking part in this phase of the study. Once students were identified as suitable, they were given information sheets about the app and the study. They were given time to read and ask questions and invited to attend a further appointment to be introduced to the app. As part of routine practice, students were asked to complete mental health and well-being outcome measures (described below) at the start of the appointment. Students were then guided to set up the Orpheus account using their student email addresses and a memorable password. They then listened to one of the tracks targeting unwanted emotions. The process was supported by the practitioner and once students felt comfortable and safe to proceed on their own, the session was terminated. Students were then encouraged to use the app as much as they needed to. At least two weeks after the appointment, students were sent follow-up outcome measures and feedback sheets through email as part of routine practice. At the end of the intervention period, all practitioners were invited to participate in one-to-one interviews.

Data collection

Chart reviews.

For this study, we reviewed routinely collected clinical notes that reported information on students’ demographics, mental health outcomes and session feedback. Service chart reviews are commonly used to explore the characteristics associated with specific health conditions or behaviours (Sarkar and Seshadri, 2014).

Demographics and presenting problems.

Based on the available information, age was calculated from the student’s date of birth. Ethnicity and gender were extracted from the student’s self-identifying options for ethnicity (e.g. white, black, Asian, mixed or other) and gender expressions (e.g. male, female, transgender or other) respectively. Mental health presenting problems were extracted from clinical case notes and further assessed using the completed anxiety and depression outcome measures.

Anxiety.

Data was extracted from two-time points on the self-reported Generalised Anxiety Disorder-7 (GAD-7) scale (Spitzer et al., 2006) – pre-intervention – at the start of the appointment when Orpheus was being introduced and post-intervention – at least two weeks after engaging with the Orpheus app. The GAD-7 consisted of 7-items that measured anxiety over the past two weeks. Items were rated from “0 = not at all” to “3 = nearly every day”. The total score was then calculated ranging from 0 to 21, where 0 represents the absence of anxiety. Cut-off scores of ≤ 9 were considered minimal to mild anxiety, while scores ≥ 10 were considered moderate to severe levels of anxiety (Plummer et al., 2016; National Institute for Health and Care Excellence, 2024). The GAD-7 is recognised as a valid and reliable instrument for screening anxiety in university settings (Byrd-Bredbenner, Eck and Quick, 2020).

Depression.

Data was also extracted from two-time points on the self-reported Patient Health Questionnaire-9 (PHQ-9) (Kroenke, Spitzer and Williams, 2001). The PHQ-9 consisted of 9 items that measured depression over the past two weeks. Items were rated “0 = not at all” to “3 = nearly every day”. Total scores, obtained by summing the responses to each item, ranged from 0 to 27, where 0 represents the absence of depression. Cut-off scores of ≤ 9 were considered minimal to mild depression, and ≥ 10 moderate to severe depression (Manea, Gilbody and McMillan, 2012; National Institute for Health Care and Excellence, 2024). The PHQ-9 is also recognised as a valid and reliable instrument for screening depression in university settings (Keenan et al., 2023; Rahman et al., 2022).

Session feedback/experience of service.

Data was extracted from clinical notes regarding feedback on the session attended or the service in general that students received on the day Orpheus was introduced. In line with service improvement recommendations (NHS Improvement, 2018), practitioners were expected to seek feedback from students by asking “How was your last session/visit with the wellbeing team?”.

Adverse events.

Clinical notes were reviewed to identify any documented adverse events; identified as any unexpected occurrence in the student or practitioner, which did not necessarily have a causal relationship with the intervention. Any adverse events arising during the study period were assessed for severity, causality, seriousness and expectedness (Gliklich, Dreyer and Leavy, 2014).

Engagement analytics

In-app usage data and analytics were used to report engagement with the intervention and symptom change over time. Completed intervention or app usage was measured as registering an Orpheus app account and listening to at least one track at least once. The app also collected self-rated data on the intensity of unwanted emotions before and after listening to the track and engaging with the tapping exercise through a 10-point scale, where 10 is completely overwhelming and 0 is completely calm.

Qualitative interviews

The primary author (SL) conducted semi-structured interviews via Microsoft Teams during August and September 2023 with mental health practitioners. After introductions and consent, a pre-approved topic guide was used to inform the interview (Supplementary material, Table 1). Interviews lasted between 40 and 70 min. The interviews were audio recorded and transcribed verbatim. Owing to the small number (n = 6) of interviews and the sensitivity of the feedback we were unable to use descriptive labels to support the quotes. During the writing of the manuscript, we also paraphrased to maintain anonymity and confidentiality.

Data management and analysis

For quantitative data, descriptive statistics were calculated for student participant characteristics based on the information available on the service charts. Exploratory statistical tests were conducted by comparing the pooled mean differences at the two-time points for the GAD-7 and PHQ-9 outcome measures using repeated measures t-tests. A two-tailed p-value of <0.05 was considered statistically significant. A reduction in GAD-7 and PHQ-9 from pre-intervention to post-intervention was considered reliable according to Jacobson and Truax’s reliable change index (Jacobson and Truax, 1991). The pre-post total GAD-7 score needed to change by 3.5 points or more and the PHQ-9 by 5.2 or more to be considered a reliable change outcome (Gyani et al., 2013). A clinically significant change occurred when the pre-post outcome meaningfully shifted in severity classification (Jacobson and Truax, 1991). Frequencies (n, %) were used to explore in-app usage data. All analyses were conducted using the Statistical Package for Social Sciences version 29.0 (IBM, 2022).

For qualitative data, a codebook informed by constructs from the TIM was developed to perform framework analysis on the practitioner interview transcripts (Gale et al., 2013). Thematic analysis was applied to student feedback documented in the service charts (Saunders et al., 2023). Two researchers independently coded the transcripts. Coding of transcripts and the organisation of categories was managed using Microsoft Excel version 24 (Mircosoft Corporation, 2024) and NVivo version 12 (QSR International, 2024).

Reflexivity and rigour

The research team consisted of a diverse group of researchers with expertise in mixed-methods research and evaluations, as well as extensive knowledge and practice in mental health and digital technologies. To enhance transparency and consistency, agreements prior to the qualitative data analysis were achieved through debriefing and weekly meetings.

Results

In-app data analysis

A total of 26 students registered an account on the Orpheus app with 39 completed visits where pre- and post-in-app scores were registered. Of the 39 visits, 37 (94.9%) showed a reduction in the intensity of the unwanted emotion (i.e. improvement in overall emotional well-being) from an average pre-score across the group of 7.5 (out of 10) to 3.7 after the intervention. The remaining two (5.1%) showed no change in intensity.

Notably, in addition to the 39 unique visits, there were 21 occasions where the pre-score was registered with no post-score indicating an incomplete session. Of these, one occasion was because the intervention had worked adequately enough that the user felt no need to continue the process for the full 12 min. Ten of these occasions were because of the user being distracted by something external to the process. None of the students indicated technical difficulties as a reason for disengagement. Four declined to give a reason and six chose “other” meaning none of the first four options.

Chart review

Routinely collected information was available for 15 students who completed the relevant outcome measures (Supplementary material, Table 2). The students’ ages ranged from 19 to 45 years (M = 26.7, SD = 8.05), with just over half in the age group of 19 to 25 years (53.33%). Most of the students identified as female (73.33%) and all students identified as white/Caucasian. Among the students, practitioners reported presenting problems including, anxiety, low mood, low self-esteem and low confidence.

Among the students, there was a significant reduction in depression scores from pre (M = 10.4, SD = 6.12) to post-intervention [M = 7.13, SD = 4.76), t(14)=2.83, p = 0.013]. For anxiety, a similar pattern was observed. Namely, scores at post-test (M = 8.2, SD = 4.57) were significantly lower than at pre-test [M = 12.8, SD = 5.83), t(14) = 3.66, p = 0.003]. In respect of individual case analysis, 20% (n = 3) and 60% (n = 9) of students experienced reliably reduced depression and anxiety scores respectively from pre- to post-test. Consequently, 53.33% (n = 8) and 46.67% (n = 7) of students showed meaningful improvements post-intervention based on PHQ-9 and GAD-7 scores, respectively (Supplementary material, Table 3).

Student feedback

Based on the routine session feedback that students provided, we extracted sentiments suggesting the “ease of set up” “ease of use” and the general “helpfulness” of the Orpheus app. There were no adverse events recorded.

Students described the signing up and initial setup of the app on their phones as being “straightforward and easy to follow”. Most expressed that the process from setting up the app to using it was “surprisingly quick”. Only one student reported that not having access to phone data or reliable Wi-Fi was a barrier, but expressed that when they eventually connected, the app was “very beneficial”. The benefits also extended to the fact that students with dyslexia found that as the app required limited reading, they found the app easier to use, with one student suggesting this approach “helped a lot”.

Although two students reported that having to listen to the track while tapping was “somewhat frustrating at first”, most students found the app helpful and mentioned they would either use it again or recommend it to others. Some students also reported that they “struggled to hold an emotion or memory in mind while having to listen to the track and do the tapping as well”. However, students found the “voice on the track to be soothing” which helped them stay connected.

Practitioner interviews

Six mental health and well-being practitioners agreed to be interviewed as part of this evaluation. Roles and responsibilities varied from counsellors to well-being practitioners, and years of experience ranged from 9 months to 17 years. Analysis of the interview data highlighted facilitators and barriers related to five domains of the TIM that influenced the implementation process – technology features, situational context, individual differences and intrinsic/extrinsic motivations.

Technology features

Some practitioners mentioned that since Orpheus was a Web app, and therefore not available on the app store, this appeared confusing, as they were mostly accustomed to clicking and downloading apps from the Apple or Play Store. Hence, practitioners expressed that having to email the login details to the students and having them wait for emails to verify their accounts could have caused some delays when they were trying to support students to download and set up the app. However, they found it easy to introduce Orpheus to the students once they “got the hang of it”:

Yeah, it just wasn't clear at the beginning, with it not being on the app store and then needing a passcode. But once we got through that initial little problem it was kind of like, ah, OK, and it was quite straightforward after that. (Practitioner 6).

After being able to get the students connected, some practitioners also commented on the app’s content suggesting that they felt “a lot was happening”:

It was too busy trying to tell students to focus on the taps and then do what I needed to do around the emotional thoughts. (Practitioner 4).

Nonetheless, almost all the practitioners expressed that after familiarising themselves with the app and practising a few times it got easier:

It was quite easy to implement because it was mainly done on the portal, and it was kind of self-explanatory. Once you've shadowed one session [from a colleague] and sort of tried it yourself, practice and things like that as well, it was then easy to move through the app with students. (Practitioner 5).

Situational context

Practitioners expressed that the way the service was set up and how Orpheus was being adopted provided a safe and comfortable environment for them to use the app with students. Almost all practitioners reported that training, resources, shadowing and having an appointed contact person for questions were instrumental in them continuing to use the app. Practitioners also highlighted that having a specific time and date to introduce the app helped them to allocate time to address any issues:

It was almost like here’s a time and date, come and attend this appointment so we can set it [the app] up together. (Participant 4).

However, practitioners reported that having more time during the session and further staff training was also important:

During the session, there was only about 20 minutes to half an hour, and you have to cover a lot in terms of any diagnosis of any risk, academic support, action plan and getting all that sent out. So, there's not a lot of time in the appointment itself. Some more guidance and examples on how others did it in the past would be helpful (Participant 4).

Another training session just around the practical stuff will be helpful. Yeah. It might just be me, but it didn't sink into my brain, because it was a while after we got the training we then moved to the student phase. (Participant 6).

Individual differences

Practitioners also reported that their own personalities and behaviours influenced how and why they chose to incorporate Orpheus as part of their practice:

When the service manager talked about starting the pilot of an app, as I'm not a huge fan of apps, I was a bit hesitant. That's probably my age group, you know. I know younger people are more used to apps and doing things on apps. I wasn't enthused, shall we say. (Participant 3).

I guess part of me is a little bit lazy. So being able to click on something and then easily use it was important. Because everything is just one click, isn't it? And if it's more than that, we give up. I give up. Also, I have absolutely no rhythm. (Participant 1).

This was also influenced by practitioners' perceptions of the student population:

And I must admit, I did feel a little bit of the discomfort from the student with me in the room. (Participant 4).

I think it could be quite useful because the student population generally are a lot on their phones, and they do use quite a lot of these sort of wellbeing apps. (Participant 5).

They [students]seem really open to the idea of trying new things. (Participant 6).

However, some practitioners noted that their mode of practice or training also influenced their decisions:

I'm a person-centred counsellor, so I'm not directive in a way and it's almost strange to bring an app and say right, use this and try this. It should be more student-led (Participant 4).

Intrinsic and extrinsic motivations

Some practitioners found the idea of technology-enabled mental health support very interesting, and they were intrigued to see how it would work in practice. Almost all practitioners also expressed that the need for more resources and additional support tools to meet the mental health demands was a key motivator for using Orpheus. All practitioners also indicated that they were motivated to keep using Orpheus when they saw improvements or received positive feedback on the impact it had on students:

She [a student] had a tool and something that she could use when she was struggling. Before Orpheus, she had nothing that she could take away and use in her own time. It gave me that little bit of hope that she wasn't on her own. (Practitioner 4).

I could see the benefit of what it was trying to do. (Practitioner 1).

[…] probably more out of internal curiosity and interest and willingness to try (Practitioner 5).

They [students] said that they found it helpful. Even from my observations, I thought it had a really, really big impact on quite a lot of students. (Participant 2).

Nonetheless, some practitioners described being less motivated to use the app owing to the limited evidence base and the fact that students forgot to use the app after they had “put in the effort” to get them set up:

I think the only thing was that the research base behind it was needed, because some of the students are doing more research science degrees, so they are questioning more about the things we offer. (Practitioner 5).

But what we found at the follow-up a few weeks later was that there was a huge drop-off. So for example they [students] would say ohh yeah yeah, I've been feeling a bit anxious but I forgot it [the app] was there. (Practitioner 3).

Discussion

Overview of findings and contributions to knowledge

Our findings add to the growing evidence base suggesting the potential benefits of appropriately implementing and integrating digital mental health support into practice (Harith et al., 2022; Ferrari et al., 2022). The findings also contribute significantly to the field of implementation science (Glasgow, Vogt and Boles, 1999; Feldstein and Glasgow, 2008; Glasgow, et al., 2019), providing specific insights because of it relates to digital mental health innovations. Using the TIM, we described processes that can inform the analysis, design, development, implementation and evaluation of similar interventions and settings. Furthermore, our study shared how we involved the target audience in the research process, validated the contents of the app with mental health practitioners and carried out usability testing and evaluations in an already high-pressured environment. This provided rich insights that can further inform formative evaluations of digital-based and technology-enhanced mental health interventions.

The promise of the Orpheus app was particularly highlighted in the student feedback relating to helpfulness and ease of use. These are important variables that contribute to adoption and future use as suggested by the TAM (Marangunic and Granic, 2015). Safety of the implementation process and reported improvements in mental health were also promising, as this contributes to important insights to meet the standards for regulating bodies (Murphy et al., 2020; National Institute for Health and Care Excellence, 2018). However, as this pilot study focused on low-risk cases, more research is needed with at-risk populations and students experiencing severe mental health symptoms if the app is to be scaled up. This is especially important when evaluating Orpheus, as some students reliably and/or meaningfully improved in symptoms related to anxiety but not depression and vice versa. Further examinations of Orpheus could explore the underlying mechanisms that impact change to explain this phenomenon.

Regarding findings on integration and future uptake of the app from a practitioner’s perspective, key information related to technology features such as the location of the app and the downloading process may suggest future upgrades and user testing. Progressive Web apps are cheaper than native apps in the app stores (Tandel and Jamadar, 2018) and, therefore, can contribute to cost-effectiveness in later trials (Gentili et al., 2022). This will be an important consideration because of mental health budgets for university settings are still limited (Office for Students, 2019). Allocated time and staff training were also highlighted as key strategies that promoted staff engagement during the implementation process. This contributes to the ongoing debate that practitioners may require additional time to learn and practice different ways of incorporating technology into practice (Borges do Nascimento et al., 2023). Interests in technology and other individual factors such as motivation also encouraged integration, further highlighting the need for a deeper understanding of the benefits of digital interventions (Taher et al., 2023; Borges do Nascimento et al., 2023). The TIM also helped explore the important role of value-based judgments, demonstrating the need to evaluate this in the early stages of implementation. Our study confirmed the significance of intrinsic and extrinsic motivations related to observed usefulness, helpfulness and uptake of the app by students, which encouraged further adoption among practitioners.

Implications for student well-being services and future research

Based on the findings of this study, the Orpheus app can be implemented as one way to offer additional resources to students with mild-moderate symptoms of anxiety and depression to help meet the growing demand for support. There are alternative apps that have also been explored for use in university settings (Harith et al., 2022; Ferrari et al., 2022), but they differ in their approach and underpinning theories. Therefore, based on the current findings and ongoing evaluations, the Orpheus app can be considered if adherence to other digital interventions is low. This is due to the easy setup process, brief intervention (i.e. 12 min) and possible single-session nature of the Orpheus app. As some practitioners found it challenging to explain the purpose and underlying principles of Orpheus at first, more training and app testing are needed before continuing to introduce the app to students, so practitioners can feel more confident. In addition to the ongoing research, future more rigorous studies are also needed to establish the effectiveness of the intervention before scaling up to a wider target group. These are important steps to help speed up the process of innovation in mental health when moving from conceptualisation to full-scale implementation. The process and findings of this evaluation further highlight the need for interdisciplinary collaborations underpinned by co-production principles to ensure mental health apps are appropriately developed and implemented to meet the needs of all end users.

In addition to this study’s contributions to the literature on digital mental health innovations, the findings contribute to the literature on student well-being in university settings. Both practitioners and students welcomed additional resources to help mitigate the rising mental health crisis among university students. The findings also highlighted that practitioners may have shared challenging experiences during the implementation phases of new interventions. Similarly, students may have had shared experiences of mental health problems and receiving mental health support, so this study can make them feel less alone. Owing to the increased Wi-Fi connectivity and growing smartphone usage on university campuses, the findings from this study further promote the opportunities for more mental health apps, especially in instances where students may not have the capacity to attend face-to-face services.

Limitations

Due to the single-site pilot design, our findings may not be generalised to other digital interventions, populations, sites or settings. Another limitation could also be the sample sizes. The sample also included mainly white students and staff, but this is not reflective of the ethnic diversity at the studied university or similar institutions in the UK (Higher Education Statistics Agency, 2023). Although the attrition and adherence rate appeared good for this evaluation, we also acknowledge high attrition is common in digital health research. Of the 26 students who volunteered to use the Orpheus app, only 15 completed routine outcome measures pre- and post-intervention. Despite the significance of the study, further caution should still be considered when interpreting the findings, as the themes highlighted are yet to be fully investigated in a wider context. Owing to the rapid advancements in technology, continuous upgrades will also be needed to enhance sustainability.

Conclusion

This study describes a crucial first phase in the development and evaluation of complex interventions and outlines key findings that can contribute to future implementation. The findings are promising, highlighting important considerations for ongoing research, staff training and app testing to further improve the implementation and integration process.

Figures

Figure 1.

The Orpheus app

Supplementary material

The supplementary material for this article can be found online.