The FHT study was a pragmatic cluster-randomized controlled trial that evaluated the effectiveness of a QI intervention [22]. Pragmatic trials, by definition, are trials that evaluate an intervention in everyday practice, with the aim of measuring the effectiveness of the intervention in routine clinical practice rather than under ideal conditions [23,24]. The implementation of the FHT software and the trial components (including implementation strategies) were applied and adapted to real-world conditions to understand and evaluate how the tool would be used in routine general practice.

The components of the complex intervention included the FHT software, training and educational sessions, benchmarking reports, and practice support. The trial was conducted between October 2021 and September 2022. Practices were randomly allocated to participate in either the intervention (follow-up of patients with abnormal blood test results associated with the risk of undiagnosed cancer) or the active control (which had access to a different FHT module). As the aim of this process evaluation was to explore the factors critical to the implementation of the cancer module, our study population comprises the 21 intervention arm practices only; results for the active control intervention will be reported separately. The study protocol has been published on the Australia and New Zealand Clinical Trial Registry (ACTRN12620000993998) [25].

FHT was integrated within the general practice EMR and consisted of a CDS tool, a web-based audit and feedback tool, and the capacity for general practices to monitor their QI activities [26]. Disease-specific modules were developed for use in FHT. The cancer module used patient information in the EMR (age, sex, previous cancer diagnosis) and results of abnormal tests associated with undiagnosed cancers. The FHT cancer module consisted of 3 central algorithms, designed to assist GPs by flagging patients with abnormal blood test results that are associated with an increased risk of undiagnosed cancer: markers of iron deficiency and anemia, raised PSA, and raised platelet count). The CDS component of the tool activates when the GP or general practice nurse (GPN) opens the patients’ medical record, displaying a prompt on screen with guideline-concordant recommendations, such as the review of relevant symptoms or appropriate investigations (Figure 1). There is also a web-based portal, containing an auditing tool; a QI monitoring tool; and access to resources, guidelines, education, and training, which can be accessed on any computer with FHT installed. Algorithms run each night, extracting data from the practice management software database (eg, Best Practice or Medical Director), processing the data locally by applying FHT algorithms (the data does not leave the practice), and categorizing the results. Examples of a CDS prompt and the audit tool are presented in Multimedia Appendix 1. Further details on the development of the tool and the cancer module explored in this study have been described elsewhere [27-29].

In the pragmatic trial, FHT was installed on general practice computers before study initiation. On the first day of the trial, practices were asked to create 3 cohorts of patients using the FHT auditing tool, one for each abnormal blood test (raised PSA, raised platelets, and markers of anemia). The cohorts included all patients identified by the FHT cancer module who had recommendations for guideline-based follow-up (as part of the trial, practices could then review the patient cohorts and determine if further follow-up was necessary). Cohorts were created again at the 6-month mark, using the audit tool, so that benchmarking information could be determined. After generating the cohorts, practices were invited to use FHT as they chose during the trial.

Implementation of the software was supported by a number of additional intervention components. This multifactorial implementation strategy was informed by the Reach, Effectiveness, Adoption, Implementation, and Maintenance framework, with strategies that were relevant and useful to general practice [30]. These components have previously been shown to increase reach; they are low intensity and high impact, with the purpose of limiting implementation workload while promoting continued engagement with the intervention [31,32]. Training on the use of FHT was offered regularly in the lead up to and in the first month of the trial, and then monthly thereafter. Each practice was assigned a study coordinator, who conducted the Zoom-based training sessions on how to use FHT, assisted with any technological queries, and facilitated requests for support throughout the trial. Practices had access to short training videos on YouTube and a range of short- and long-form written training guides. In addition, 6 Project ECHO (Extension for Community Healthcare Outcomes) [33] educational sessions were run on the topics of cancer diagnosis and QI, each consisting of a 10-minute didactic session, a 10-minute case discussion, followed by an open discussion for approximately 20‐30 minutes. The ECHO sessions were delivered via a webinar, and general practice staff were invited to attend. Quarterly benchmarking reports were provided to practices to review their progress in the follow-up of patients who had been flagged by the tool, and to compare their progress to other practices in the trial. All practices were required to nominate a practice champion to lead the implementation of FHT in their practice and to be the primary point of contact with the study coordinator during the trial, managing the installation and technical queries, facilitating ongoing use of the tool, identifying staff for process evaluation interviews and to disseminating trial related information to the practice. The goal of the practice champion in this study was to mirror the pragmatic approach of the intervention (eg, they were asked to filter and disseminate information to the practice using an approach that best reflects their individual practice needs and current processes).

The study was approved by the Faculty of Medicine, Dentistry and Health Sciences Human Ethics Sub-Committee at the University of Melbourne (ID:2056564). While practices consented on behalf of all practice staff to participate in the wider trial, additional written consent was obtained for all interviews. Interview participants were compensated A $100 (US $64.83) for their time. Practice champions also consented separately and were compensated A $200 (US $129.66) for their role as practice champions. All participant data were deidentified and kept anonymous.

Data were collected via qualitative interviews, usability surveys, technical queries, engagement logs, and educational session surveys. For the semistructured interviews, all practice champions were contacted via phone and email to participate in an interview in the first and last months of the trial. The practice champion was most commonly a practice manager (PM) or GPN, but GPs occasionally took on this role during the trial (eg, due to staff changes). The semistructured interviews were conducted over the phone. The interviews were conducted by study researchers (SC, NL, and BH; see the following section on researcher characteristics). The duration of the interviews ranged between 15 and 42 minutes. The interview guides were developed using the Clinical Performance Feedback Intervention Theory framework [34] and were pilot-tested during earlier optimization work on the FHT cancer module [27]. The interviews explored installation, intervention delivery, implementation barriers and facilitators, goals, and usability (see Multimedia Appendix 2 for interview schedule). The interviews explored similar themes at each timepoint, although earlier interviews included questions around goals and intention, and the final interviews explored long-term implementation and sustainability. GPs and GPNs were also recruited for interviews in month 6 of the trial. These interviews have been reported separately [35], as the purpose of the clinical interviews was to explore the acceptability of the clinical recommendations and impact on clinical practice, rather than explore the implementation of the wider intervention.

Usability surveys were sent to practice champions in months 1 and 12 of the trial, with the request to distribute them to the rest of the practice. The survey was delivered via web using REDCap (Research Electronic Data Capture; Vanderbilt University) [36] and included 30 questions (multichoice or free text). This survey was anonymous but captured general demographic information about the user and the general practice in which they work. The survey then explored the use and experience with the intervention (eg, length of time using the tool, what components have been used, and feedback and engagement with the intervention components). The survey also included a System Usability Scale (SUS) a 5-point Likert scale that quantifies the perceived usability of FHT [37]. The usability survey was developed by the study implementation team and is available in full in Multimedia Appendix 3).

Postsession ECHO surveys were sent to all ECHO session participants via REDCap after each educational session and collected both demographic information and feedback on the specific learning outcomes of each webinar. The survey consisted of 23 multiple-choice or free-text questions. An example survey from one of the webinars is included in Multimedia Appendix 4.

Information on the number of installations in each practice, the number of individual users, and recommendation queries (submitted through the technology by the practice) was collected using the FHT technology. Technical reports, including any technical queries by the practice throughout the trial, were recorded by the study coordinator. All engagements between the practice and the study team (study coordinator and technical team), were recorded by the study coordinator and categorized by content (eg, technical queries, training, and administrative items). Implementation diaries were kept by study coordinators to record contextual information (eg, changes in COVID-19 pandemic guidelines, immunization rollout, and general practice initiatives) throughout the trial.

SC is a PhD candidate at the Department of General Practice and Primary Care, University of Melbourne. BH, a senior qualitative research fellow in the department is the implementation lead for the FHT trial. NL is a postdoctoral research fellow who was the study coordinator for the active control arm of the trial. All are female and experienced in qualitative research and conducting semistructured interviews. Some interview participants were known to the interviewer, given their position in delivering the intervention and supporting the implementation in practices throughout the trial.

Recorded interviews were transcribed and imported into NVivo (version 12; Lumivero). Process evaluation data were analyzed independently (SC and BH), prior to trial effectiveness outcomes, so as not to bias the interpretation of the results. Each researcher independently conducted a structured, deductive content analysis of the interview transcripts to extract key themes in the data. The results of the content analysis were collated, and themes were presented to the research team. To promote trustworthiness, analytical codes and emerging concepts and categories were discussed at multiple points in the analysis. Positionality was discussed by the coding team, including how established relationships, biases, and experiences may influence their relationship to the study data, and reflexive notes were kept [38,39]. The interpretation of the key findings and discrepancies in interpretations was discussed with the wider team. The results of the evaluation were then mapped onto the UK Medical Research Council (MRC) framework [40,41].

While several frameworks are available to explore and evaluate the implementation of an intervention, the MRC framework was chosen as it is designed for evaluating complex interventions. It has previously been shown to be useful in evaluating the delivery of new technologies in complex environments and in instances of a multi-faceted implementation approach [42,43]. The framework includes overarching themes of context, implementation, and mechanisms of impact and provides a mechanism for understanding the implementation successes and failures (Figure 2) [40,41]. In the figure, the data sources from the trial are mapped onto the 4 process evaluation components as outlined by the MRC framework (implementation, context, mechanisms of impact, and outcomes). The figure outlines the core components and questions underpinning each theme, and the process data used to answer these questions.

A total of 21 practices participated in the process evaluation. Characteristics of the participating practices are described in Table 1. Characteristics of the interview participants are outlined in Table 2. Participation in other components of the process evaluation (usability survey, ECHO surveys) and additional general practice details are outlined in Multimedia Appendix 5. In summary, 25 interviews were conducted with 19 practice champions in the first and last months of the trial. A total of 12 usability surveys were completed, and 13 post-ECHO session surveys. Usability survey responses included a mix of PMs (n=4), GPNs (n=4), and GPs (n=3), as well as one receptionist (n=1).

Results have been mapped onto the 3 themes of implementation, context, and mechanisms of impact.

The results of the cluster randomized controlled trial did not demonstrate a significant improvement in follow-up in the intervention arm [22]. At 12 months, 76.2% (2820/3709) of patients with abnormal test results in the intervention arm had been followed up compared with 70% in the control arm, with an estimated difference of 2.6% (95% CI −2.8% to 7.9%). No significant differences were identified in the secondary analyses or in the time to follow-up of abnormal tests for patients flagged by the tool. The following results of the process evaluation provide some context for the null outcome of the trial and suggest areas for improvement in the development and implementation of CDS and audit software for cancer diagnosis in general practice.

There were 3 core themes on implementation: intervention delivery, installation, and general practice characteristics, each underpinned by different evaluation data sources. Intervention delivery was supported by data from engagement logs and educational session surveys, installation and general practice characteristics were supported by data from technical reports, and all 3 drew from qualitative interview data.

The intervention consisted of multiple components: the FHT software components (CDS, an auditing tool, and QI monitoring) and the supporting trial components (educational ECHO sessions, zoom-based training sessions, benchmarking reports, and other web-based learning components that practices could opt-in to use). The uptake of the supporting elements of the trial was generally low, except for the initial formal training sessions. GPs, GPNs, and PMs from all intervention practices were invited to the Project ECHO sessions, yet attendance ranged from 2 to 9 people per session, a mix of GPs and GPNs. Three key barriers were assessed as driving the low uptake of these trial components. First, the supporting components of the intervention were promoted via phone calls, newsletters, and regular emails to the practice champion, so it is possible that the knowledge of each session may not have reached the whole practice, dependent on how the practice champion decided to distribute this information to the practice (eg, internal email systems). The second barrier is the time and resource cost associated with each component. For example, attendance at training sessions and ECHO sessions (1 h each), during or after work hours, was not feasible for many clinical staff. The final barrier relates to recognized need and usability, with many practices reporting that they could use the CDS tool and the cancer recommendations adequately, without the need for additional education or training.

The installation of the software was completed in the month prior to study initiation, with practices having access to a “practice” module on diabetes in the 2 weeks prior to study initiation so any technical issues could be addressed. The installation, which was done remotely and without much interruption to the practice, was reported to be a smooth process for most. For those who required additional assistance, the use of a study coordinator and technical support ensured PMs felt well-supported during this process.

Due to the pragmatic approach of the trial, practices determined how many workstations in their practice would have FHT installed at the start of the trial. A total of 14 practices had FHT installed on all clinical computers. Five practices had FHT installed on only one computer at trial initiation, and of these, 4 made the decision to add FHT to additional computers later in the trial. Implementation logs and technical reports indicate that 3 practices were offline for a short period of time (range: 2‐6 wk), although this does not appear to have had a significant impact on the use of the system.

There was a large variation in the number of patients identified for follow-up across practices. Three inner-city practices, which had a younger and transient patient population, reported that the cancer module may not be useful in their clinic, given the low number of patients flagged by FHT. For example, in one practice, only 14 patients were flagged for follow-up during the entire 12-month trial period. While these practices acknowledged that the FHT cancer module was less useful for them, it did not deter them from continuing to use the tool after the trial, where they would have access to additional FHT modules (see Software Usability section).

In exploring context, there were 2 prominent themes: the COVID-19 pandemic and staff turnover. Both themes were underpinned by engagement logs, implementation diaries, and qualitative interviews.

The FHT trial was conducted during the COVID-19 pandemic. In Victoria, restrictions were placed on how and when people could leave their homes, with Melbourne experiencing lockdowns for 262 days during the pandemic. There was a major shift in usual care, and many consultations were conducted via telehealth. During 2020, there was an 8% reduction in cancer-related diagnostic tests nationally, with greater reductions seen in Victoria [44]. The trial continued during a nationwide COVID-19 immunization rollout in primary care, and the burden on general practice was high. There were reports throughout the trial that practices could not devote as much time as they would have liked to FHT or to attend the ECHO sessions due to competing webinars related to COVID-19.

Consequently, staff turnover was a common theme throughout the trial, and the resultant loss of information and increased resource pressure featured heavily in the month 12 interviews. A total of 9 practice champions left their practice during the trial, with 2 practices ending the trial with no replacement. Many interviewees talked about the magnitude of staff turnover during the pandemic and how it was a barrier to use and to keep up momentum in the study.

We found 4 mechanisms associated with the delivery of the intervention: adoption and integration, training and support, software usability, and clinical recommendations. The sources of data varied within each theme. Technical reports, usability surveys, and interviews supported adoption and integration. Training and support were underpinned by engagement logs, education session surveys, and interviews. Software usability was supported by the usability survey, interviews, and engagement logs. The final theme of clinical recommendations was elucidated from technical reports (in particular, recommendation queries), which were further explored in the educational sessions and qualitative interviews.

The majority of practices reported that they did not use the QI and audit and recall components of the tool, only the CDS, which was delivered at the point of care. The CDS was considered easy to use and quick to learn and was therefore easily integrated into the clinical workflow by matching the resources available in a busy general practice. However, the audit, recall, and QI components of the tool encountered a number of barriers. First, in comparison to the CDS tool, where recommendations are actively delivered to the GP, the audit and recall tool requires the user to visit a web page and log on to access this part of the tool. Second, there were additional layers of complexity and multiple steps involved in order to identify, review, and recall patients identified in the audit tool.

The level of engagement between the study coordinator and most participating practices was high, and the support provided by the research and technical team facilitated the continued involvement of practices in the study. No practices in the intervention arm withdrew during the study period.

Practice staff who attended training sessions or used the web-based resources found the training adequate enough to use the tool, and practice champions reported that they would be comfortable training other members of the practice who could not attend. However, in most interviews, especially with the GPs who did not attend the training sessions, it became evident that components of training on how to use FHT did not reach the entire practice. For example, many GPs were unaware of the patient deferral button (which allows GPs to pause recommendations for a patient for a specified period of time) or that there are patient resources available. The post-ECHO session surveys highlighted that the education and case discussion components of the ECHO sessions were useful to GPs and GPNs in managing more complex patient scenarios, but did not influence the way in which the tool was used.

Of the 12 usability survey responses, 11 (92%) would recommend FHT to others. As part of the usability survey, respondents filled in a SUS [37]. The results of the survey align with the separate qualitative results from the clinical interviews in that FHT is reported to be easy to use, simple, and intuitive [35]. The average SUS score from the respondents was 74 (out of 100), consistent with an above-average score (average score=68; score >70 is considered good).

Acceptance and perceived usefulness of the FHT software were indicated by the number of practices agreeing to continue using the FHT software posttrial. A total of 18 of the 21 practices opted to continue using the software after the trial ended (practices were offered a 3 mo extension), and 17 practices opted to continue using the tool into 2023‐24.

The software included a menu option to “report recommendation query” if the GP or GPN thought the recommendation was appearing in error or wanted further information. Five queries about the clinical recommendations in FHT, from 3 practices, were received during the trial. The most frequent recommendation query centered around the clinical recommendations for raised platelets. The risk of undiagnosed cancer increases at a platelet count threshold of 400×10⁹/L, but different laboratories report an upper limit of either 400 or 450 × 10⁹/L; this caused some confusion among GPs if a patient was flagged with a count in the range of 400‐450x 10⁹/L. This issue was addressed in training sessions and regular communications (monthly emails, newsletters), but the perceived error may have impacted some GPs’ willingness to use the tool and their trust in the recommendations. Interestingly, there were no queries about the recommendations for raised PSA (the FHT recommendations were based on current Australian guidelines for PSA follow-up with a lower limit of 3ng/mL in men over 50, which contrasts with some laboratories that report a lower limit of the normal range of 4ng/mL). Established referral pathways and familiarity with the abnormal test as a cancer marker (raised platelet is a relatively new marker of cancer) may have been a contributing factor to this difference in response.

In this study, we describe a comprehensive process evaluation exploring the delivery of a complex intervention as part of a pragmatic, randomized trial, where a module to support cancer diagnosis was implemented in general practice. The process evaluation describes implementation gaps and the mechanisms that drive implementation successes and failures in order to provide context to the outcomes from the trial [22].

The FHT cancer module intervention did not demonstrate a significant improvement in the follow-up of abnormal test results in the patients flagged by the tool. While we hypothesize that the high-performing practices across both arms may have led to a ceiling effect (ie, there was limited room for improvement given the high rates of follow-up in both arms), an absence of any intervention effect may in part be due to implementation barriers, primarily relating to practice characteristics and contextual factors. There was limited ability for some specific practices to engage with the tool when their patient population was not suited to the FHT module that was implemented. Given this variation in the relevance and usefulness between practices, the use of the FHT cancer module may be better targeted to certain practices based on size, location, and patient demographics.

In comparison to interventions with only one component, complex interventions require more time and resources, and are, unsurprisingly, more difficult to implement [31,45]. We found that the uptake of the supportive components of the intervention was low, aside from some initial training on the software. It was also indicated in the interviews that the supporting components were not considered necessary to use the CDS. While the implementation of new software in general practice requires some training and support, the results of this process evaluation indicate that a scaled-back approach to implementation, one which aligns with the time and resources available to general practice, may have been sufficient for the CDS component of the tool [46]. However, given the null outcomes of the trial, the low uptake of the audit tool, and significant contextual factors (COVID-19 pandemic), more work is needed to determine the usefulness of each component, or combination of components, in supporting this type of change in practice.

Implementing new technologies in general practice is a complex and dynamic process, and despite the potential to improve patient outcomes, many tools have low uptake after implementation [47,48]. The trial consisted of a number of implementation strategies that aimed to optimize the uptake of FHT in routine care, and these methods were applied primarily at the professional level (eg, education or training strategies targeting health care professionals and identification of practice champions) [49]. We found that the use of a practice coordinator facilitated the continued involvement and engagement of practices throughout the trial, similar to previously reported successful implementation strategies used in complex evaluations delivered in general practice. One overview of reviews concluded that practice facilitators, who work with practices in areas such as QI, problem-solving, and education, are almost 3 times as likely to adopt evidence-based guidelines, and practice facilitation improved the adoption of guidelines associated with many chronic diseases [32]. But given the large amount of staff turnover, driven by the COVID-19 pandemic, identifying, maintaining, and replacing practice champions was difficult and resulted in a loss of information and a barrier to engagement for some practices.

This process evaluation was extensive, with a multi-modal approach to collecting process data, including interviews, surveys, technical and software data, engagement logs, and implementation diaries. Interviews and usability surveys were carried out at 2 time points during the trial, to address the dynamic nature of implementation barriers and facilitators and how perceptions of the tool can change over time. This substantive evaluation provides context to a complex intervention and the environment in which it was implemented.

There were, however, some limitations. While all practices were invited to take part or contribute to each component of the process evaluation, there were 3 practices who did not participate in an interview at any timepoint or complete any surveys. The opt-in method for the interviews and surveys meant that we may not have sufficiently captured the views of practices who were less engaged with the intervention. These 3 practices did contribute some data to the process evaluation, through software data, technical information, and engagement logs, which were captured from all practices involved in the trial.

The burden of the COVID-19 pandemic in general practice and the resultant impact on staffing was a core theme throughout the process evaluation and provided context when interpreting the trial results. A second limitation was that the pandemic also likely impacted the time, availability, and resources for general practice staff to participate in the interviews and contributed to the low response rate for the usability survey. To mitigate this, we provided numerous opportunities for users to engage in interviews and respond to surveys throughout the trial and promoted such activities through the continued engagement with each practice champion.

Finally, we had originally planned on including some additional software use statistics to complement the qualitative components of the intervention; however, incomplete data prohibited our ability to do so. Software use data would have allowed us to triangulate users’ responses via interviews and surveys with their time using the software, including what parts of the tool they used and when. Future studies would benefit from including software statistics to cross-check the qualitative results.

There are implications for both research and practice. While the FHT cancer module did not increase the proportion of patients followed up according to guidelines, the process evaluation highlighted factors around usability, which facilitated the adoption and integration of the CDS component of the tool. This, coupled with the acceptability findings from separate clinical interviews [35], and the willingness of the majority of practices to continue using the tool after the trial finished, indicates that different modules developed for use in FHT should be explored, as well as CDS tools for cancer diagnosis more broadly. There are also considerations for designing complex interventions that involve the use of a new technology. Given the low uptake of the supporting components of the tool, but indications of use and acceptability of the CDS component of the software, it is unclear whether a multifaceted implementation strategy is useful when implementing new CDS tools, especially if it has been carefully co-designed to meet the needs of users. Future work should be undertaken to determine if a scaled-back approach, which meets the time and resource availability of general practice, could be as effective in supporting the delivery of novel CDS tools.

This process evaluation highlights the implementation and process-related gaps that could be addressed in future studies that aim to implement diagnostic support tools for cancer in general practice. While some of the factors were context-specific (eg, driven by the COVID-19 pandemic), barriers such as time, resources, and practice variations, alongside considerations of design elements, could be built upon to optimize future CDS and QI programs.