This study used mixed methods to evaluate the implementation of a complex CDSS for patients with UWL, using patient data collected in clinics through interviews with participating general practices and an electronic health record (EHR) audit.

Future Health Today (FHT; The University of Melbourne) is a software platform that has been developed for use in general practice, streamlining the identification and management of chronic diseases. While the development of the tool has been described elsewhere, it underwent an extensive co-design process with GPs and consumers [15,16]. FHT is integrated with the EHR and has 2 core components: First, it is a CDSS that provides proactive, guideline-based recommendations at the point of care; the second component is a web-based portal, which supports quality improvement (QI) activities in general practice, has the ability to audit and recall patient groups within the practice population, and provides access to education and resources.

The FHT tool supports a wide range of conditions, and different modules are developed for use in FHT [17,18]. This study explores the use of the FHT UWL module, which flags patient records with UWL indicated in the EHR in the past 6 months. Development of the UWL module was extensive, beginning with the development and validation of phenotyping and risk prediction algorithms to identify patients with “unintended weight loss” terms in structured EHR fields (reason for visit, diagnosis, and prescriptions) while also excluding those with indications of intentional weight loss. Additional exclusion criteria, including recent cancer diagnoses, intentional weight loss reasons (eg, bariatric surgery or weight loss prescriptions), and/or increasing weight measurements, were further added [19]. We did not use actual weight loss as an inclusion criterion, given the low prevalence of weight measurements in Australian general practice data [20]. Algorithm criteria (age, sex, blood test results, and smoking status) determined the patient’s risk profile and provided the recommendation delivered to the GP.

The phenotyping and validation process and the risk profile calculations have been described elsewhere [19]. Briefly, after applying inclusion and exclusion criteria to identify patients with UWL, the module provided 3 types of recommendations according to cancer risk:

Figure 1 shows a simplified diagrammatic representation of the rule-based algorithm rules.

The content and wording of the recommendations and the supporting resources were developed with input from GP and consumer and were informed by evidence from a previous scoping review [21]. An example of a recommendation as it appears in the EHR is shown in Figure 2.

Recruitment of general practices occurred by approaching practices participating in the FHT program (ie, practices that already had FHT installed). Practice managers and/or practice owners were contacted and provided with study information. All recruitment documentation was created with assistance from the FHT Consumer and General Practice Advisory Group. In agreeing to participate in the study, practices were required to review a list of patients with UWL, nominate 1 person from the practice to participate in an interview, and partake in a clinical audit.

We invited, via email, practices that had at least five patients identified by the UWL algorithm. Practices recruited to participate in the pilot UWL implementation study had the FHT UWL module switched on for at least 6 months. At study initiation, the FHT team created a list of patients (also referred to as "cohort") who met the criteria for UWL, which could be accessed via the FHT web-based portal. Practices were also offered a 30-minute, Zoom (Zoom Communications, Inc)–based training session once the module was switched on. Each practice nominated an appropriate person (eg, GP, general practice nurse, and QI lead) or team of people to review the cohort of patients with UWL. The purpose of the cohort review was to determine which patients should be recalled for further follow-up or whether the recommendations should be deferred. After review of the patient list, practices were asked to fill in a log with the kind of action performed (eg, patient recalled or deferred) and the reasons for this action (eg, weight loss was intentional, patient is already being followed up, and patient requires further investigation). After the cohort review was complete, practices were invited to use FHT as they chose for the remainder of the study.

Ethical approval was granted by the University of Melbourne Human Research Ethics Committee STEMM 1 (Ethics ID: 25308). Participants were provided with a plain language statement explaining the study. Informed consent was obtained from all participating general practices. Practices were paid Aus $1000 (approximately US $660) for participation. Participation was voluntary, and all interview participants were reimbursed for their time (Aus $100, equivalent to US $67).

We recruited 5 general practices from a pool of 40 practices that had the FHT program installed. Practices were located in Queensland and Victoria, Australia. Four practices were located in metropolitan areas, and 1 practice was located in a rural area based on the Australian Modified Monash Model [27]. The mean number of patients identified as having UWL 6 months previous to cohort creation was 13 (SD 5) patients per practice. Practice characteristics are described in Table 1.

We conducted 5 interviews with 7 practice staff. Interviews were conducted via Zoom after the list of patients had been reviewed by practice staff and at least a month had passed to give patients the opportunity to come back for a follow-up appointment if they had been recalled by the practice.

Participant characteristics are presented in Table 2.

We first describe “affective attitude” to contextualize the staff’s general attitude toward the CDSS. Furthermore, 3 main deductive themes became evident from our analysis using the 2 frameworks: internal organizational policies, procedures, and culture; and workflow and communication and intervention coherence. Remaining relevant themes from the frameworks are summarized, with exemplary quotations provided in Multimedia Appendix 4.

Overall, participants liked the concept of the tool. For quality assurance purposes, the cohort tool, with the ability to create lists of patients with specific conditions and the detailed information it provided on the patient, was found to be very useful. Regarding the UWL cohort, they particularly liked that the cohorts were composed of only a few patients, as expected given the low prevalence (1.5%) of UWL reported in the literature.

Clinical and administrative staff generally favored QI initiatives and supporting software, with 1 staff member reporting frequent use of FHT in management plans and health assessments.

One clinical staff member mentioned finding both features useful:

Despite FHT being installed for at least a year in all 5 practices, 5 (70%) participants were not well acquainted with the cohort tool in FHT. Even after co-design efforts, provision of written materials, and virtual training offers, several challenges emerged in the implementation of the UWL module. The inclusion and exclusion criteria of the algorithm were not well understood. Users were uncertain about how patients progressed through the patient management system (from “awaiting action” to “deferred,” “actioned,” or “no further action needed”), indicating confusion about this component of FHT.

After reviewing the lists, clinicians identified that several patients on the list were intentionally trying to lose weight (see “Clinical Audits” section) or had only one recorded weight measurement and staff were confused by the inclusion of these patients. Technical issues further complicated matters, as the tool malfunctioned during use by the first practice. In this case, the practice staff reviewed the cohort and actioned the patients accordingly (marking them as “deferred”: no action needed or “recalled”), but the system did not save these actions, causing all patients to remain in the “awaiting action” category, causing additional confusion.

Despite these challenges, there were positive aspects. The tool’s visual elements and patient summaries were seen as helpful facilitators. Most users found reviewing patients in the cohort straightforward:

Internal processes arose as the most influential factor in the use of the cohort tool.

GPs typically document most patient information in the free-text clinical notes field of the EHR. However, the CDSS algorithm is currently unable to read these notes, relying only on structured text fields such as “reason for visit” or “diagnosis,” creating a substantial barrier to its effective use. This mismatch between clinician documentation and the CDSS data input requirements limited the tool’s ability to provide comprehensive support.

In 2 practices, the manager asked GPs whether they would be open to a CDSS system that would issue reminder pop-ups at the point of care when a patient presented. They collectively opted not to have it installed as they were concerned about alert fatigue, showing a lack of understanding of how frequently this tool for this algorithm would appear, as UWL is a relatively uncommon presentation.

Quotes exemplifying each one of these factors or processes can be found in Table 3.

Workflow and communication factors were described as crucial determinants in the use of the module. This theme, closely intertwined with internal processes mentioned earlier, stood out for both its prevalence and the depth of insights it provided.

The ease of integrating the tool into existing workflows varied among practices. Those with a special interest in data analysis and QI found it easier to embed the cohort tool into their routines. However, the lack of integration of the system with the practices’ message software was a common issue, hindering direct action on strategies such as reminders and recalls.

The process of reviewing the cohort was generally consistent and considered relatively easy across practices.

Exemplary quotes:

The algorithms identified 62 patients with possible UWL. We extracted data from a total of 60 patients. Two patients were excluded from analysis as records could not be accessed, given (1) no longer an active patient and (2) no weight loss recorded as a reason for the visit or in their past history. The tool correctly identified 36 of 60 (60%) patients with UWL established by clinical assessment of the information in the EHR.

Patients with UWL confirmed in the EHR were older and equally distributed by sex, while patients with intentional weight loss recorded in the EHR were younger and mostly women. Intention to lose weight was recorded as free text in the clinical notes in 98% of cases, with only 20% of patients having intention recorded in a structured field (reason for visit or diagnosis). Sixty-one percent (22/36) of patients with UWL were at moderate to high risk of cancer. Five (8%) of 60 patients were recalled for further investigation. Details of patients’ characteristics and CDSS deployment can be found in Table 4.

Ninety-two percent (55/60) of all patients were deferred, as practices deemed no further follow-up was needed. Reasons for deferral were divided into three main groups: (1) unintentional weight loss with known cause, (2) patient incorrectly included in cohort (eg, intentional weight loss, data error, and increased weight), and (3) unintentional weight loss with cause unknown being followed up already. Table 5 presents the frequency of reasons for referral.

A summary of clinical characteristics and follow-up of the 36 patients with UWL is described in Table 6 (more details in Multimedia Appendix 5). Following their initial UWL consultation, the vast majority of patients with UWL (34/36, 94%) received comprehensive follow-up care. This care encompassed subsequent GP appointments, specialist referrals, various investigations, and/or imaging studies as needed. On average, patients attended a median follow-up of 8 (IQR 10) visits after their initial UWL consultation until the audit date.

One-third of patients (12/36, 33%) presented to the first UWL visit with UWL as their only symptom. Almost half of patients (17/36, 47%) presented with other nonspecific symptoms such as appetite loss and lethargy, and the most common accompanying specific symptoms were gastrointestinal symptoms (13/36, 36%), followed by mental health symptoms (7/36, 19%).

Eighty-six percentage (n=31) of patients had been investigated with blood tests. The most common tests ordered were full blood count tests (n=30, 83%), liver function tests (n=30, 83%), creatinine testing (n=25, 69%), iron studies (n=23, 64%), calcium testing (n=22, 61%), and thyroid function tests (n=18, 50%).

Diagnostic imaging and endoscopic procedures were frequently used in the investigation of UWL. Nearly two-thirds (22/36, 61%) of patients underwent imaging studies or endoscopic examinations. Of these, more than half (13/22, 59%) underwent CT imaging, which included abdominal, lung, and/or pelvic regions.

While direct colonoscopy referrals were relatively low (5 patients), a more common approach was to refer patients to specialists for consideration of endoscopic procedures. One-quarter (9/36, 25%) of patients were referred to either a gastroenterologist or, in rural practices, a general surgeon for consideration of upper and/or lower endoscopy.

More than half of the patients (20/36, 56%) received a diagnosis related to their UWL, as determined by their GP and documented in the EHR. The most prevalent diagnoses associated with UWL were mental health conditions, followed by cancer.

Four (11%) of 36 patients were diagnosed with cancer, including chronic lymphocytic leukemia, myeloma, breast cancer, and renal cancer. However, only the renal cancer case was diagnosed following the UWL consultation. The other 3 (8%) patients should have been excluded from the study based on the predefined algorithm, as these diagnoses were made prior to the UWL visit and within the preceding 5 years.

Our study aimed to pilot the implementation of a CDSS to identify patients with UWL at risk of cancer in primary care settings. This initiative involved extensive preliminary work, including the use of primary care databases to identify patients with UWL and calculate their cancer risk using a UK-developed risk prediction model [19]. We also co-designed recommendations and resources and tested the CDSS in simulated scenarios before implementation. The initial findings suggested that the CDSS would be both acceptable and feasible, prompting us to proceed with the implementation (unpublished data).

The implementation process involved asking clinical staff to review a list of patients flagged by the tool and make decisions to defer or recall patients based on their clinical expertise. Using 2 relevant frameworks [22,26], our analysis of staff interviews revealed that most found the concept of identifying patients with UWL at risk of cancer acceptable, and the review process was easy to understand. The most influential factors affecting specific use were the practices’ internal processes, culture, workflow, and communication practices.

We observed 2 distinct opinions regarding the use and feasibility of the CDSS for implementation. Although FHT had been installed and used by all practices for over a year, staff familiar with QI activities within the FHT program found the tool useful and feasible. They reported that the process was not burdensome and could be integrated into their usual workflow. Conversely, practices unfamiliar with the tool found it less helpful, citing high misclassification of patients with intentional weight loss, lack of integration with their EHR, and inability to perform recalls through their regular workflow. This dichotomy aligns with existing literature, which identifies trust in the tool and seamless integration into workflow as major factors influencing CDSS implementation success [11]. To overcome some of these barriers, a module such as this could be made actionable within the EHR (recalls, tasking, and documentation editable directly from the cohort view) and provide configurable, role-specific triggers to reduce irrelevant pop-ups (which are a known cause of alert fatigue) [29]. Operational measures include role-tailored training, clear follow-up protocols, and decision aids for practice nurses [30]. At the organizational level, provenance and governance processes, local champions, piloted roll-outs across diverse practice types, and explicit evaluation of implementation outcomes (adoption, fidelity, and acceptability) using implementation frameworks (eg, Normalization Process Theory and Proctor’s outcomes) will improve representativeness and scalability [31,32]. Together, these levers—designed and evaluated in partnership with frontline staff—should make integration more seamless, reduce cognitive burden, and increase the likelihood that the tool supports safe, equitable care across a wider range of practices.

Another significant barrier was the lack of allocated time for QI activities, particularly for GPs. This finding is consistent with broader literature on the challenges of implementing QI initiatives in primary care settings [33,34].

Clinical EHR audits showed impressively high follow-up care rates, with 94% (34/36) of patients with confirmed UWL receiving appropriate follow-up and a median of 8 follow-up visits per patient. Our study revealed an exceptionally high follow-up rate, with over 50% (20/36) of referrals made to various specialists. Australian and international literature reports delays in referral for unspecific symptoms compared with specific symptoms [35,36]. However, our findings are consistent with results from our previous randomized controlled trial that used FHT to identify patients with abnormal test results at risk of cancer. In that study, even control practices demonstrated follow-up rates exceeding 70% [17].

The high follow-up rates observed in our study may be reflective of the broader patterns of continuity of care in Australian general practice. The vast majority of Australian patients attend only 1 or 2 general practices annually, and older patients and those with chronic conditions demonstrate particularly high continuity of care [37]. Nevertheless, our findings raise important questions about the representativeness of practices participating in research studies overall. It is possible that these practices are inherently high performing compared to the broader landscape of general practice, potentially skewing our understanding of typical follow-up rates and care quality in primary care settings. Future research needs to explore this issue more thoroughly by comparing the performance metrics of research participating practices with those of a random sample of general practices, investigating the characteristics and motivations of practices that regularly engage in research, and developing strategies to involve a more diverse range of practices in primary care research. Understanding these factors is crucial for accurately assessing the generalizability of research findings and for developing interventions that can be effectively implemented across the entire spectrum of primary care practices. Regardless of the latter, the high rates of investigation and imaging in our cohort (86% had blood tests and 61% had imaging) are consistent and may reflect the accessibility of diagnostic tools in the Australian health care system. This accessibility has been credited for Australia’s comparatively better cancer outcomes relative to similar high-income countries worldwide [35].

The symptom profile of patients with UWL presents an interesting diagnostic challenge. One-third (12/36, 33%) of patients identified by the UWL module had no other recorded symptoms, and nearly half (17/36, 47%) had nonspecific symptoms such as appetite loss and lethargy recorded in any field in the EHR. The high prevalence of no other symptoms or coexisting nonspecific symptoms aligns with international literature and underscores the difficulty in differentiating between benign causes and more serious underlying conditions [8,38,39]. Nicholson et al [39] highlight the complexity of interpreting weight loss in primary care, especially when it cooccurs with other nonspecific symptoms, emphasizing the need for careful clinical assessment and follow-up.

Mental health symptoms were also common in our cohort. Just more than half of the patients received a UWL-related diagnosis, primarily mental health conditions or cancer.

The most prevalent diagnoses in our cohort were mental health conditions, cancer, and chronic obstructive pulmonary disease, consistent with international literature. A study by Withrow et al [6] described the most prevalent serious conditions for people with UWL, stratified by age and sex. Given our cohort’s median age of 68.5 years and equal sex distribution, our findings align with their reported common diagnoses for this age group: cancer, diabetes, and chronic obstructive pulmonary disease in men; and depression, thyroid disorders, and cancer in women.

The algorithm identified 36 (60%) of 60 patients with UWL. Identifying UWL in EHRs is clinically and technically challenging due to inconsistent documentation practices and the predominance of free text in Australian primary care records [40,41]. In this study, “intention” of weight loss was recorded in the clinical notes for 98% of cases. Without access to unstructured data such as the clinical notes, algorithms such as this, or even more sophisticated natural language models and AI applications have very limited potential. Furthermore, if CDSS performance depends wholly on the quality of the recorded symptoms, then if documentation is incomplete, detection will inevitably be affected. Human scribes in primary care have been shown to raise productivity, increase face-to-face time by 39% to 57%, and improve chart quality and physician satisfaction without reducing patient satisfaction [42]. Ambient AI scribes promise to scale this documentation capture, but evaluations show omissions are frequent and occasional high-severity inaccuracies occur, so clinical oversight and careful product selection are essential. Studies in Canada and the United Kingdom reported strong note-generation capabilities across multiple AI scribes but also marked variability in accuracy, usability, and technical performance [43]. Future research should focus on perfecting these technologies. A pragmatic approach would be to layer scribe-captured narratives into CDSS workflows, enabling richer free-text phenotyping (eg, unintentional weight loss with cosymptoms) while preserving clinician control and opportunities for validation, thereby balancing improved capture with safety and trust. The number of false negatives—patients with UWL not identified by the algorithm—also remains unknown, given that we did not review records not flagged by the algorithm due to cost and time constraints. We were therefore only able to identify true positives and false positives identified by the tool.

The algorithm’s failure to exclude 3 patients with cancer (2 cases of lymphoma and myeloma, possibly due to information being solely in clinical notes, and 1 case of breast cancer with a coded diagnosis) highlights an unexpected algorithmic error and underscores the challenges in coding and the need to ensure coding is standardized and reviewed over time. Measured weight loss was not incorporated into the algorithm as an inclusion criterion due to a low percentage of 2 consecutive measurements in preliminary analysis; the impact of this omission on the correct identification of patients remains unclear and warrants further investigation. The study’s small scale, with a limited number of patients, significantly restricts its generalizability. Although relevant and necessary to ensure patient safety and good allocation of resources, larger, more diverse patient cohorts are necessary for more robust and widely applicable findings.

These limitations highlight areas for improvement in future research, emphasizing the need for enhanced data access, refined algorithmic processes, and larger study populations to increase the reliability and applicability of methods for identifying patients with UWL.