Computerized tools that aid patient-provider communication and share medical knowledge are proliferating. Many such tools have also been demonstrated in randomized trials to improve clinical care [1]. These include tools that can support patient self-management (SM) [2], patient decision aids [3], point-of-care clinical decision support [4,5], and Web-based tools that can connect health care teams and patients outside of traditional face-to-face clinic visits, such as tools that automate collection of important patient-reported outcomes (PROs) and feed this information to the clinical care team [6]. These knowledge transfer and communication tools can be broadly categorized as PROs and SM tools. There is high enthusiasm that such tools can help make clinical care more safe, effective, and patient-centered [7].

Despite increasing optimism about the potential for PRO and SM tools to improve clinical care, there are many barriers to their successful implementation [8,9]. These tools can be complex, with multiple components that engage not only patients but also multiple members of the clinical care team [10]. Determining how they best fit into a local health system context is often unclear [11]. Furthermore, the extent to which these tools have been tested varies. Relatively, few have been found effective in clinical practice outside of initial efficacy trials, whose purpose is to consider performance in ideal situations [8,12,13]. At the same time, it is not practical for hospital and health system personnel to spend years formally evaluating these and other systems before implementing them.

Health systems and larger clinical communities interested in taking advantage of promising PRO and SM tools need a rapid but still systematic and trustworthy process for identifying, prioritizing, and adapting tools for local implementation [9,14]. Methods of rapid analysis have been developed to aid pragmatic application of research, such as ethnographic style analysis [15,16] and assessment of health technology literature [17,18]. To our knowledge, however, no methods exist to address our question “How can health systems rapidly identify and evaluate technology-based tools that claim to improve clinical care to prioritize them for local use?”

One area where PRO and SM tools have growing policy impetus is oncology care. For example, the Oncology Care Model (OCM) is a pay-for-performance model that emphasizes PRO measures and is being implemented by 192 practices and 14 payers nationwide, including our own academic cancer center [19].

Thus, OCM provided an ideal test case for developing and evaluating a rapid evidence review process to review PRO and SM tools, with a goal of enabling experts to (1) rapidly evaluate evidence for complex computerized tools and (2) prioritize which tools are put into practice. We called this novel process the rapid evidence review panel (RERP) for PRO and SM tools. This methodology study describes our novel evidence review process and how it worked in the context of prioritizing, for local use, complex computerized tools to improve the patient experience of cancer care. We intend for this to be an efficient process that can rapidly unfold at the organizational level.

The RERP process is just 1 important step in a multistep user-centered framework for developing and implementing new tools. The RERP process is not meant to encompass aspects of tool development or evaluation activities around actual tool implementation. Rather, the focus of RERP process is on rapid and pragmatic evidence review that can be conducted within busy health systems. The goal of the RERP process is to help health system leaders prioritize, using current best evidence, which existing tools may be the most feasible and important to implement. The RERP process is important because, if health systems are able to adopt such a process in a systematic and ongoing fashion, it opens a potential path for important new technologies to be adopted more quickly. The RERP process fits within the larger Design and develOpment, Testing early iterations, Testing for effectiveness, Integration and implementation (DoTTI) development and evaluation framework, as portrayed in Figure 1 [20]. The DoTTI framework offers a complete development model for digital tools for patient use. The involvement of patients as stakeholders in the development of PRO and SM tools is essential to ensure that tools meet patient needs and expectations.

We developed our process to take advantage of existing measures, rapid evidence review methods, consensus-based decision-making methods, and rapid qualitative analysis methods [17,18,21]. The process we developed attempts to streamline the information provided to an expert panel and enable the panel to meet just twice to evaluate and prioritize multiple interventions, once in a 1-hour introductory teleconference and again in a face-to-face 5-hour meeting. This time frame may be adjusted according to the quantity of manuscripts needing to be reviewed.

Given this limited amount of time, it is not practical or efficient for panelists to review full manuscripts, fully review the literature, or individually evaluate evidence. Instead, the RERP process makes use of established evidence review tools and frameworks to ensure a rapid process that is also credible. By shifting the labor to a smaller project team that can collect and synthesize relevant information in advance of expert panel review, the expert panel’s evidence review can be accelerated. Our project team consisted of an oncology subject matter expert, an evidence-based medicine expert, a project manager, and a research specialist. Our project team required approximately 3 months to assemble the evidence presented in the RERP meeting.

We then used a rapid template-based coding method using the tailored implementation for chronic disease (TICD) framework and developed a categorization scheme for interventions to rapidly interpret the expert evaluations from the RERP [22]. Our aim was to use these findings to inform local effectiveness, implementation, or hybrid studies [23]. We describe each step of the RERP process in detail (Figure 2) below.

A number of procedures exist to conduct rapid reviews of scientific literature [17,18,21]. We chose to conduct a thorough environmental scan (Figure 3) [17]. We first sought to identify the relevant topic domains. In the context of OCM’s incentives to improve the patient experience of cancer care, we focused on PRO and SM tools related to improving cancer and cancer treatment–related symptoms. We identified symptom domains by reviewing all published care guidelines from major professional organizations writing guidelines for any aspect of the cancer care continuum (prevention, screening, diagnosis, treatment, and prognosis). Our review included the following organizations: American Society of Clinical Oncology, National Comprehensive Cancer Network, European Organization for Research and Treatment of Cancer, National Cancer Institute, US Preventive Services Task Force, American Academy of Hospice and Palliative Medicine, American Cancer Society, and the Oncology Nursing Society. This review established the set of possible symptom domains for further study (see Multimedia Appendix 1). From this larger set of symptoms, we prioritized those that applied to multiple different cancers treated in a cancer center to be more relevant to a broader group of patients (eg, we included chemotherapy-induced nausea and vomiting, but excluded highly disease-specific symptoms such as lymphedema in breast cancer patients). Such highly disease-specific symptoms are certainly important for consideration but were not the focus of our review.

An informationist then performed a systematic search for PRO and SM tools that targeted one or more of the selected symptom domains and evaluated in randomized controlled trials (RCTs; search strategy described in detail in Multimedia Appendix 2). We chose to focus our search on randomized controlled efficacy trials because an initial search identified few to no implementation or effectiveness studies of PRO and SM tools in these domains. The search strategy we developed to identify PRO and SM tools will be of particular interest to those interested in implementing PRO and SM tools and is described in detail in Multimedia Appendix 2. Moreover, the standardization and internal validity of RCTs aid a rapid and rigorous expert panel evaluation. However, we recognize the need to sometimes move beyond the RCT, particularly in the context of complex interventions such as decision support, where local context and clinical workflows are likely to be key factors in determining the success of the intervention [24-26]. In the absence of large pragmatic trials and implementation studies, single-center and multicenter efficacy studies are likely the best starting points for identifying promising tools.

A content expert on the project team (DK) then reviewed the abstracts of all RCTs retrieved by the search strategy. Our criteria for selecting a local content expert are as follows: (1) clinical expertise in oncology, (2) interest in technology-enabled interventions to improve the patient experience of cancer care, and (3) time and ability to work closely with the evidence-based medicine expert and conduct a thorough review. Although the content expert led the abstract review process, the process also included weekly meetings with an evidence-based medicine expert (TC) to review the abstracts and rationale for inclusion or exclusion. The content expert excluded interventions that were not technology and knowledge based or were not targeting one of the selected symptom domains. Afterwards for further review, he selected those interventions reporting at least some evidence of efficacy in the abstract. Full manuscripts were retrieved for these trials and were read in full by the oncologist. Some manuscripts were excluded at this stage because of the limited clinical relevance of the findings. The oncologist then assigned effect size and reach scores to each RCT based on a process developed by the National Cancer Institute (ie, using the Research-Tested Intervention Programs review process) [27]. Those RCTs with combined scores (effect size+reach) of greater than or equal to 4 were presented to the RERP as the final product of this environmental scan. In total, 14 RCTs fit the above criteria for presentation to the RERP. Finally, a member of the team with experience in evidence-based evaluation (TJC) applied quality of evidence scoring to each RCT following the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) working group approach [28]. GRADE outcomes’ tables were created for the primary outcomes of each RCT (see tables in Multimedia Appendix 3).

We used a modified Delphi panel process. Modified Delphi panels are widely used in health research as a method to elicit group judgment that includes multiple rounds of rating, panelist discussion of judgment, and group facilitation to mitigate bias [29]. The modified Delphi was chosen as the best method of evaluation because it is seen as credible, widely used, and can quickly elicit expert consensus. Using the modified Delphi strategy, we conducted the RERP meeting in 3 parts: an introduction, initial rating, and rerating. We allowed time for discussion and questions in each part.

There are 3 tasks for data analysis: (4a) prioritize interventions for implementation, (4b) identify features of the interventions that contribute to positive or negative perceptions of feasibility or impact, and (4c) identify perceived barriers to and facilitators for putting the intervention into practice.

The purpose of the RERP process is to identify high-priority PRO and SM tools for further study and/or implementation. Although systematically identifying and prioritizing the most promising tools is an important first step, successfully implementing these complex tools will still require adaptations based on detailed knowledge of local workflows and context. After identifying SM and symptom tracker tools as effective, impactful, and feasible through the RERP process, we hosted a validation panel with a diverse set of stakeholders within our health system. These included hospital and clinical leadership, hematologists, oncologists, nurses, nurse educators, physician assistants, patient navigators, and other professionals from the University of Michigan Rogel Cancer Center. Using evidence summaries provided to them, these panelists were asked to validate, for locally focused purposes, the knowledge generated previously by a national panel of experts about the feasibility and impact of software tools intended to help improve the patient experience of cancer care. In addition, after having reviewed and commented on the scientific evidence about these tools and its meaning to a national group of oncologist experts, these panelists were asked to review and comment on an early design concept for a user customizable decision support app with features of SM and symptom tracker tools.

We were able to rapidly review 14 manuscripts about computerized tools related to the OCM using our method. Panelists’ feedback indicated that participation was valuable and intuitive. What follows are the results we gained by doing so.

Our environmental scan and evidence review process yielded 14 RCT-tested interventions associated with at least moderate impact and reach. Evidence quality was variable, with most trial outcomes graded as being based on low to moderate quality evidence. Multimedia Appendix 3 provides the evidence review of the 14 RCTs from our environmental scan.

Participants rated interventions on impact and feasibility (Table 2). Agreement increased from the first to the second rating. Overall, most interventions were ranked more highly after discussion.

We found that these complex interventions contained multiple features and that the panel had opinions about many of these features. We identified clear differences in how interventions were ranked based on the type of intervention being studied. We identified 3 main types of interventions among the 14 RCTs reviewed related to improving the patient experience of cancer care:

Using the TICD framework, 6 major constructs were identified as barriers to implementation: quality of evidence, cultural appropriateness, patient behavior, availability of necessary resources, information systems, and payer or funder policies. No themes were identified as major facilitators. SM support interventions were seen as having fewer barriers to implementation, including being more appropriate for the workflow, more in line with patient behavior, carrying less legal risk, and having better evidence for their success.

Taken together, the results of this mixed-methods analysis allowed us to not only understand which specific interventions were considered most high priority but also to learn that SM support interventions may be perceived as more impactful and feasible to implement.

The slow progress from research to practice is well documented [33,34]. The approach we describe here, a rapid evidence review for PRO and SM tools, is intended to balance the goals of rigor and efficiency for an evidence-based method to prioritize promising communication and decision-support technologies. Clinical experts found the evidence review structure to be engaging and the content sufficient to make judgments, and they were able to quickly and effectively prioritize a heterogeneous set of PRO and SM tools.

We observed that the RERP panel strongly favored implementation of SM and communication tools over PRO tools and indicated that this was largely because of less need for clinician involvement and lower legal risk. In addition, panelists expressed much skepticism about the feasibility of implementing PRO tools, despite high evidence of their success in the RCTs. We observed increased levels of agreement in the second round of rating, which is an expected feature of the modified Delphi process, after panelists come together and discuss the rationales for their initial ratings [31].

Health systems cannot put all effective tools into practice, no matter how promising. This prioritization process can be used by health systems and practices seeking to employ PRO and SM tools as the basis for local implementation studies or larger pragmatic effectiveness studies. Furthermore, the results of our evaluation highlight how our medical oncology experts favored SM support tools over tools utilizing PROs. This is surprising given what seems to be growing evidence of the effectiveness of these interventions to improve quality of life [35] and perhaps even lifespan [36]. Technological, workflow, cultural, and legal barriers caused our panel to evaluate these technologies as less feasible and impactful. Further evaluation of PRO and SM tools will help elucidate the extent to which these views about the challenges of implementing PRO-based tools are shared across institutions. Local evaluation can help clarify expectations and planning for implementation at individual institutions. Finally, the use of a systematic evidence review method such as that described here can help ensure that decision making for the implementation of new tools considers both the experience of relevant clinical experts and empirical findings from a diverse body of research literature. In addition, barriers exist that are because of the nature of PRO and SM tools and the research reporting process. Inadequate reporting of technology interventions makes evidence review difficult [37]. Furthermore, rapid technological change can outpace conducting and publishing RCTs, which further outpaces evidence review [38]. However, this further emphasizes the need for a rapid process to facilitate evidence being translated into practice as soon as the evidence is available.

Our rapid evaluation process has limitations. Although the RERP process was designed to limit the amount of time and resources it takes to complete the review and prioritization process, the time it takes to prepare the materials and synthesize evidence for the panelists is still nontrivial creating a potential barrier if resources are limited. However, a relatively small project team could follow our process and accomplish the majority of the work before convening the RERP. As the amount of evidence to review increases, the amount of preparation time needed may increase.

The focus of this particular rapid evidence review was on the clinician-facing aspects of existing tools, particularly the feasibility and impacts of integrating these complex technologies into clinical workflows. However, we recognize that this is only a first step in implementing new technologies. Fully successful implementation also requires incorporation of patient viewpoints [39-41]. For example, patient engagement is necessary to ensure that these tools strike the right balance between providing patients’ information and protecting their privacy [42]. Technology developers and health system leaders tasked with implementing and evaluating new tools need a robust process that incorporates all key stakeholders, including patients [39-41].

Although the RERP process provides a method to address the critical question, “Which PRO and SM tools should we prioritize for further study and implementation?,” it does not solve all of the challenges health systems face when seeking to use these complex tools to improve clinical care. It is likely that implementation challenges, both resulting from infrastructure limitations and clinician concerns, have limited utilization of these tools. To scale up use of PRO and SM tools in different clinical contexts nationally, a computational infrastructure that can support interoperable applications is necessary to support data collection and curation. PRO and SM tools may be an excellent use case for machine-encoded, computable biomedical knowledge curation, and execution platforms.

In the context of a relatively narrow and recent area of study, technology-based communication and support tools to improve the patient experience of cancer care, we identified numerous RCT-tested tools. To achieve the important task of improving the patient experience of cancer care, we needed a systematic and trustworthy process for identifying and prioritizing the most promising tools for further study and implementation.

Although health systems focused their efforts solely on tools with randomized trial evidence showing they can improve patient-important outcomes, the number of potential tools will likely exceed the system’s capacity to put them into practice. Moreover, these technologies can be complex. Integrating novel tools into clinical workflows has proven challenging [43]. Thus, even more than with other types of interventions, randomized trial’s evidence of the tool’s ability to improve outcomes may not translate into effectiveness in real-world settings. The RERP process presents a method to streamline the process of guideline review and data collection while maintaining a rigorous evidence-based grounding. We took advantage of multiple existing frameworks to streamline our process while maintaining rigor: current evidence searches and environmental scan procedures [17,44], the National Cancer Institute’s Research-Tested Intervention Program’s review process, the modified Delphi panel process, the GRADE ratings and summary of findings tables, and the TICD coding framework. Using these existing frameworks for each part of the evidence search and review process allowed for a systematic process that was feasible to complete within approximately 4 months.

In addition, the identification of potential useful features or perceived implementation barriers by experts (4b and 4c) may help health system leadership understand how the high-priority tools need to be adapted before implementation. The evaluation of the benefits and drawbacks of specific features of tools may inform the design or configuration of new technologies before implementation. For example, a system architect may consider deleting or modifying some features seen as barriers and including other features viewed as helpful. Thus, important next steps include taking the findings of an RERP to a local group of decision makers for validation and to determine how tools need to be adapted to fit a local context.

Before PRO and SM tools, or other digital tools, may be broadly used, proper assessment of their potential feasibility and impact using an RERP process may be beneficial. The RERP process presented here may enable health care administrators to make more efficient and effective decisions about the implementation of novel technologies in clinical practice.