Our research utilized focus groups and open-ended questionnaires distributed firstly at a showcase event dedicated to development of the REACT website, and secondly during a trial of REACT within community settings. Six focus group interviews preceded with a trial of the website involving the “think aloud” technique [29,31] were conducted with members of public and health care professionals (who could potentially use REACT in the future) in Greater Manchester, UK. Furthermore, the research team organized an event showcasing REACT to members of public. Participants of the event were invited to participate in research, use the REACT website, and provide their feedback in an open-ended questionnaire [32]. The same open-ended questionnaires were also used in a trial within community settings where members of the public could fill in the REACT questionnaire with assistance of a health care professional.

Data were collected between June 2017 and January 2018, with a timeline for each research step illustrated in Figure 1. Participant recruitment, data collection, and data analysis continued simultaneously, allowing for website iterations between different points of data collection and check for acceptability of the modifications in subsequent stages of data collection. The majority of changes to the website were implemented between July 7 and August 21 2017, and between August 23 and October 2, 2017. Following the focus group on October 3 2017, only minor changes were implemented to the website as the feedback gained in those studies predominantly featured themes and ideas previously mentioned by research participants, thus pointing to data saturation [33]. The issue of data saturation was also discussed and agreed upon by members of the steering group overseeing the research project, meeting on a monthly basis. Ethical approval was obtained from the University of Manchester Ethics Committee (Ref: 2017-2065-3599). All participants participated in the research voluntarily and provided written consent.

Our research into REACT involved 6 focus groups with 15 members of public and 20 health practitioners, and 24 open-ended questionnaires with self-reported evaluation of the REACT website provided by members of public.

Three focus groups involved members of public and 3 involved health care practitioners. The “think aloud” technique was used to explore users’ website experience. Qualitative research, especially involving “think aloud” procedures, is considered the most appropriate method to gain insights about user perceptions and for usability testing of websites [29,30,34]. The “think aloud” technique relies on research participants reporting their experiences, thoughts, and ideas while using the website. This approach addresses the issue of data loss which can be experienced when information is collected after website use. This technique typically leads to identification of between 80% to 90% of usability problems of an evaluated website [35-37]. Focus groups were preferred over one-to-one interviews as they allow for views and opinions to be developed and discussed, at the same time allowing for reporting of individual opinions [38].

Apart from focus groups, we also utilized open-ended questionnaires enabling self-reporting of user experiences without the presence of a researcher, often referred to as asynchronous remote usability evaluation [39]. Such techniques are deployed due to the benefits of collecting usability data from many participants in a relatively short amount of time [39]. Use of open-ended questionnaires is also a technique used in market research for the purpose of evaluating consumer and user views about a product or service [32]. Apart from being easy and convenient to implement, this technique is appreciated for its ability to elicit spontaneous views and less prone to response bias [40].

Focus groups one, two, and six were made up of members of the public. Focus groups three to five involved health care professionals. Members of the public (men and women) aged 40 years and older, with or without a previous history of cancer, and with a range of backgrounds were invited to participate in the 3 focus groups. We targeted people who could potentially use REACT in the future. Individuals with a history of cancer were not excluded, as (1) having cancer does not mean someone will not develop another type of cancer in the future; and (2) those individuals could evaluate the symptoms they had experienced and raise issues in the event that the REACT algorithm was not accurate. Our sampling objective was to obtain diverse representation of views, not to compare views of different groups of people. Focus groups participants were recruited through Macmillan Cancer Support, via posters around the university and community centers within the Greater Manchester area (eg, gyms and libraries); and from the NHS Cancer Bowel Screening Program. Participants were reimbursed for their time with a £20 high street voucher except for two participants who did not accept it. Participant characteristics are summarized in Table 1.

In addition, health promotion or prevention professions that could in the future take an active part in implementing REACT into health services ecosystem in the UK were invited to participate in 3 focus groups. Participants were (1) employees of a community pharmacy (n=10), (2) NHS Health Check workers within Greater Manchester (n=5), and (3) members of the NHS Cancer Prevention and Early Intervention group in Greater Manchester (n=5). These focus groups were organized as part of presentation sessions about REACT to the Clinical Commissioning Groups, where GPs and other health professionals were present.

Participants evaluating the REACT website through open-ended questionnaires were recruited through (1) an event organized by Greater Manchester Cancer Vanguard (GMCV), the founder of research into REACT, and (2) a trial of the REACT website within community settings. The showcase event was a presentation of research undertaken into REACT and was open to the public and GMCV associates (patient groups, health care representatives, and industry), and was advertised through various channels, such as the GMCV website, newsletters, social media channels, and email. The event was attended by 32 participants and 10 of them agreed to provide evaluation of REACT through an open-ended questionnaire.

As the feedback from all the data collection pointed to use of REACT in assisted manner, with a health care professional present, we also evaluated user experiences during a trial of REACT in community settings. A local community pharmacy agreed to recruit participants (pharmacy customers) and assist them with filling in the REACT questionnaire. Following the evaluation, participants were asked to provide their feedback through the use of the same open-ended questionnaires used after the GMCV showcase. A total of 14 questionnaires were collected during that research phase.

The focus groups were conducted by 3 moderators—female researchers with experience of conducting qualitative research (a market researcher, an epidemiology researcher, and an academic clinician)—as well as the software engineer responsible for the design of the REACT website and a note taker. Each focus group lasted between 60 and 90 minutes. Most of the focus groups were performed in a room at the University of Manchester, and 2 focus groups with health care practitioners were performed at the participants’ workplace. Field notes were taken in each focus group [41].

The user experiences of REACT were collected during a trial of the website by using the “think aloud” technique [29]. During the “think aloud” procedures participants in each focus group were split into smaller groups of 2 to 3 participants, with each group accompanied by one of the focus group moderators. This separation was aimed at obtaining independent views, unbiased by the influence of the majority of the focus group participants.

After the trial of the REACT website, participants were asked (after merging into one group) a series of questions about different pages and sections of the website (eg, landing page or cancer questionnaire page). The questions about different pages were accompanied by a screenshot from that page.

Finally, participants were asked about their general impressions of viewing and using the website within the health care service (eg, on their own or with the help of health care practitioner). Depending on the area of expertise (ie, members of the public or health care practitioners) emphasis was placed on different questions. For instance, members of the public were questioned more about the user experience than the practitioners were; and the opposite was the case for the questions about potential implementation of programs such as REACT within health care services.

The link to open-ended questionnaire was emailed to the participants of the GMCV showcase who had previously agreed to participate in research. Those individuals were provided with a temporary link to the REACT website (available only to those participants for a week) and they were asked to use the website and to fill in the open-ended questionnaire. The questions asked in the questionnaire are provided in Multimedia Appendix 1. The same questionnaire was later used in the trial within community settings, when research participants filled in the questionnaire after completing the REACT assessment in their community pharmacy.

The interviews were audio-recorded and transcribed verbatim by a professional agency. The data obtained from open-ended questionnaires were exported from the survey software into a Word document. Data were analyzed with a thematic analysis approach [33]. Four researchers (the same group that was involved in data collection procedures) read all the documents and searched for patterns in the data, focusing on the patterns that related to research questions and objectives of the study.

The coding of usability evaluation data was guided by 2 different perspectives: (1) navigation strategy, pointing to used navigation tools, and (2) navigation problems (or facilitators), pointing to potential barriers (or facilitators) to completing the cancer risk evaluation activity [29,42].

The data about website design were coded into 2 main categories: (1) website design and (2) risk presentation. In the website design section, 3 factors were identified: content and related functionality, readability of the cancer questionnaire, and website look and feel. In the risk presentation section, we identified problems about different types of risk presentation: numeric, visual, and evaluative. The attitudinal and perceptual data about REACT and its future implementation into health care services were coded by highlighting existing cognitive or emotional states as well as barriers and facilitators to implementation.

The initial analysis was performed by 4 researchers (the same group that was involved in data collection procedures). The coders showed fairly high levels of consistency in coding the themes, with kappa statistics between 0.71 and 0.87 [43]. The final validation of the results was performed by members of the steering group overseeing the research project.

Sample characteristics for the members of public group are provided in Table 1. Practitioners participating in this research held various positions, including pharmacists, pharmacy senior management, NHS Health Check workers and management, a multi-agency group, inclusive of a GP from the NHS in Greater Manchester, and cancer awareness facilitators. Of the 20 participating health care practitioners, 13 were female.

In general, the feedback in relation to the website and its implementation into health care services was consistent across members of the public (regardless of whether they had cancer history or not) and health care practitioners. Therefore, the views of these groups are summarized together. In cases where the rationale for certain preferences differed between the groups, the differing views were elaborated.

Finally, the view of the majority of participants was that REACT should be offered in community settings with assistance of a health care practitioner and individuals should not attempt to assess their risk on their own. While we discuss the details on pathways to delivery of REACT later, it is important to mention this issue before discussing the results in more depth, as some of the comments relate to assisted delivery of this service.

The initial reaction to REACT was very positive, with all participants appreciating the value and importance of such an intervention. While some members of public were aware of symptoms of specific cancers (eg, bowel cancer), in most cases, their awareness of different cancers and its symptoms was limited. Hence, they appreciated the capabilities of REACT. As one participant stated:

One of the greatest challenges seen about the future of such a tool was related to creating a positive image of this intervention. This was important, considering the perception and stigma of cancer as a deadly disease. Participants reported that it was of utmost importance to emphasize the positive aspects of early detection tools, and their relevance to cancer management and improving survival rates. This was emphasized by the following quotes from participants:

Some participants suggested using testimonials from cancer survivors or celebrity endorsement to promote early detection and the role REACT can play in it.

Following those suggestions, the content of the REACT website emphasizes early detection success rates (Figure 2, content based on a graphic created by Cancer Research UK) and also contains individual testimonials at the carousel (changing display) available on the landing page.

Participants identified some key features related to a user-friendly design of the website, which can be classified into 3 key categories: content and associated functionality, clarity and readability of the cancer questionnaire, and look and feel of the website.

Risk presentation proved the most difficult aspect in developing REACT, as people wanted to ensure the risk score was understandable and at the same time motivating (serious) enough to lead to action. In general, participants preferred a simple frequency format of risk presentation (eg, x in 100) over percentages (eg, x %). Furthermore, they felt more comfortable with having their risk information provided in both written (numeric) format and a visual format.

Interestingly, when the risk estimation was presented as a graph, participants preferred to have the graph representing one’s disease risk in relation to affected individuals (as illustrated on the left side of Figure 5; in this case, the score would be 5 out of 11, where 11 refers to number of people with the same symptoms diagnosed with the disease). When graphs were used as reflective of the entire population at risk (following the same example discussed above with 5 people, this would be reflected as 5 out of 100 people with symptoms like yours, majority of whom were not diagnosed with the disease, as only 11 were), this made an impression of a very small risk. However, the visuals with the entire population were still desirable, and favorably received if presented as icon arrays (blocks or stick figures such as the “waffle chart” visual used in REACT, as illustrated on the right side in Figure 5).

Regarding the evaluative labels about personalized risk estimates, participants were satisfied with a distinction between low, medium, and high risk. Yellow was used to show low and medium risk, and red to illustrate high risk. Those decisions were again guided by feedback that having any of the symptoms would increase anxiety, and also can signal another disease. In earlier versions of the REACT website, green was used instead of yellow to point to low risk, but while this was liked by members of public, it was discouraged by health care practitioners. For members of public, green was seen as a “safety zone”, assurance that the symptoms are not cancerous. However, health care practitioners saw the “green light” option as potentially leading to complacency in case of non-cancerous symptoms that could signal another illness:

To minimize unnecessary anxiety, the meaning of the score was explained with the following sentence:

As the data collection process allowed for website iterations between different data collection sessions, the process of website changes is illustrated in Table 2.

As indicated earlier, and illustrated throughout the reported results, research participants indicated the preferred methods of delivering and receiving advice from REACT. While some participants believed that the advice could be offered to public (via a public website), the majority believed the best way forward was through a guided approach supported by health care practitioners. This type of delivery was strongly recommended by health care practitioners. As cancer is an emotive topic, it was believed that while some individuals could cope with the results they obtained, others could experience stress and anxiety about their results. Considering the fact that some individuals might need help with interpreting and answering the REACT questions (eg, what is normal to me and how do I report it in the questionnaire) and results (eg, I have a low risk of cancer but I am still concerned about the symptom(s) I had reported), as well as using technology, it was believed that they might need professional help in order to complete and understand their cancer assessment correctly. Considering this feedback, it was not recommended that REACT or similar websites are available in the public domain.

Participants believed that REACT can be offered at a variety of locations, where the needed support can be offered. Apart from community pharmacies and NHS Health Check services, respondents who filled in the open-ended questionnaire indicated that REACT would be a desirable addition in voluntary organizations (87% agreed), leisure centers (79%), council offices (75%), and workplaces (75%), to a lesser degree in benefit offices (46% agreed this was a good idea).

In this study we provide novel insights into how members of public and health care practitioners perceive a Web-based intervention providing personalized symptom-based cancer risk estimation. We report the process of developing such a website and evaluate opportunities for introducing such an intervention within health care services in the UK. Our results show that there is a need for such a tool and that it would be well received. The best way to offer it to the public appears to be through a guided approach, where a trained individual supports members of the public in the process of risk assessment and evaluation.

Website design, content, functionality, look and feel as well as risk evaluation and presentation are all important factors affecting perceptions of usability of such a website. Users want a positive image of early cancer detection, a simple website with the ability to evaluate and detect multiple cancers, and value added in the form of explanation of their risk score and further lifestyle-based information about potential reduction of their future risk of having cancer.

While medical content and sources of information on such a website are important, individuals want those tools to have a “human face” and community feel that can be conveyed by the use of real-life images of people from various backgrounds and links to social media, community groups and portals. In terms of risk presentation, a direct approach to cancer communication is preferred. All risk presentation types (ie, numeric, visual, and evaluative) are appreciated, with two types of visual risk representation desired: with affected individuals only and the entire population at risk.

The results of this study into a wider context need to be interpreted with consideration of the limitations. First, this research is limited to the context of the UK health care system and the 5 evaluated cancers. Consequently, future research can evaluate different cancer types and as the number of questions in the REACT questionnaire increases, further work using cognitive techniques [62] will be needed to further refine question wording and response options. Furthermore, due to differences in health care systems across the world, implementation pathways for interventions like REACT might be different for different countries.

Second, participants representing members of public were a small and self-selected group of individuals. This means that our sample could be limited to individuals who show a strong interest in their health or even in (avoiding) cancer, especially considering the fact that some were previously diagnosed with the disease. While we tried to address this limitation and reach a diverse group of participants by recruitment at various locations and using different means, we cannot exclude the possibility that self-selection impacted our results. This issue will be further explored in the next stage of this research project involving an evaluation of REACT. This future evaluation will consider the ability of REACT to detect symptomatic patients as well as its impact on GP workload, secondary care referral rates, and impact on health economics.