Systematic chemotherapy continues to be the main treatment modality for almost all major cancer types [1]. Chemotherapy is associated with a myriad of symptoms and adverse treatment side effects that can range from mild to life-threatening, severe, and disabling [2]. Therefore, early recognition and effective management of these symptoms by both clinicians and patients are critical to reducing physical and psychological treatment sequalae [2].

There is growing evidence in support of using patient-reported outcome measures (PROMs) for improving symptom management [3-5]. The increased number of mobile phone users creates opportunities for developing models of supportive care that use these technologies for monitoring PROMs to improve home-based, proactive “real-time” symptom monitoring and management [6-8]. Several Web-enabled PROMs systems have been trialed in oncological settings [4,8-10] and shown to support patients in managing chemotherapy-related symptoms [3,11], improve symptom control [11,12], and enhance patient-clinician communication [4]. However, few of the current systems have been developed in line with best-practice guidelines in user-centered design nor have verified the system usability during the stages of system development [5,13], which could impact their use by patients and clinical integration in practice settings [14].

The aim of this study, which is part of a larger project to enhance the provision of timely, high-quality, person-centered supportive care, is to evaluate the usability of a mobile phone–based technology, the Advanced Symptom Management System (ASyMS), from the perspective of Canadian patients with cancer (colorectal and lymphoma) receiving chemotherapy in a controlled usability testing environment. The secondary aim of our end-user testing is to explore users’ performance and satisfaction with the system interface and their perspectives and experience with the system and the content of ASyMS [9]. In addition, this study assesses the ASyMS against a set of human factors design guidelines and heuristics to increase the likelihood of discovering more design features and function issues that could impact user experience and willingness to use the system.

The ASyMS, one of the more advanced remote monitoring systems, is a mobile phone–based device designed to monitor and manage chemotherapy-related toxicity in the home setting. It enables real-time remote monitoring of cancer symptoms using PROMs [15]. The ASyMs uses innovative risk prediction modeling and decision-support tools that allow for timely, high-quality, person-centered supportive care for better treatment toxicity management [10,16].

Patients using the ASyMS complete an e-symptom PROMs questionnaire to assess the occurrence, severity, and distress associated with each symptom. After completing the questionnaire, patients immediately receive evidence-based, self-care advice on the mobile phone based on the specific symptoms reported, which facilitates the self-management of symptoms. Leveraging evidence-based algorithms, symptoms reported through the device that meet a threshold criteria (ie, high level of severity) trigger alerts to cancer care clinicians, usually nurses, who on the receipt of an alert can view patients’ symptom reports on a secure webpage and contact patients directly at home by telephone, enabling the initiation of proactive clinical interventions (Multimedia Appendix 1) [17].

The ASyMS was developed in the United Kingdom based on the extensive patient and clinician engagement, and its utility and acceptability have been tested in UK populations [18,19]. The effect of the ASyMS intervention on patient outcomes is uncertain and is being tested in a large multisite trial in European countries [16]. We undertook a study to test the usability of the ASyMS program to identify its potential for the uptake in a Canadian cancer population. The ASyMS program was installed on an Android mobile phone with a 5.00-inch touchscreen display with a resolution of 720 pixels by 1280 pixels. Figure 1 and Multimedia Appendix 2 show a preview and some features and functions of the ASyMS.

We used a mixed-method approach (qualitative and quantitative data sources) to increase the depth of evaluation and support methodological triangulation to improve the reliability and validity of findings [20]. Mixed-methods also allow for a more comprehensive understanding of participants experience and enable the identification of specific usability issues [20,21]. A usability study evaluates how a specific process or product works for individuals and the extent to which a user can use a product to achieve specific goals (interaction between user and task in a defined environment) [22,23].

Data collection combined user-based testing using a think-aloud technique, semistructured qualitative interviews based on a qualitative descriptive methodology [24], and short answer quantitative questionnaires; all these methods have been used widely for usability testing [25]. Specifically, think aloud is a user-related method for assessing usability where users are encouraged to verbalize their perceptions out loud as they interact with the system [26]. Participants’ experiences with the system evaluated through qualitative interviews and questionnaires can inform potential for future uptake [27].

Estimation of the sample size for a usability test depends on several variables, including types of test users available, the mission criticality of a system (any factor that is essential for system operation), and problem discovery rate (the number of usability issues that can be uncovered by users) [28,29]. Although, it has been shown that 80% of the usability problems can be detected with 4 or 5 participants in a usability testing [30], Faulkner [29] found that the minimum percentage of identified usability issues increased from 55% to 82%, and the mean percentage of issues increased from 85% to 95% when the number of participants was increased from 5 to 10. Thus, in this study, we aimed to recruit a minimum of 10 patients. We used a purposive sampling method to ensure maximal variation in end-user characteristics, specifically younger (age <50 years) and older (age >50 years) adult patients with diverse cancer types (colorectal or lymphoma), males and females, and those with and without experience in using mobile technology.

The Institutional Review Board Approval was obtained from the University Health Network (UHN) to conduct the study prior to recruitment (#15-9432). Patients were recruited from ambulatory follow-up clinics at the Princess Margaret Cancer Center, a cancer research center affiliated with the University of Toronto as part of the UHN. The inclusion criteria were that patients received, at least, one cycle of chemotherapy for treatment of their cancer (colorectal or lymphoma), were aged >18, and able to participate in usability testing for “think aloud” in English. All participants gave informed written consent for participation in the study.

The main goal of user-centered methods is to involve real users, elicit their views and experiences of the intervention to identify usability issues [31,32]. To meet the aim of this study, usability sessions were videorecorded from multiple angles, and participants were encouraged to share their thoughts verbally as they progressed through a set of predefined tasks (think aloud) [26]. We aimed to elicit feedback and identify design, functionality, and usability issues. In addition, participant experiences, thoughts, feelings, and satisfaction with the ASyMS were assessed by an audiotaped, semistructured, face-to-face qualitative interview with participants and through completing a short questionnaire (modified Telehealth Acceptance Measure, TAM), immediately after usability testing sessions (Multimedia Appendix 3).

The TAM questionnaire comprises 10 questions on a Likert scale ranging from 1 to 7; higher scores indicate greater motivation to use telehealth, more favorable perceptions of the successful use of telehealth, greater patients’ belief that significant others would like them to use telehealth, and more positive attitude toward using telehealth. The TAM questionnaire is designed to assess patients’ motivation to use telehealth and includes questions that are derived from the theory of planned behavior, a model that explains the factors that underpin people’s motivation to act [33]. We used this questionnaire to indicate participants’ overall motivation and readiness to use the ASyMS device, assess participants’ perceived behavioral control, subjective norms, attitudes toward the device, and the extent to which individuals perceive that significant others want them to use the device.

Participants attended usability testing sessions at the Healthcare Human Factors labs at the Centre for Global eHealth Innovation, UHN. Each participant was advised that the aim was to test the ASyMS device and not participants. In addition, participants received written and verbal information regarding the testing procedure, and a brief introduction to the ASyMS before usability testing commenced.

Before starting the session, participants completed a demographic questionnaire. Participants were given a case scenario and a simulated symptom experience they might have during one of their chemotherapy cycles. Participants were requested to follow the tasks provided to them on the ASyMS device, representing typical user goals. Throughout testing, each participant was requested to perform specific tasks that consisted of the following: completing the e-symptom questionnaire (PROMs) on the ASyMS device; finding information about side effects and self-care; filling out the anytime section of the symptom questionnaire; and finding a history of side effects (Multimedia Appendix 4). A trained moderator guided participants through the testing procedure but did not intervene or disrupt the thinking-aloud process. Furthermore, from the observation room, behind a one-way mirror, 2 observers watched the interaction, made notes about what was verbalized, and observed to inform the analysis, and ensured the entire session was recorded. Each task was timed during the test.

After the usability sessions, participants completed the posttest questionnaire to assess their perceptions about the usability of the ASyMS (Multimedia Appendix 3). In addition, they participated in a face-to-face interview regarding the utility and acceptability of the ASyMS in managing chemotherapy symptoms, parts of the content or aspects of the system they liked or disliked, and the reason for their response. The complete testing procedure for all steps averaged approximately 2 hours (range 1.5-2 hours).

The audio and video recordings from the usability and interview sessions were transcribed. The thematic analysis was used to identify all emerging issues and the relations between the themes [34,35]. The identified issues were coded and categorized according to the type and frequency of occurrence [35]. Data collection and analysis continued until no more patterns or themes were emerging from the data [36]. Two members of the research team reviewed the transcripts. Any discrepancies between reviewers were resolved through discussion or the involvement of a third reviewer, if necessary. All qualitative data were coded using NVivo 10 qualitative data analysis software. In addition, a set of variables related to the participants’ performance, including the number of errors each participant made, requests for help, the time taken to complete the task, participant feedback, observers and moderator’s notes, and reviewing the videos, were used to identify a list of usability issues. Descriptive statistics (means, medians, ranges, frequencies, or percentages) were used to summarize these data.

Table 1 presents the characteristics of the study participants. Of 10 participants, 7 were male and 3 were female, with an average age of 68 (range 18-78) years. Most participants (n=8) had higher education (college or university). All participants had their own mobile phone, of which 70% (7/10) had a smartphone, whereas 30% (3/10) owned a regular cell phone (not a smartphone). In addition, 60% (6/10) of participants mentioned that they were comfortable or very comfortable using these devices; 80% (8/10) were comfortable using the internet.

Using the video analysis, the task completion times, the number of errors made by participants while completing tasks, and the number of times they asked for help are shown in Table 2. We followed the TAM developers instructions to score and interpret the TAM (Multimedia Appendix 3). Overall, 80% of participants (8/10) scored >4 on Q2, Q4, and Q5 (mean=5.8), indicating high motivation to use the ASyMS device. In addition, 70% of participants (7/10) scored >4 on Q3, Q6, and Q7 (mean=5.6), indicating they had positive perceptions of the successful use of the ASyMS, and all participants (n=10) scored >4 on Q8 and Q9 (mean=6.1), showing they believed that significant others would like them to use the ASyMS. Furthermore, all participants (n=10) scored ≥5 on Q10 (mean=6.3), suggesting a positive attitude toward using the ASyMS device in future (Table 3).

The thematic analysis of the interview transcripts and participants’ feedback generated 3 overarching themes and related subthemes: ASyMS user-friendliness, with subthemes of design, navigation, and ease of use of the ASyMS; usefulness of the ASyMS (content quality and richness), with subthemes of self-care advice and information on the ASyMS, and appropriateness of the ASyMS questions; and intention to use, with subthemes of acceptance and satisfaction with using the ASyMS in future.

Both the quantitative and qualitative data from the usability testing identified several design and functionality issues for the ASyMS’s device that may negatively impact its efficient use. Each of the recognized issues was mapped to source events (ie, participants’ feedback, errors, and moderator observation). Moreover, each of the issues was classified in one of the 8 usability heuristics for mobile devices (ie, match between system and the real world, ease of input, and screen readability) [37].

The identified issues shown in Table 4 mostly relate to the navigation of the ASyMS device.

Participants through usability testing and the interview commented on the need for a more advanced and attractive design, with better functionality and features in ASyMS to better address the needs of end users, as indicated below:

Some older participants (age>65) commented that they would prefer to use ASyMS on a device with a larger screen (with larger font size) and a more effective color scheme that better draws users’ attention toward specific elements on the screen.

As participants were not familiar with the system prior to the usability session and no tutorial of the ASyMS and its functionalities was given, they often felt insecure about their actions and asked for assistance and approval before performing tasks. Most participants mentioned that they would need some time to learn and get familiar with the ASyMS device before they could start to use it regularly. By the end of the usability sessions, when participants had gained some experience with using ASyMS, all participants agreed that with experience in using the ASyMS, they would get familiar with its features and definitely use it more efficiently, as noted in the following participant quote:

Regarding the self-care advice and information provided in the library, almost every participant commented that the ASyMS provides a lot of quality information. Two participants suggested that using the information would be easier if the self-care information was better categorized.

One older (aged 70 years) participant, who felt comfortable using a smartphone and had a high level of education suggested that it would be helpful if the information and advice could be customized to the specific therapy that patients receive. In addition, he commented that the self-care information he sought was not covered in the self-care advice as he experienced different treatment toxicities and suggested to modify and enhance the self-care advice.

All participants mentioned that the ASyMS would be a valuable device to use for managing cancer treatment side effects. Results from the modified TAM and the interviews suggested that almost all participants were satisfied and pleased with their experience in using the ASyMS device, and this positively influenced their attitude toward using the ASyMS in the future. Participants indicated confidence that the problem of communication with their health care provider can be solved (to some extent) by using such a device, and it will help participants to manage their symptoms quicker than the other current available options.