From September to November 2024, we conducted usability testing in a single-arm longitudinal proof-of-concept study at an academic medical center in the mid-Atlantic United States. A sample size of 8 participants was selected for the usability evaluation, consistent with the study’s objective of identifying usability issues [20] with RAI Support and obtaining user feedback to inform intervention optimization rather than testing intervention efficacy.

Adult patients were identified from medical record review and through a thyroid cancer nonprofit organization’s publicly available Facebook (Meta Platforms Inc) page, on which the study was advertised. The 2 sources for recruitment were intended to understand the spectrum of experiences of survivors of differentiated thyroid cancer (DTC), RAI symptom burden profiles, and website feedback from individuals across the United States who were treated across a variety of hospital settings.

Eligible patients met the following inclusion criteria: (1) age of ≥18 years; (2) able to provide informed consent in English; (3) diagnosed with DTC (papillary or follicular carcinoma) in the past 3 years; (4) received RAI treatment within the past 3 years; (5) access to a computer, tablet, or smartphone with internet connectivity; and (6) completion of the Radioiodine-6 screener [21] with a total score of ≥7.6 or a score of ≥3 on at least 1 out of the 3 RAI symptom subitems (mouth, eyes, or nasal). Exclusion criteria included non–English-speaking patients and individuals with cognitive, psychiatric, or hearing impairments that could limit their ability to consent or participate in the study.

Following institutional review board approval (#00005285), potential participants were contacted through introductory emails or mailed study flyers, followed by a Health Insurance Portability and Accountability Act (HIPAA)–compliant videoconference with research staff to assess eligibility through the completion of a screening questionnaire. Informed consent was obtained through a 15-minute telephone call or a HIPAA-compliant videoconference. Participants reviewed and signed the informed consent form electronically. A total of 8 patients diagnosed with DTC and treated with RAI within the past 3 years were enrolled.

Our Facebook recruitment approach also led to 29 “imposter participants,” defined as individuals who provide false identities and information for the purpose of participating in research studies [22-24]. Imposter participants were identified by a combination of the following indicators: nonworking telephone numbers or telephone numbers for businesses, refusal to turn on their cameras (a requirement for usability participation), use of similar (yet unusual) email domains, and inconsistent internet connectivity.

To increase rigor and guard against imposter participants, study staff modified the screening procedure to include open-ended questions about the name of the hospital or clinic setting in which participants received treatment. We then ended Facebook recruitment to reduce the likelihood of additional imposter participants. Of note, imposter participants were not included in the final data analysis.

Participants completed baseline and postintervention surveys via REDCap (Vanderbilt University) [25] (for details, refer to the Measures section).

Study staff trained in qualitative research methods developed the semistructured usability testing interview guides (including author AC, a female psychologist, author KM, a female human factors scientist, and a nonauthor, female research and human factors scientist). Interviews lasted approximately 90 minutes.

Study staff (authors) trained in qualitative research conducted the interviews: AMJ (a female research assistant), JP (a male research assistant), and RM (a male research assistant). A second member of the research team (JAL, a female research assistant, and JP), trained in qualitative data research, took detailed notes during the interviews to track usability task outcomes.

Study staff used a think-aloud method to prompt participants to provide feedback as they navigated key website elements, including the log-in process, home page, welcome video, symptoms page, salivary symptom management page, and resource page. Feedback focused on content clarity, visual appeal, and ease of navigation (Figure 1). All recorded usability testing interviews were transcribed verbatim and deidentified prior to data analysis. At the end of the usability testing interviews, participants completed a System Usability Scale (SUS) questionnaire [26] via REDCap. Participants were then asked to continue using RAI Support for 4 weeks. At the conclusion of the 4 weeks, they completed a follow-up questionnaire.

RAI Support has been previously described [9]. RAI Support is a theoretically driven web-based intervention to support self-management of RAI symptoms among survivors of DTC. RAI Support is informed by social cognitive theory [27] and was developed following person-based design principles, with survivors of DTC and health care providers (endocrinologists, endocrine surgeons, nuclear medicine providers, registered oncology dietitians, social workers, and psycho-oncologists) as integral members of the intervention development process [28]. The web-based intervention is asynchronous and consists of content related to education on RAI symptoms; brief provider-led videos on symptom management strategies for mild to moderate symptoms; communication and self-care, including mindfulness exercises; and DTC survivorship resources. Website programming was produced by DreamSoft4u, Pvt. Ltd [29].

Participants completed an exit questionnaire to assess their perceptions of the helpfulness of components of the RAI Support intervention, likelihood of recommending the program to survivors of DTC, satisfaction with intervention length, and overall satisfaction with the program. Responses were assessed using 10-point Likert scales for the majority of items.

Intervention back-end use data were captured through the RAI Support back-end system. Metrics captured included frequency of log-ins and number of website pages visited. Use data were exported in aggregate form for analysis to complement self-report measures of intervention engagement.

A mixed methods approach was used, incorporating quantitative data from questionnaires, usability logs, and the web-based intervention back-end system, as well as qualitative data from usability interviews for analysis. Descriptive statistics were used to summarize the demographic and clinical characteristics and web-based intervention use of participants. Signals of change in outcomes of interest were evaluated using within-group effect sizes calculated with the Rosenthal effect size formula (r=Z/√N), where Z is the standardized Wilcoxon signed-rank test statistic and N is the number of paired observations [34,35]. Cohen guidelines for r effect sizes were used and defined as “small” for r=0.1; “medium” for r=0.3; “large” for r=0.5 [35]. Signed effect sizes were reported to indicate the direction of change, with directionality defined based on follow-up minus baseline. Positive values indicate increases from baseline, whereas negative values indicate decreases.

Content analysis was used to analyze the usability testing interviews [36,37]. To gain an initial understanding of the content, 3 authors (AC, AMJ, and JAL) independently reviewed the interview transcripts and met biweekly to identify and refine preliminary codes and examine data discrepancies until team resolution. Coding was conducted using ATLAS.ti Scientific Software Development GmbH (version 8.15.3; Lumivero LLC). Three authors (AC, AMJ, and JAL) collaboratively developed a codebook, consolidating initial codes into broader categories through iterative discussion. These categories were further synthesized into major themes and subthemes to capture key patterns in participant feedback regarding the context underlying usability issues and potential methods for optimization. Thematic saturation [38] was operationally defined as the point at which review of interviews did not identify new themes or subthemes relevant to usability. Using this approach, no new themes or subthemes emerged from the final interview data, indicating thematic saturation within the scope of the usability evaluation.

Ethics approval was obtained in accordance with the principles of the Declaration of Helsinki. The protocol was reviewed and approved by the Georgetown University Institutional Review Board (#00005285). All participants provided signed informed consent. Study data were deidentified prior to data analysis. Participants were compensated with a US $50 gift card for study participation.

A total of 76 individuals were assessed for eligibility (Figure 2). A total of 47 (61.8%) individuals contacted study staff in response to Thyroid Cancer Survivors’ Association Inc (ThyCa) Facebook advertisements, and 29 (38.2%) individuals were screened for eligibility from medical records. In total, 24 participants declined study participation: 19 were not interested and 5 were too busy. Overall, 64 individuals were excluded, including 24 who declined and 40 who were ineligible for the following reasons: 3 were diagnosed with DTC >3 years ago, 1 received RAI >3 years ago, 1 did not receive RAI, 1 was not an English speaker, 29 were imposter participants, and 5 did not meet screening criteria. Among those who were eligible, 12 participants consented to the study.

Of the total sample of consented participants (n=12), 1 was lost to follow-up and 3 withdrew, leaving a final sample of 8 consented participants who completed baseline, usability testing, and postassessment procedures. In the final study sample (N=8), the average age of participants was 45 (SD 8.2; range 35-61) years. In terms of clinical characteristics, the average times since diagnosis and RAI treatment were 1.2 (SD 1.1) and 0.6 (SD 0.4) years, respectively. Most participants were diagnosed with papillary thyroid cancer (7/8, 87.5%), and most received an RAI dose of 100 to 150 mCi (n=6, 75%). See Table 1 for demographics and clinical characteristics.

Eight participants completed usability testing sessions. The mean SUS score was 88.1 (SD 8.7), indicating above-average usability. All participants effectively completed each task. Ease of use, as measured by completion of a task on the first attempt, ranged from 100% to 87.5%, and error rates ranged from 0% to 25%. Participants rated the clarity of the welcome page (mean 4.3, SD 0.8) and the welcome video (mean 5.0, SD 0; task 1), indicating that the page was perceived as clear and the video as very clear. Mobile device screen sharing blocked the visuals needed to complete tasks 3 (RAI symptom library) and 4 (health care provider–led salivary gland massage video) for 1 participant. When the screen sharing issue was resolved with prompting from study staff, ease of use for this participant increased to 100% for task 5. Refer to Table 2 for an overview of the usability outcome metrics, including error rates by task.

Participants reported that they found the website content “informative,” “helpful,” and that they would recommend the website to someone else experiencing RAI symptoms. User testing feedback provided insight into barriers to user engagement, contextual information about the root causes of barriers to user engagement with the prototype, and areas for improvement. Overall, participants highlighted navigation across website pages as a primary area for improvement. Participants noted challenges in locating specific content and suggested adding hyperlinks to facilitate navigation. Participants also reported being timed out of the website and suggested increasing the session idle time. Example suggestions for website improvement and refinement are illustrated in Table 3.

Participant scores on the exit questionnaire (n=7) indicate moderate to high ratings of helpfulness for the website videos: managing dry mouth (mean 7.14, SD 2.11), changes in taste (mean 6.5, SD 2.65), and relaxation and mindfulness (mean 6.83, SD 1.94). Back-end use data indicate that, on average, 6 (85.7%) of the 7 website users engaged with 3 out of the 4 content sections during the 4-week home intervention period. Refer to Multimedia Appendices 1 and 2 for the participant exit questionnaire and back-end metrics.

Preliminary effect size signals of change were calculated for participants who completed baseline and postassessment measures (n=7). The distributions of scores on outcomes of interest are presented in Table 4. Overall, preliminary effect size signals of change were found in self-efficacy (r=0.32), global mental health (r=0.31), and HRQOL (r=0.45), with positive effect sizes indicating increases from baseline. A small negative effect size was observed for RAI informational needs (r=−0.15), indicating a favorable decrease in informational needs from baseline. RAI symptom burden and psychosocial informational needs remained relatively stable throughout the study, and global physical health decreased across study time points.

Findings from this study add to the literature by showing high usability and preliminary proof-of-concept data for a website designed to support symptom management following RAI treatment among patients with DTC. In addition, this is among the first studies to evaluate the usability of a web-based intervention for symptom management in an understudied population of survivors of DTC. Overall, participants found the RAI Support intervention to be informative and the health care provider–led videos to be helpful. Participant feedback identified user-centered areas to enhance the intervention’s ease of use, including navigation across webpages and interactive features to foster user engagement. Future adequately powered studies should further examine temporal patterns and predictors of intervention engagement, including whether sustained or repeated use varies by baseline RAI symptom burden, clinical characteristics, or recruitment source, to better understand for whom and under what circumstances the RAI Support intervention is most engaging. These focal points are consistent with broader gaps identified in the implementation literature on digital health interventions in thyroid cancer care. A recent scoping review of digital health interventions in thyroid cancer care synthesized intervention characteristics, effectiveness, and uptake, as well as reported barriers and facilitators, and identified few interventions focused on symptom self-management following primary treatment [39], highlighting the limited availability of digital approaches addressing this need. Within this context, among participants with complete pre-post RAI intervention data, we found evidence of small to medium clinically meaningful changes in self-efficacy for managing chronic symptoms, RAI informational needs, global mental health, and HRQOL. These findings support the potential of digital interventions such as RAI Support as a valuable and feasible approach to address symptom management needs, with further research and refinement likely to enhance their utility.

Results did not support RAI Support’s preliminary effect size signals of change on RAI symptom burden, informational needs for personal growth, and global physical health. One explanation for these findings is that the RAI Support intervention is intended to provide noncurative symptom management strategies, which may not eliminate the persistent nature of the frequency and severity of RAI symptoms [40-42]. Additionally, these data did not capture information on common medical comorbidities among survivors of DTC such as myocardial infarction, osteoporosis, hyperthyroidism, and hypothyroidism [10,11], which may explain the pre-post differences in PROMIS Global Physical Health scale scores in the sample. Future studies with larger samples that capture common medical comorbidities are warranted to further evaluate these outcomes. Another explanation for these findings is that the static informational content of the website may not fully address the range of patient needs. Future research could explore more integrative and interactive features, such as direct messaging with health care professionals or cancer-specific conversational agents to provide tailored guidance and deeper patient connection [43,44]. Recent interventions using a range of web-based elements yielded a reduction in chemotherapy-related side effects and improved effectiveness in self-care behaviors in patients with cancer [43,44]. These findings also contribute to a broader shift toward risk-adapted, digitally enabled thyroid cancer care. Advances in molecular testing and AI-assisted cytopathology are refining the diagnostic evaluation of indeterminate nodules, while biomarker approaches are being examined to refine selection for RAI [45-47]. Together, these changes further underscore the potential value of scalable digital survivorship interventions, such as RAI Support, for addressing persistent symptoms and supporting patients throughout posttreatment care.

There are several limitations of this study. Despite recruitment efforts through a publicly available thyroid cancer nonprofit Facebook page, most participants were recruited based on purposeful sampling from electronic medical record review. This led to participants in the study being relatively similar, with the majority being female, well educated, and from higher-income backgrounds (>US $200,000 per year). Although DTC does predominantly affect female individuals, having the preponderance of study participants come from such analogous backgrounds limits the generalizability of the study to larger and more diverse DTC populations. Usability and acceptability need to be confirmed in patients with broader socioeconomic, educational, and digital literacy backgrounds before implementation.

The small sample size of the study was appropriate to address the aims of the usability evaluation and ORBIT (Obesity-Related Behavioral Intervention Trials) model proof-of-concept testing, which focused on identifying preliminary effect size signals of change [20,48]. However, this limited sample size and the absence of a comparison condition further limit the generalizability of the findings and do not allow analyses of statistical differences or preliminary intervention effects. The study also covered a short time frame for participants, with user engagement and clinical outcomes tracked only over a 4-week period. This limited interval of analysis prevents a full understanding of the long-term benefits of the intervention for patients after RAI treatment.

An additional limitation is that the study encountered 29 imposter participants. Although those individuals were excluded, this recruitment issue is relevant to feasibility, recruitment, and generalizability of remote usability testing procedures and should be interpreted with caution [20,48]. The potential influence of outlier scores is another limiting factor, such as for PROMIS Global Physical Health, particularly given the small sample size. Future pilot feasibility trials should incorporate larger participant samples from different geographic areas, participants with varying intervals since RAI treatment, and a control condition to allow more rigorous evaluation of the RAI Support intervention.

This study demonstrated a positive usability testing evaluation of the web-based RAI Support prototype for RAI symptom management among survivors of DTC, a population that has received limited survivorship-focused attention. The study also identified preliminary effect size signals of change in self-efficacy in survivors of DTC, global mental health, and HRQOL. Ongoing research has incorporated the patient-recommended refinements and will examine the feasibility and acceptability of the RAI Support intervention through a pilot randomized controlled trial.