Published on in Vol 8, No 3 (2022): Jul-Sep

Preprints (earlier versions) of this paper are available at, first published .
Providing Care Beyond Therapy Sessions With a Natural Language Processing–Based Recommender System That Identifies Cancer Patients Who Experience Psychosocial Challenges and Provides Self-care Support: Pilot Study

Providing Care Beyond Therapy Sessions With a Natural Language Processing–Based Recommender System That Identifies Cancer Patients Who Experience Psychosocial Challenges and Provides Self-care Support: Pilot Study

Providing Care Beyond Therapy Sessions With a Natural Language Processing–Based Recommender System That Identifies Cancer Patients Who Experience Psychosocial Challenges and Provides Self-care Support: Pilot Study

Original Paper

1de Souza Institute, University Health Network, Toronto, ON, Canada

2Department of Psychiatry, University of Toronto, Toronto, ON, Canada

3College of Professional Studies, Northeastern University, Toronto, ON, Canada

4Faculty of Medicine, University of Toronto, Toronto, ON, Canada

5The Michael G DeGroote School of Medicine, McMaster University, Hamilton, ON, Canada

6Research Centre for Data Analytics and Cognition, LaTrobe University, Melbourne, Australia

7Department of Computer Science, University of Toronto, Toronto, ON, Canada

8The Department of Supportive Care, Princess Margaret Cancer Centre, Toronto, ON, Canada

9Faculty of Social Work, University of Toronto, Toronto, ON, Canada

10Centre for Addiction and Mental Health, Toronto, ON, Canada

Corresponding Author:

Yvonne W Leung, PhD

de Souza Institute

University Health Network

222 St Patrick St Office 503

Toronto, ON, M5T 1V4


Phone: 1 844 758 6891


Background: The negative psychosocial impacts of cancer diagnoses and treatments are well documented. Virtual care has become an essential mode of care delivery during the COVID-19 pandemic, and online support groups (OSGs) have been shown to improve accessibility to psychosocial and supportive care. de Souza Institute offers CancerChatCanada, a therapist-led OSG service where sessions are monitored by an artificial intelligence–based co-facilitator (AICF). The AICF is equipped with a recommender system that uses natural language processing to tailor online resources to patients according to their psychosocial needs.

Objective: We aimed to outline the development protocol and evaluate the AICF on its precision and recall in recommending resources to cancer OSG members.

Methods: Human input informed the design and evaluation of the AICF on its ability to (1) appropriately identify keywords indicating a psychosocial concern and (2) recommend the most appropriate online resource to the OSG member expressing each concern. Three rounds of human evaluation and algorithm improvement were performed iteratively.

Results: We evaluated 7190 outputs and achieved a precision of 0.797, a recall of 0.981, and an F1 score of 0.880 by the third round of evaluation. Resources were recommended to 48 patients, and 25 (52%) accessed at least one resource. Of those who accessed the resources, 19 (75%) found them useful.

Conclusions: The preliminary findings suggest that the AICF can help provide tailored support for cancer OSG members with high precision, recall, and satisfaction. The AICF has undergone rigorous human evaluation, and the results provide much-needed evidence, while outlining potential strengths and weaknesses for future applications in supportive care.

JMIR Cancer 2022;8(3):e35893



Cancer and its treatment can significantly decrease the psychological well-being of patients and their families. Emotional distress, particularly related to symptoms of depression, is common among cancer patients and is associated with poor treatment adherence, reduced quality of life, and higher mortality rates [1-3]. The COVID-19 pandemic has amplified this psychological burden, resulting in a global rise in mental distress, especially among cancer patients, because of immunological concerns [4]. Virtual care, such as care from online support groups (OSGs), has become increasingly important in health care delivery, particularly with the more recent impact of COVID-19 that has resulted in the need for social distancing and minimal travel. OSGs offer a convenient and economical solution for those who cannot attend in-person support groups, and successfully reduce patient distress and anxiety [5-8].

Synchronized professionally led OSGs engage participants in therapeutic interactions. Group leaders facilitate the sharing of personal experiences among group members with similar challenges. The aim is to foster a mutually supportive environment to achieve an increased sense of empowerment via the vicarious learning that occurs through group membership and an increased sense of control through being better informed about the conditions [9].

A recent paradigm shift in health care, described as the learning health system, refers to a system of care involving the extraction of actionable information to inform clinical decisions whilst measuring patient experience responses for continued quality improvement [10]. Advances in artificial intelligence (AI), such as machine learning–based natural language processing (NLP), afford the development of learning systems that allow for real-time monitoring and responding to multiple participant care needs in virtual care settings. In a larger project, we leveraged machine learning–based NLP technology to monitor group session activities, track participant outcomes, detect psychosocial concerns in real time, and respond to these concerns automatically [11]. Our AI-based co-facilitator (AICF) system was developed to (1) identify participants at risk for increased emotional distress; (2) monitor in-session engagement and group cohesion levels, providing real-time alerts for the therapist; (3) generate postsession participant profiles that visualize individual emotional trajectories and psychosocial concerns; and (4) automatically suggest tailored online resources to participants based on their messages and participant profiles. Thus, the AICF personalizes support without adding burden to the patient or the therapist. Further, the application of medical resource recommender systems within the AICF can enhance individualized patient access to quality-verified resources that are tailored to the unique needs of patients. This study will report on the training process and performance of the AICF recommender system.

There are numerous applications of AI systems for health care delivery, including treatment recommendations, health education, and symptom management for patient populations [12-15]. A medical information search engine and recommender system called personal health information recommender (PHIR) provides personalized information based on individual patient profiles [13]. PHIR has a knowledge base of 855 online resources, which are registered by experts in the cancer domain. This knowledge base allows PHIR to tailor resources based on the user’s medical conditions and user ratings on the resource selection history, and to perform similarity matching. PHIR incorporated qualitative feedback from physicians and patients to improve its performance and has shown promising results. Additionally, Vik [12] is a conversational agent equipped with intent classification and entity recognition to provide personalized text messages in response to common questions about medical conditions. The results of a blinded randomized controlled study of 142 breast cancer patients demonstrated noninferiority in user-rated quality between answers provided by Vik and those provided by a physician [12].

Although these AI applications were rated by users before deployment, the actual outputs of these recommender systems have seldom undergone rigorous testing or evaluation by human medical experts. More studies are needed to demonstrate the efficacy of health care recommender systems, particularly for supportive care in cancer [16].

Platform and Training Data Set

de Souza Institute offers CancerChatCanada (CCC) that has national, professionally led, synchronous, and text-based OSGs for cancer patients and caregivers in collaboration with 6 provincial agencies in Canada. OSGs vary in length, aims, and group intervention models. All groups are manually based and consist of 6 to 8 sessions. Patient participants were recruited through CCC as well as the webpage and social media accounts (Facebook and Twitter) of de Souza Institute. Patients had to be diagnosed with cancer and able to speak English to be included. The exclusion criterion was the presence of distress needing immediate psychological care. Group sessions built on each other, with each session focusing on a specific theme. In sessions, therapists facilitate discussions based on weekly readings, address concerns, attend to the emotional needs of the members as they emerge, and employ group therapeutic factors that promote a continuous sense of mutual support among 6 to 10 members [17]. The OSG typically employs self-management skills that can empower participants as suggested by the Chronic Care Model [18]. The model posits that through empowering patients with self-management knowledge and resources, patients will become informed and engaged as active participants of care, contributing a collaborative partnership with the health care team toward improved outcomes [18]. OSG therapists routinely recommend additional information postsession that provides education around diagnoses and various coping practices, such as mindfulness and positive psychology interventions, to enhance self-management support. To date, CCC therapists have curated about 37 online resources and webpages that cover a variety of topics on cancer self-management, including physical and psychological symptoms, diagnostics, and treatment options (surgery, hormonal therapy, biotherapy, and chemotherapy), as well as caregiving issues, such as loss and bereavement, for all cancer patients. Additionally, there is a body of information on advanced or metastatic diseases and their diagnosis; management of symptoms, such as pain, constipation, diarrhea, anxiety, and depression; end-of-life discussions, such as advanced care planning; concerns faced by young people; and lifestyle guides on food safety and exercise. To enhance the virtual care system, we designed the AICF to identify psychosocial concerns and automatically suggest the most relevant online resources based on in-session conversations.

Ethics Approval

This study has been approved by the University Health Network Research Ethics Board (CAPCR Study ID 18-5354). Participants provided informed consent before signing up for the OSG.

NLP-Based AICF Algorithm

The AICF [11] was developed using an NLP-based approach with customization capabilities (Figure 1). First, a corpus of CCC chat sessions (approximately 80,000 messages) was used to train the AICF using word2vec, a word embedding model [19]. This model enabled creating a vector representation for each word in the corpus, thus positioning semantically similar expressions in closer proximity. Second, a team of therapists provided a list of common psychosocial concern keywords (Figure 2) that were fed into the trained word2vec model as inputs to generate semantically similar expressions by participants in session transcripts. Next, we queried for semantically similar expressions in the annotated sample. This enriched vocabulary list was used to extract concerns expressed in conversations. This allowed capturing of the terms and phrases related to each concern from patient posts. Third, once the concerns were identified, a concern-response matrix was used to match the best-suited resources for the patient. Finally, individual attributes were used to score the list of clinical resources to create the most appropriate recommendations. These attributes included age, cancer type, patient type (eg, caregiver status), symptoms of depression and anxiety, and engagement level in the group. This resulted in highly customized recommendations that best suited each patient (Figure 1).

Figure 1. Overview of the artificial intelligence–based co-facilitator recommender system framework. CCC: CancerChatCanada.
View this figure

Psychosocial Concern Identification and Resource Database Evaluation

The team developed a literature-based list of psychological concerns relevant to cancer patients (Multimedia Appendix 1) and organized them into a taxonomy that formed the basis for AICF resource mapping (Figure 2) [20,21].

We reviewed 37 online resources curated by CCC therapists. Each resource was evaluated on a set of parameters adapted from the SQuaRE-Aligned Portal Data Quality Model (SPDQM), a model for website content quality evaluation [22,23]. This method aligns with the International Organization for Standardization (ISO) standard for software and data quality [24]. The quality parameters used are as follows: (1) accessibility, (2) understandability, (3) relevancy, (4) validity, and (5) attractiveness and readability (Multimedia Appendix 2).

Each online resource was rated on a Likert scale from 1 to 3 for the parameters listed above (1, poor quality: the resource should not be recommended; 2, moderate quality: the resource should be recommended to the patients with specific concerns or requests; 3, high quality: the resource would be recommended).

Only resources of moderate to high quality were included in the final list of resources available for AICF’s recommendation, and included resources were paired with the most appropriate psychosocial concerns (Table 1). Each resource was rated twice by 2 evaluators (BP and RH) who were blinded to each other’s rating. Consensus was reached through discussion with a third evaluator (YWL) to resolve discrepancies.

Figure 2. Taxonomy of the common psychosocial challenges of cancer patients. Patient concerns identified by the artificial intelligence–based co-facilitator were scored by a team of medical students and clinical experts based on a taxonomy created using their domain expertise. Sx: symptoms.
View this figure

AICF Performance Evaluation

The AICF was applied to the chat history of new OSGs, and outputs were scored by 2 medical students (BP and RH) using a confusion matrix. Recall, precision, and F1 score were used as evaluation metrics [25]. F1 score is the harmonic mean of precision and recall, which takes both false positives and false negatives into account to produce a single measure of performance.

Using the established concern domains (Table 1), the team assessed whether the AICF system (1) correctly identified each output instance (true positive), (2) incorrectly identified an output instance (false positive), (3) correctly identified the lack of an output (true negative), or (4) missed the concern in a statement (false negative). All false-positive and false-negative recommendations were analyzed for their underlying reasons and addressed to improve the AICF in subsequent rounds.

Given that the AICF was designed to read deidentified data sentence by sentence, the human raters were lenient regarding true-negative outputs that may have potentially indicated a concern, but the subject and context of how the concern applies remained ambiguous. The raters would rate “true negative” on the following phrase example: “Yes, my social worker tells me that all I can do is listen and be there for him. But that's really hard to do.” This phase may be interpreted as the struggles of a caregiver or a patient having difficulties coping; the role of the support group member and subject of concern remains ambiguous, and accurate resources cannot be recommended without additional context. Likewise, the rater rated “true negative” on the following phrase: “most don’t want to feel bad, and they can say very heartless things.” This phase may be interpreted as a support group member sharing personal feelings or as an observation that was used to connect and empathize with other members in the chat. Although flagging such ambiguous phrases may increase the sensitivity of the AICF, given that the objective of the AICF is to provide appropriate resources while avoiding information overload, we were lenient with negative outputs that had such ambiguity.

The evaluation results were used to retrain the model, while linguistic rules, part-of-speech tagging, and filtering based on the patient profile were applied to handle exceptions such as negations, past tense, and idioms of expression. Evaluators’ feedback using their domain expertise was used to improve AICF’s performance over the evaluation rounds until it achieved F1 >0.80 before deployment in real-time OSG sessions for beta testing [26].

Participants received an email containing the AICF-tailored recommendations postsession. Users evaluated AICF’s recommendations. Automatically generated emails asked the current participants to further evaluate the system. The users judged each recommended resource on usefulness by answering the following question: “Our system has recommended some resources for you based on the last chat session. Please let us know if the links are helpful or not by clicking on the Useful or Not Useful button below.” We also recorded the number of clicks on the recommended resources. Participant characteristics are presented in Table 2.

Table 1. The finalized concern-resource matrix.
ConcernsType of resource

WebsitePhone lineLearning modulesOnline support groupPDFApps/gamesVideosMindfulness
Newly diagnosedPatient and Caregiver
Cancer Connection
Newly Diagnosed
Cancer support helplineMyGrief.caCancer ConnectionaNucare Manual
Anxiety & depressionWorried, Scared or Anxious
Sadness and Depression
Cancer support helplineMyGrief.caSleeping Well Manual
Nucare Manual
Distress & intense emotionsManaging StressCancer support helplineMyGrief.caNucare Manual
Grief & lossLoss and GriefCancer support helplineMyGrief.caLiving My CultureNucare Manual
Isolation & lonelinessCancer support helplineCancer ConnectionNucare Manual
COVIDCOVID-19 and cancerNucare Manual
Finances & employmentCancer and WorkReturning to Work
Caregiver supportCaregiver helplineCancer Support Community
Support for familiesFamily SupportCancer in my family
Symptom management: InsomniaSleeping Well Manual
Symptom management: PainPain and treatment side effects
Sexual healthSexual Health
Symptom management: GeneralSymptom Management
Adolescents & young adultsYoung Adult Cancer
Cancer Fight Club
Young Adult Cancer

aNot available.

Table 2. Participant characteristics.
CharacteristicValue (N=48), n (%)

Female43 (90)

Male4 (8)

Unknown1 (2)
Age group (years)

18-240 (0)

25-343 (6)

35-448 (17)

45-5410 (21)

55-6418 (37)

65+9 (19)

British Columbia18 (37)

Ontario14 (29)

Alberta7 (15)

Other provinces9 (19)
Type of cancer

Breast24 (50)

Gynecological3 (6)

Colorectal5 (10)

Head and neck3 (6)

Other cancers12 (25)

Unknown1 (2)
Treatment status

Active treatment8 (17)

Posttreatment22 (46)

Other18 (37)

A total of 35,600 outputs from the AICF on the CCC chat history were extracted over 3 evaluation rounds. The months of the data collected were February 2020, April 2020, and June 2020. A random sample of 20% unique statements with AICF’s decision outputs (n=7190) was evaluated by human raters using a confusion matrix. Example phrases from each category of the matrix are provided in Multimedia Appendix 1.

Among false negatives, the AICF failed to recognize culturally specific idioms of concern, which was reflected in the high number of errors. For example, the AICF failed to recognize the phrases “heart feels heavy” or “want to run away” as distress, “exhaustion” as fatigue, and “HER2” as breast cancer. Keywords in false-negative outputs were identified by human evaluators and used to retrain the AICF algorithm for improvement. As a result, the second and third rounds of evaluation added 75 and 17 new terms, respectively, to the AICF concern bank. This adjustment improved the false-negative rate from 54.8% (69/126) in the first round to 30.8% (16/52) in the second round and 6.9% (2/29) in the third round (Table 3).

Table 3. Classification accuracy.
VariableRound 1 (N=4774), nRound 2 (N=1195), nRound 3 (N=1221), n

True positive8485106

True negative456410581086
Total inaccurate1265229

False negative69162

False positive573627

Phrase ambiguity2895

Reference to future/past1973

Reference to others41013

Offering opinion6106

False-positive outputs were classified into 1 of the following 4 subcategories: (1) Phrase ambiguity: there was insufficient information in the statement to fully assess whether a key concern was present; (2) Reference to the future or past: the statement maker was sharing a possible future or past event with other group members; (3) Reference to others: the statement maker refers to a concern pertaining to a person other than themselves; (4) Offering an opinion: the statement maker is offering their personal opinion or experience regarding a concern mentioned by another group member.

False positives were addressed by additional tagging techniques tailored for each underlying reason. The details are elaborated in the Discussion. Although the rate of false positives increased over 3 evaluation rounds (Table 3), this was most likely due to an increased sensitivity of the recommender system arising from the expanded vocabulary that was applied to address the false negatives, and the net result of these adjustments was an improvement in the F1 score from 0.571 in round 1 to 0.766 in round 2 and 0.880 in round 3 (Table 4).

Figure 3 illustrates the patient experiences with the AICF recommender system. The recommender system was tested in a convenient sample of 5 OSGs, reaching 48 participants. Each participant was recommended an average of 11.3 unique resources, ranging from 2 to 40. Twenty-five (52%) of these participants clicked at least one of the recommended resources. These 25 participants viewed an average of 4.4 (39.1%) tailored resources. Among the participants who viewed resources, 19 (76%) rated them as “useful” (Figure 3).

Table 4. Precision, recall, and F1 score following each round of artificial intelligence–based co-facilitator evaluation.
RoundPrecisionRecallF1 score
Figure 3. Patient experience with the artificial intelligence–based co-facilitator (AICF) recommender system. CCC: CancerChatCanada.
View this figure

Principal Findings

This study evaluated AICF’s performance in identifying concerns and recommending resources for cancer patients based on transcripts from OSGs. The large amount of available information online can be overwhelming for resource seekers, especially for those who are affected by cancer. The aim of the AICF is to recommend high-quality resources that are tailored to concerns identified on the basis of each patient’s OSG chat history. A recommendation system based on patients’ needs expressed in the group discussion can potentially reduce the burden on patients to find the correct information and the burden on online therapists who need to respond to multiple participants simultaneously. The preliminary results show that the initial performance was low, indicated by an F1 score of 0.571, although accuracy was high (97.4%). For subsequent evaluation rounds, the AICF was retrained on the basis of feedback from human evaluators, which improved the performance to an F1 score of 0.880 by the third round of evaluation. These results demonstrate that the AICF displays sufficient accuracy in identifying concerns expressed by OSG participants and recommending relevant resources that can help to increase tangible support and service quality without incurring increased workload for therapists. Nineteen (76%) patients who viewed the AICF-recommended resources found them useful.

The AICF is a one-of-a-kind recommender system running behind the scenes of an OSG service without imposing on the therapist or participants. To date, there have been very few studies adopting a human expert in their system validation process. Compared to previous recommender systems, such as PHIR and Vik [27], the AICF adopted a human evaluator feedback loop and exhibited high performance and enhanced personalized support. The AICF performed a robust human evaluation on over 7000 outputs to produce values for accuracy, recall, and F1 scores. The AICF is unique in that it aims to provide automatic detection of psychosocial concerns and delivery of tailored resources for self-management. This technology augments therapist-led OSG sessions, while the other systems relied on patients actively seeking resources and using a search engine for resource delivery.

Among the recommender systems designed for health care delivery, the AICF is highly comparable to a conversational agent, Vik [27], based on common medical questions and physician answers. Vik uses intent classification and entity recognition to process user input texts. Intent classification identifies keywords from the user’s textual inputs and classifies them into one of the predetermined question categories. Entity detection identifies names or titles in the user’s inputs and classifies them into predefined categories. However, the AICF is embedded in an OSG and uses a statistical and rule-based approach with word embeddings, in which a subset of relevant keywords is extracted as intent, serving as inputs to the recommender. Furthermore, the AICF differs from Vik in that we incorporated user profile information, such as type of cancer, age, engagement level, anxiety and depression symptomatology, and caregiver status, to produce highly tailored recommendations. Another major difference is the fact that Vik was trained with a database consisting of questions asked by the users to their health professionals, while the AICF was trained on chat history data consisting of human-to-human text-based conversations in OSG format. The training data allowed the AICF to understand more diverse psychosocial concerns, but they are more complex to process.

The AICF showed high accuracy (97.4%) in the initial assessment, and it was stable over evaluation rounds. False negatives were reduced by expanding AICF’s vocabulary bank to include key terms that had been missed, resulting in a greater than 8-fold reduction in the false-negative rate between round 1 and round 3. However, continuous monitoring and retraining by feedback from human raters will be required for the AICF to be sensitive to idiom use in different contexts and scenarios. Future work should explore the use of a language model [28] to detect the idiomatic and metaphorical parts in sentences.

For false positives, the AICF identified concerns that were deemed incorrect by the human evaluators. These were categorized into the following 4 subcategories: (1) phrase ambiguity, (2) reference to the future or past, (3) reference to others, and (4) offering an opinion.

Phrase Ambiguity

There was insufficient information in the statement to fully assess whether a key concern was present. The text was primarily characterized by short messages in which contextual information was missing. One or more keywords of psychosocial concern were present and were picked up by the AICF in the absence of contextual information. Phrase ambiguity was complicated by using a deidentified data set. All identifiable personal health information, including user handle names, hospital names, and doctor names, had been removed in accordance with the guidelines from the research ethics board. This often leads to disjointed data in which phrases are fragmented. Currently, the AICF is unable to link conversations between specific participants, resulting in lost information regarding who is replying to whom or which statements the speaker agreed or disagreed with. We also removed the therapist’s statements to minimize the contamination of group outcomes arising from the therapist’s validation of the group discussion. This resulted in the loss of contextual information for the AICF, contributing to the number of false positives identified. Future work should look into different deidentification methods that can better protect the linkage of conversations without compromising participant confidentiality.

Reference to the Future or Past

The statement maker was sharing his/her past experiences or future events that had not yet happened. For example, participants shared an experience, triggering the AICF to identify a concern and recommend resources. However, the phrases indicated to the raters that the participant was no longer actively dealing with the identified concern. To address these themes in false-positive outputs, a speech tagging technique was added to the algorithm after the second round of validation to detect the use of past and future tenses. Future work will explore other word embedding models, such as sense2vec, to improve performance further.

Reference to Others

The AICF identified concerns and recommended resources to participants when participants were in fact referencing the stories of a third party. The addition of a linguistic rule to detect story-telling, such as the use of third person pronouns, may help improve false-positive outputs. Once more chat transcripts become available, it will be a feasible adjustment to further improve the precision of the AICF.

Offering an Opinion

The statement maker was offering his/her personal opinion or experience regarding a concern mentioned by another group member. Future studies can explore modeling the relationship between messages to recognize the conversation thread.


Apart from the limitations identified above, which are common as AI continues to improve itself, the 37 curated resources included in our recommender system can be seen as a relatively small set of information support resources in cancer care. These resources were selected for their relevance to assist OSG participants in dealing with the psychosocial challenges of living with different cancers. However, such resources could also be seen as too generic by participants and insufficient to meet their needs for a specific cancer. This may partially explain the fact that only 52% of the participants accessed a recommended resource. Additionally, the resources included were rated by medical professionals; however, there is obvious merit to additional evaluation by a more neutral party whose health literacy is more representative of the general public and patients who would utilize the AICF. The patient population included in this study was also likely more technologically savvy compared to the general public given that they had to be competent in maneuvering online webpages and social media to sign up for the OSG. While this may not be representative of all cancer patients, with the continued rise of internet usage and the strong need for additional remote support options with the COVID-19 pandemic, we predict that the population this system is geared toward will continue to become more representative of the general cancer patient population over the years. Next steps will involve conducting focus groups with OSG participants to explore their opinions on the AICF and expanding the resource rating team to include diverse backgrounds and perspectives in the rating process. Future work will expand the psychosocial resources to include those for particular cancers and develop NLP to recognize specific cancer types. Future work should also assess ethnicity/cultural parameters related to the AICF system.


Owing to increased mental health care demands and barriers for accessing in-person care, virtual care has become paramount in the provision of supportive care. We have embedded the AICF within OSGs to increase personalized support and expand patient self-management capacities by recommending credible online resources. All these goals can be achieved without additional work from therapists. Future projects include user focus groups, development of cancer-specific recommender systems, expansion to additional languages, and ultimately randomized controlled trials to inform effectiveness and ensure further development of policies, such as mandating AI-enhanced OSGs as the first line of patient education to build self-management capacities for cancer and chronic diseases. Eventually, this line of research will inform our health system on the use of AI for future personalized supportive care delivery.


This research was funded by the Ontario Institute for Cancer Research Cancer Care Ontario Health Services Research Network.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Themes, keywords, and examples of artificial intelligence–based co-facilitator outputs.

PDF File (Adobe PDF File), 42 KB

Multimedia Appendix 2

Resource evaluation tool.

PDF File (Adobe PDF File), 119 KB

  1. Xia W, Jiang H, Di H, Feng J, Meng X, Xu M, et al. Association between self-reported depression and risk of all-cause mortality and cause-specific mortality. J Affect Disord 2022 Feb 15;299:353-358. [CrossRef] [Medline]
  2. Katon WJ. Epidemiology and treatment of depression in patients with chronic medical illness. Dialogues Clin Neurosci 2011;13(1):7-23 [FREE Full text] [Medline]
  3. Rosenstein DL. Depression and end-of-life care for patients with cancer. Dialogues Clin Neurosci 2011;13(1):101-108 [FREE Full text] [Medline]
  4. Chen Q, Liang M, Li Y, Guo J, Fei D, Wang L, et al. Mental health care for medical staff in China during the COVID-19 outbreak. The Lancet Psychiatry 2020 Apr;7(4):e15-e16. [CrossRef]
  5. Finfgeld DL. Therapeutic groups online: the good, the bad, and the unknown. Issues Ment Health Nurs 2000 Jul 09;21(3):241-255. [CrossRef] [Medline]
  6. Gratzer D, Goldbloom D. Making evidence-based psychotherapy more accessible in Canada. Can J Psychiatry 2016 Oct 10;61(10):618-623 [FREE Full text] [CrossRef] [Medline]
  7. Male DA, Fergus KD, Stephen JE. Professional online support group facilitators: Guarantors of maximal group utility. International Journal of Group Psychotherapy 2016 Oct 25;67(3):314-336. [CrossRef]
  8. McCaughan E, Parahoo K, Hueter I, Northouse L, Bradbury I. Online support groups for women with breast cancer. Cochrane Database Syst Rev 2017 Mar 10;3:CD011652 [FREE Full text] [CrossRef] [Medline]
  9. van Uden-Kraan CF, Drossaert CHC, Taal E, Shaw BR, Seydel ER, van de Laar MAFJ. Empowering processes and outcomes of participation in online support groups for patients with breast cancer, arthritis, or fibromyalgia. Qual Health Res 2008 Mar;18(3):405-417. [CrossRef] [Medline]
  10. Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc 2015 Jan;22(1):43-50 [FREE Full text] [CrossRef] [Medline]
  11. Leung YW, Wouterloot E, Adikari A, Hirst G, de Silva D, Wong J, et al. Natural language processing-based virtual cofacilitator for online cancer support groups: Protocol for an algorithm development and validation study. JMIR Res Protoc 2021 Jan 07;10(1):e21453 [FREE Full text] [CrossRef] [Medline]
  12. Bibault J, Chaix B, Guillemassé A, Cousin S, Escande A, Perrin M, et al. A chatbot versus physicians to provide information for patients with breast cancer: Blind, randomized controlled noninferiority trial. J Med Internet Res 2019 Nov 27;21(11):e15787 [FREE Full text] [CrossRef] [Medline]
  13. Iatraki G, Kondylakis H, Koumakis L, Chatzimina M, Kazantzaki E, Marias K, et al. Personal Health Information Recommender: implementing a tool for the empowerment of cancer patients. Ecancermedicalscience 2018 Jul 11;12:851. [CrossRef] [Medline]
  14. Katzman JL, Shaham U, Cloninger A, Bates J, Jiang T, Kluger Y. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Med Res Methodol 2018 Feb 26;18(1):24 [FREE Full text] [CrossRef] [Medline]
  15. Taban M. A recommender system for breast cancer patients. Memorial University Research Repository. 2014.   URL: [accessed 2022-06-07]
  16. Wiesner M, Pfeifer D. Health recommender systems: concepts, requirements, technical basics and challenges. Int J Environ Res Public Health 2014 Mar 03;11(3):2580-2607 [FREE Full text] [CrossRef] [Medline]
  17. Yalom ID. The Theory and Practice of Group Psychotherapy, 5th Edition. New York, NY: Basic Books; 2008.
  18. Wagner EH. Chronic disease management: what will it take to improve care for chronic illness? Eff Clin Pract 1998;1(1):2-4. [Medline]
  19. Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. arXiv. 2013.   URL: [accessed 2021-03-12]
  20. Casellas-Grau A, Font A, Vives J. Positive psychology interventions in breast cancer. A systematic review. Psychooncology 2014 Jan 29;23(1):9-19. [CrossRef] [Medline]
  21. Ludolph P, Kunzler A, Stoffers-Winterling J, Helmreich I, Lieb K. Interventions to promote resilience in cancer patients. Dtsch Arztebl Int 2019 Dec 23;51-52(51-52):865-872 [FREE Full text] [CrossRef] [Medline]
  22. Moraga C, Moraga M, Calero C, Caro A. SQuaRE-Aligned Data Quality Model for Web Portals. 2009 Presented at: Ninth International Conference on Quality Software; August 24-25, 2009; Jeju, Korea (South). [CrossRef]
  23. Rocha Á. Framework for a global quality evaluation of a website. Online Information Review 2012;36(3):374-382. [CrossRef]
  24. ISO/IEC 25012:2008 Software engineering — Software product Quality Requirements and Evaluation (SQuaRE) — Data quality model. ISO.   URL: [accessed 2022-06-07]
  25. Van Rijsbergen CJ, Van Rijsbergen CJK. Information Retrieval. London: Butterworths; 1979.
  26. Niu Y, Zhu X, Li J, Hirst G. Analysis of polarity information in medical text. AMIA Annu Symp Proc 2005:570-574 [FREE Full text] [Medline]
  27. Chaix B, Guillemassé A, Nectoux P, Delamon G, Brouard B. Vik: A chatbot to support patients with chronic diseases. Health 2020;12(07):804-810. [CrossRef]
  28. Devlin J, Chang M, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. 2019.   URL: [accessed 2021-05-23]

AI: artificial intelligence
AICF: artificial intelligence–based co-facilitator
CCC: CancerChatCanada
NLP: natural language processing
OSG: online support group
PHIR: personal health information recommender

Edited by A Mavragani; submitted 22.12.21; peer-reviewed by A Buttazzoni, L Guo; comments to author 19.04.22; revised version received 14.05.22; accepted 29.05.22; published 29.07.22


©Yvonne W Leung, Bomi Park, Rachel Heo, Achini Adikari, Suja Chackochan, Jiahui Wong, Elyse Alie, Mathew Gancarz, Martyna Kacala, Graeme Hirst, Daswin de Silva, Leon French, Jacqueline Bender, Faye Mishna, David Gratzer, Damminda Alahakoon, Mary Jane Esplen. Originally published in JMIR Cancer (, 29.07.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.