de Souza Institute offers CancerChatCanada (CCC) that has national, professionally led, synchronous, and text-based OSGs for cancer patients and caregivers in collaboration with 6 provincial agencies in Canada. OSGs vary in length, aims, and group intervention models. All groups are manually based and consist of 6 to 8 sessions. Patient participants were recruited through CCC as well as the webpage and social media accounts (Facebook and Twitter) of de Souza Institute. Patients had to be diagnosed with cancer and able to speak English to be included. The exclusion criterion was the presence of distress needing immediate psychological care. Group sessions built on each other, with each session focusing on a specific theme. In sessions, therapists facilitate discussions based on weekly readings, address concerns, attend to the emotional needs of the members as they emerge, and employ group therapeutic factors that promote a continuous sense of mutual support among 6 to 10 members [17]. The OSG typically employs self-management skills that can empower participants as suggested by the Chronic Care Model [18]. The model posits that through empowering patients with self-management knowledge and resources, patients will become informed and engaged as active participants of care, contributing a collaborative partnership with the health care team toward improved outcomes [18]. OSG therapists routinely recommend additional information postsession that provides education around diagnoses and various coping practices, such as mindfulness and positive psychology interventions, to enhance self-management support. To date, CCC therapists have curated about 37 online resources and webpages that cover a variety of topics on cancer self-management, including physical and psychological symptoms, diagnostics, and treatment options (surgery, hormonal therapy, biotherapy, and chemotherapy), as well as caregiving issues, such as loss and bereavement, for all cancer patients. Additionally, there is a body of information on advanced or metastatic diseases and their diagnosis; management of symptoms, such as pain, constipation, diarrhea, anxiety, and depression; end-of-life discussions, such as advanced care planning; concerns faced by young people; and lifestyle guides on food safety and exercise. To enhance the virtual care system, we designed the AICF to identify psychosocial concerns and automatically suggest the most relevant online resources based on in-session conversations.

This study has been approved by the University Health Network Research Ethics Board (CAPCR Study ID 18-5354). Participants provided informed consent before signing up for the OSG.

The AICF [11] was developed using an NLP-based approach with customization capabilities (Figure 1). First, a corpus of CCC chat sessions (approximately 80,000 messages) was used to train the AICF using word2vec, a word embedding model [19]. This model enabled creating a vector representation for each word in the corpus, thus positioning semantically similar expressions in closer proximity. Second, a team of therapists provided a list of common psychosocial concern keywords (Figure 2) that were fed into the trained word2vec model as inputs to generate semantically similar expressions by participants in session transcripts. Next, we queried for semantically similar expressions in the annotated sample. This enriched vocabulary list was used to extract concerns expressed in conversations. This allowed capturing of the terms and phrases related to each concern from patient posts. Third, once the concerns were identified, a concern-response matrix was used to match the best-suited resources for the patient. Finally, individual attributes were used to score the list of clinical resources to create the most appropriate recommendations. These attributes included age, cancer type, patient type (eg, caregiver status), symptoms of depression and anxiety, and engagement level in the group. This resulted in highly customized recommendations that best suited each patient (Figure 1).

The team developed a literature-based list of psychological concerns relevant to cancer patients (Multimedia Appendix 1) and organized them into a taxonomy that formed the basis for AICF resource mapping (Figure 2) [20,21].

We reviewed 37 online resources curated by CCC therapists. Each resource was evaluated on a set of parameters adapted from the SQuaRE-Aligned Portal Data Quality Model (SPDQM), a model for website content quality evaluation [22,23]. This method aligns with the International Organization for Standardization (ISO) standard for software and data quality [24]. The quality parameters used are as follows: (1) accessibility, (2) understandability, (3) relevancy, (4) validity, and (5) attractiveness and readability (Multimedia Appendix 2).

Each online resource was rated on a Likert scale from 1 to 3 for the parameters listed above (1, poor quality: the resource should not be recommended; 2, moderate quality: the resource should be recommended to the patients with specific concerns or requests; 3, high quality: the resource would be recommended).

Only resources of moderate to high quality were included in the final list of resources available for AICF’s recommendation, and included resources were paired with the most appropriate psychosocial concerns (Table 1). Each resource was rated twice by 2 evaluators (BP and RH) who were blinded to each other’s rating. Consensus was reached through discussion with a third evaluator (YWL) to resolve discrepancies.

The AICF was applied to the chat history of new OSGs, and outputs were scored by 2 medical students (BP and RH) using a confusion matrix. Recall, precision, and F1 score were used as evaluation metrics [25]. F1 score is the harmonic mean of precision and recall, which takes both false positives and false negatives into account to produce a single measure of performance.

Using the established concern domains (Table 1), the team assessed whether the AICF system (1) correctly identified each output instance (true positive), (2) incorrectly identified an output instance (false positive), (3) correctly identified the lack of an output (true negative), or (4) missed the concern in a statement (false negative). All false-positive and false-negative recommendations were analyzed for their underlying reasons and addressed to improve the AICF in subsequent rounds.

Given that the AICF was designed to read deidentified data sentence by sentence, the human raters were lenient regarding true-negative outputs that may have potentially indicated a concern, but the subject and context of how the concern applies remained ambiguous. The raters would rate “true negative” on the following phrase example: “Yes, my social worker tells me that all I can do is listen and be there for him. But that's really hard to do.” This phase may be interpreted as the struggles of a caregiver or a patient having difficulties coping; the role of the support group member and subject of concern remains ambiguous, and accurate resources cannot be recommended without additional context. Likewise, the rater rated “true negative” on the following phrase: “most don’t want to feel bad, and they can say very heartless things.” This phase may be interpreted as a support group member sharing personal feelings or as an observation that was used to connect and empathize with other members in the chat. Although flagging such ambiguous phrases may increase the sensitivity of the AICF, given that the objective of the AICF is to provide appropriate resources while avoiding information overload, we were lenient with negative outputs that had such ambiguity.

The evaluation results were used to retrain the model, while linguistic rules, part-of-speech tagging, and filtering based on the patient profile were applied to handle exceptions such as negations, past tense, and idioms of expression. Evaluators’ feedback using their domain expertise was used to improve AICF’s performance over the evaluation rounds until it achieved F1 >0.80 before deployment in real-time OSG sessions for beta testing [26].

Participants received an email containing the AICF-tailored recommendations postsession. Users evaluated AICF’s recommendations. Automatically generated emails asked the current participants to further evaluate the system. The users judged each recommended resource on usefulness by answering the following question: “Our system has recommended some resources for you based on the last chat session. Please let us know if the links are helpful or not by clicking on the Useful or Not Useful button below.” We also recorded the number of clicks on the recommended resources. Participant characteristics are presented in Table 2.

This study evaluated AICF’s performance in identifying concerns and recommending resources for cancer patients based on transcripts from OSGs. The large amount of available information online can be overwhelming for resource seekers, especially for those who are affected by cancer. The aim of the AICF is to recommend high-quality resources that are tailored to concerns identified on the basis of each patient’s OSG chat history. A recommendation system based on patients’ needs expressed in the group discussion can potentially reduce the burden on patients to find the correct information and the burden on online therapists who need to respond to multiple participants simultaneously. The preliminary results show that the initial performance was low, indicated by an F1 score of 0.571, although accuracy was high (97.4%). For subsequent evaluation rounds, the AICF was retrained on the basis of feedback from human evaluators, which improved the performance to an F1 score of 0.880 by the third round of evaluation. These results demonstrate that the AICF displays sufficient accuracy in identifying concerns expressed by OSG participants and recommending relevant resources that can help to increase tangible support and service quality without incurring increased workload for therapists. Nineteen (76%) patients who viewed the AICF-recommended resources found them useful.

The AICF is a one-of-a-kind recommender system running behind the scenes of an OSG service without imposing on the therapist or participants. To date, there have been very few studies adopting a human expert in their system validation process. Compared to previous recommender systems, such as PHIR and Vik [27], the AICF adopted a human evaluator feedback loop and exhibited high performance and enhanced personalized support. The AICF performed a robust human evaluation on over 7000 outputs to produce values for accuracy, recall, and F1 scores. The AICF is unique in that it aims to provide automatic detection of psychosocial concerns and delivery of tailored resources for self-management. This technology augments therapist-led OSG sessions, while the other systems relied on patients actively seeking resources and using a search engine for resource delivery.

Among the recommender systems designed for health care delivery, the AICF is highly comparable to a conversational agent, Vik [27], based on common medical questions and physician answers. Vik uses intent classification and entity recognition to process user input texts. Intent classification identifies keywords from the user’s textual inputs and classifies them into one of the predetermined question categories. Entity detection identifies names or titles in the user’s inputs and classifies them into predefined categories. However, the AICF is embedded in an OSG and uses a statistical and rule-based approach with word embeddings, in which a subset of relevant keywords is extracted as intent, serving as inputs to the recommender. Furthermore, the AICF differs from Vik in that we incorporated user profile information, such as type of cancer, age, engagement level, anxiety and depression symptomatology, and caregiver status, to produce highly tailored recommendations. Another major difference is the fact that Vik was trained with a database consisting of questions asked by the users to their health professionals, while the AICF was trained on chat history data consisting of human-to-human text-based conversations in OSG format. The training data allowed the AICF to understand more diverse psychosocial concerns, but they are more complex to process.

The AICF showed high accuracy (97.4%) in the initial assessment, and it was stable over evaluation rounds. False negatives were reduced by expanding AICF’s vocabulary bank to include key terms that had been missed, resulting in a greater than 8-fold reduction in the false-negative rate between round 1 and round 3. However, continuous monitoring and retraining by feedback from human raters will be required for the AICF to be sensitive to idiom use in different contexts and scenarios. Future work should explore the use of a language model [28] to detect the idiomatic and metaphorical parts in sentences.

For false positives, the AICF identified concerns that were deemed incorrect by the human evaluators. These were categorized into the following 4 subcategories: (1) phrase ambiguity, (2) reference to the future or past, (3) reference to others, and (4) offering an opinion.

Apart from the limitations identified above, which are common as AI continues to improve itself, the 37 curated resources included in our recommender system can be seen as a relatively small set of information support resources in cancer care. These resources were selected for their relevance to assist OSG participants in dealing with the psychosocial challenges of living with different cancers. However, such resources could also be seen as too generic by participants and insufficient to meet their needs for a specific cancer. This may partially explain the fact that only 52% of the participants accessed a recommended resource. Additionally, the resources included were rated by medical professionals; however, there is obvious merit to additional evaluation by a more neutral party whose health literacy is more representative of the general public and patients who would utilize the AICF. The patient population included in this study was also likely more technologically savvy compared to the general public given that they had to be competent in maneuvering online webpages and social media to sign up for the OSG. While this may not be representative of all cancer patients, with the continued rise of internet usage and the strong need for additional remote support options with the COVID-19 pandemic, we predict that the population this system is geared toward will continue to become more representative of the general cancer patient population over the years. Next steps will involve conducting focus groups with OSG participants to explore their opinions on the AICF and expanding the resource rating team to include diverse backgrounds and perspectives in the rating process. Future work will expand the psychosocial resources to include those for particular cancers and develop NLP to recognize specific cancer types. Future work should also assess ethnicity/cultural parameters related to the AICF system.

Owing to increased mental health care demands and barriers for accessing in-person care, virtual care has become paramount in the provision of supportive care. We have embedded the AICF within OSGs to increase personalized support and expand patient self-management capacities by recommending credible online resources. All these goals can be achieved without additional work from therapists. Future projects include user focus groups, development of cancer-specific recommender systems, expansion to additional languages, and ultimately randomized controlled trials to inform effectiveness and ensure further development of policies, such as mandating AI-enhanced OSGs as the first line of patient education to build self-management capacities for cancer and chronic diseases. Eventually, this line of research will inform our health system on the use of AI for future personalized supportive care delivery.