Web-based care has become increasingly important in health care delivery as a means to accessibly reduce emotional distress. Online support groups (OSG) offer a convenient solution to those who cannot attend in-person support groups [1-3]. Professionally led OSGs occur in real time with participants engaging with a therapist and other participants in the group. Therapists facilitate the sharing of personal experiences to foster a mutually supportive environment. OSG participants report an increased sense of empowerment and control, as well as improved knowledge about their conditions [4].

Cancer Chat Canada (CCC) offers web-based professionally led, synchronous, text-based support groups to patients with cancer or caregivers across 6 Canadian provinces with a text-based nature allowing for anonymity while reaching people in rural areas. All groups provided via CCC are manual-based, consisting of 8-12 sessions. Each session focuses on a specific theme, homework readings, and web-based activities. Participants exchanged their experiences and ideas through a chatbox on the CCC platform. During sessions, therapists facilitate discussions based on the weekly readings, address issues or concerns, attend to emergent emotional needs of the members, and employ therapeutic techniques that promote a continuous sense of mutual support among the 6-10 members [5].

For group interventions to be effective, therapists encourage authentic emotional expression while effectively monitoring and addressing signals of distress [6]. However, the absence of visual cues, along with the simultaneous entries by multiple participants, can impose challenges for therapists to attend to all participants’ needs during the session [4]. Therapists’ failure to recognize and respond to participants’ expressions of distress can reduce the participants’ perceived level of support, safety, and trust in the group, leading to disengagement and attrition [7].

One way to reduce attrition and improve OSG services is through tracking and monitoring group cohesion [8,9]. A cohesive group experiences a sense of warmth, comfort, acceptance, affiliation, and support from other members they value [5]. Group cohesion is associated with positive participant outcomes, including reductions in distress and improvements in interpersonal functioning [5].

Traditionally, group cohesion is measured by participant self-report instruments, such as the Harvard Community Health Plan Group Cohesiveness Scale[10] and the Group Cohesion Scale Revised [10]. Alternatively, it can be measured by content analysis, where analysts assign ratings to the participants’ statements [11]. While useful, these approaches have limitations of participant recall bias, measurement fatigue in self-reports, and time and cost of labor in post hoc qualitative analyses.

Previous studies demonstrate that a higher frequency of first-person singular pronouns use (ie, I, my), also referred to as “iTalk” or self-referential language, is a linguistic marker of general distress and is associated with negative psychological outcomes such as depression and suicidal behaviors [12-14]. In contrast, collective identity language use (ie, our group, us) was instrumental to group attachment [15]; with greater uses of references to the group as a whole and to other members predicting reduced symptoms of grief [11]. Aside from content analysis, such as Psychodynamic Work, Object Rating System [16], many studies adopted computerized textual analysis systems such as dtSearch [17], Linguistic Inquiry, and Word Count (LIWC) to track levels of cohesion through text [18-21]. In particular, Lieberman et al [20] detected group cohesion by combining LIWC to count the proportion of group referential language use and dtSearch to count words indicative of positive connotations (ie, hope, altruistic, accept, affection) within 10 words of such group referential language in an OSG for patients with Parkinson. However, Alpers et al [19] questioned the software’s ability to process complex communications, suggesting that future studies should develop systems that analyze the context of discourse for real-time analysis.

Given the evidence, group cohesion can be systematically measured by a well-designed computer analytical system. We designed the Artificial Intelligence–based Co-Facilitator (AICF) to contextually identify therapeutic outcomes, including group cohesion from conversations, and produce real-time analytics [22-25]. AICF can track basic emotions, including joy, sadness, anger, trust, fear, anticipation, disgust, surprise, and psychological outcomes such as distress, group cohesion, and hopelessness for each participant in the OSGs [22,23]. AICF extracted emotions from the text by parsing through over 120,000 lines of chat messages from a training data set to multiple levels of granularity: word, phrase, sentence, post, and posts by each user [26]. AICF employed several natural language processing (NLP) techniques, such as Word2Vec [27] and text classification models. Classification models were trained to classify posts containing group cohesion mentions to determine the level of group cohesion in this web-based conversation setting. Each level of extraction served as an input for calibration for the subsequent extraction to increase accuracy [26,28,29]. AICF could, therefore, track and inform facilitators of each participant’s level of cohesive statement use in their posts.

We hypothesized that AICF could detect first-person plural pronoun use (eg, we, our) in OSGs and group-references language use (“we-talk”) as group cohesion, machine learning–based NLP could also identify a broader definition of group cohesion, including expressing gratitude, mutual support, and sense of belonging.

This study is focused on the development of a method to train and evaluate AICF’s ability to detect group cohesion among cancer OSG members.

The steps involved in the training and development of AICF’s cohesion detection is outlined below.

The study protocol including the human participant recruitment method was approved by the University Health Network Research Ethics Board (confirmation number: UHN REB#18-5354). All identifiable information was removed from the quotes in this report. Participants were compensated with a CAD (73.34 USD) gift card upon the completion of the focus group.

AICF, an ensemble of NLP and machine learning algorithms combined with annotation and human scoring, offers a novel way of measuring the group cohesion changes for each group member and alerting the therapist of these changes in real time. This affords therapists the opportunity to allocate their attention and resources for effective facilitation. The objective was to determine whether AICF can detect group cohesion beyond the first-person plural pronunciation use. The findings indicate that it is feasible to measure group cohesion in text-based complex human interactions using AICF. The level of congruency with human scoring suggests that it can be a helpful tool to therapists in improving the group cohesion outcome.

This study has opened an avenue to person-centered and process-outcome research using AI combined with human inputs to improve the quality of care, which otherwise is a labor-intensive research process. Initially, after being trained with 1000 annotated group cohesion statements processed by word embeddings and the domain expertise from therapists, AICF was able to achieve reasonable F₁, precision, and recall scores. Furthermore, training the algorithm using only word embeddings allows AICF to identify the various cohesion themes that emerged, which are consistent with previous research 34]. These themes include expressing support, reassurance, a sense of belonging, trust, deepening emotional disclosure, gratitude, remarking on shared experiences, reflecting on positive aspects of the group, and anticipating future chats. The findings suggest that training AICF to monitor therapeutic responses in web-based care is promising.

When combined with the human scoring examples in the algorithm, as little as 20% of the outputs, AICF obtained a high F₁-score. The human rater detected both false-positive examples (eg, “Just have to find what works for you, I listen to a lot of audible books while I do chores, it's a mental distraction and really helps me”) and false-negative examples (eg, “Thanks so much to all of you, for being in this moment. You've helped me get ready for yet another week.”). These examples contributed to the rule-based algorithms as a second layer of analysis. While precision values remained relatively low in both rounds (0.99 vs 0.98), the recall value improved from 0.52 to 0.70 due to a reduction in false-negative classifications. These increases strongly suggest that a continuous effort to train AICF using human input can lead to a higher level of accuracy in detecting group cohesion.

After running LIWC on a test data set, its performance was evaluated by a human scorer. The precision, recall, and F₁-scores were lower compared to the performance of AICF. Unlike AICF, which is capable of identifying group cohesion expressions and idioms, LIWC is programmed to identify certain keywords. For a false-negative example, LIWC was unable to detect the following quote as an instance of group cohesion due to the lack of the keyword “we”: “I feel like I’ve suddenly inherited a whole group of sisters.” Another instance of an LIWC false positive was that LIWC dictionary categories “family,” “positive emotion,” and “affiliation” falsely detected group cohesion from this quote: “My husband has helped me see that it isn't something I did, or who I am.”

This study successfully trained a machine learning system to detect cohesive statements in contrast to qualitative content analysis, which tends to be onerous and prone to human errors when dealing with large amounts of data [11]. Emerging computer programs such as Discourse Attributes Analysis Program (DAAP) [36] and LIWC [35,37] offer an iterative psycholinguistic approach to coding transcripts of psychotherapy for therapeutic moments [18,21]. For example, DAAP is based on a weighted dictionary that assigns weights to different words instead of solely detecting them as belonging to various categories where all matching words contribute equally to the generated scores. This method allows for greater accuracy in measuring different concepts compared to human coding while processing large amounts of data. However, these weighted dictionary approaches can be limited by a fixed number of instances that can be detected, and only one keyword can be considered in each matching rather than taking contexts into account. Additionally, they do not consider emerging words, phrases, idioms of expressions, word order, negation, and context-dependent factors, as well as their post hoc nature [19]. In this study, the word embedding approach was used to create contextual variables from the keywords to successfully detect a reasonably broad definition of cohesiveness. Thus, work will continue toward improving the accuracy of AICF in upcoming OSG sessions.

AICF is based on a previously trained ensemble called Patient-Reported Information Multidimensional Exploration (PRIME) that was primarily trained on Australian web-based forum data [29]. Thus, Canadians may have used expressions or idioms that were unfamiliar to the original PRIME system and, therefore, not detected (eg, “My head is swirling” to describe feeling overwhelmed or “the clock is ticking” to describe an impending end of life). The local idioms and expressions were handled by the rule-based approach; ideally, AICF would be (re)trained with a large amount of local data in order to capture such idioms and expressions.

Currently, the interactional nature of the statement is not incorporated into AICF, including responses to other members’ or therapists’ statements. Furthermore, AICF cannot consistently distinguish whether participants are speaking about the group or about people outside of the group. When data accumulate, this distinction will become more obvious and refine AICF’s detection ability within the context of an OSG.

The performance of AICF’s group cohesion classification was evaluated in comparison to scores by 2 human experts, whose scoring was guided by the same criteria. However, given the nuanced nature of a group process like cohesion, there was still an element of personal judgment and openness to interpretation in the statements. Finally, emojis were not considered in the algorithm; future studies need to incorporate them as expressions of group cohesion.

ICF has been running in the background on 3 CCC groups and will soon be deployed for beta-testing on 10-12 groups. Participants will be filling out a survey package that includes the psychometrically validated questionnaire that tabs group cohesion for further validation. For algorithm development, sequencing the emotions expressed by each participant will be explored to capture more accurate emotional profiles.

The use of large language models (LLMs), such as ChatGPT, has revolutionized natural language understanding in the field of affective computing. Research suggests that an LLM called ROBERTa [38] has been equipped with emotion knowledge that contains 14 human conceptual attributes of emotions, including 2 affective, 6 appraisals, and 6 basic emotions. Future work will incorporate LLMs into our system to enhance AICF’s ability to detect group cohesion and other significant clinical outcomes. For example, the LLM has already understood the syntactic difference between first-person and third-person pronunciation uses and their contexts. Combining both of these emotional attributes and syntax, we are able to better formulate an equation to calculate the tendency of a writer to be self-focused or other-focused. This will truly improve the accuracy and precision of group cohesion detection.

Lastly, In this study, 5 LIWC dictionary categories were used to capture the concept of group cohesion. Future studies may test whether there is a way that will improve the performance of group cohesion prediction using LIWC by (1) adding more categories, (2) reducing some categories, and (3) adding weighting to each criterion.

AICF will explore ways to measure multiple processes comprising group climate, including the level of participation, expression of emotion, signs of cohesion, avoidance, and therapeutic factors such as conflict, altruism, universality, interpersonal learning input and output, catharsis, identification, self-understanding, and instillation of hope [5,9,31]. If successful, AICF will be applied alongside the mobile health chatbot technology to provide a scalable, automated monitoring and referral system that screens users for specific symptoms, recommends individualized web-based and community resources, tracks each user’s psychological outcomes through, and refers them to local therapists when necessary.

Optimal OSG delivery requires rapid alerts for therapists to effectively monitor markers of positive and negative responses within the group. This study has demonstrated that advanced machine learning algorithms combined with human inputs can reasonably detect the clinically meaningful indicator of group cohesion in OSGs. Future research in utilizing LLMs in AICF could enhance the capabilities in understanding the context, given the capability of creating a highly customized model in a short time. Therefore, AICF has the potential to assist therapists by highlighting issues that are amenable to intervention in real time, which allows therapists to provide greater levels of individualized support.