We conducted an open-label, longitudinal, prospective study between July 12, 2017, and April 7, 2020, at 14 centers in France.

This study was conducted according to the 1975 Declaration of Helsinki, revised in 2008, and the guidelines of the International Conference on Harmonization of Good Clinical Practice in Biomedical Research. The Ouest II national ethics committee in Angers approved the study on November 8, 2016, and the Agence Nationale de Sécurité du Médicament approved it on November 22, 2016 (approval: 2021–A01670–41). All of the patients provided written informed consent, which included the points of analysis, the method of data collection, and the primary and potential secondary statistical analyses. All patient data were anonymized and no financial compensation was provided.

Patients with lymphoma who were aged 18 years or older and had a high risk of relapse were considered eligible for this trial. They could have T-cell lymphoma in the first partial or complete response, Hodgkin lymphoma in the second partial or complete response, or diffuse large B-cell lymphoma in the first partial or complete response with a revised high International Prognostic Index score (≥3) or in the second partial or complete response. Patients who had undergone autologous stem cell transplantation were not excluded. Eastern Cooperative Oncology Group performance status between 0 and 2, an internet connection, and affiliation to the French social security system were required. Patients were recruited during follow-up consultations by the referring physician at each center.

The exclusion criteria were an initial symptom score <7, progression within 3 months of the last treatment, brain or meningeal involvement, history of another cancer treated within 3 years—with the exception of skin cancer (except melanoma) and in situ cervical cancer—pregnancy, breastfeeding, and any psychiatric pathology that may prevent compliance with the protocol.

An initial symptom score was established in the previous Sentinel study. The e-request algorithm was more sensitive for patients who were not very symptomatic at inclusion and had an initial score of less than 7 (by summing scores from 0 to 3 for symptoms concerning cough, dyspnea, pain, anorexia, and asthenia: 0=no problem, 1=mild problem, 2=moderate problem, and 3=severe problem) [12].

Randomization was planned through minimization once patients were enrolled in the study and programmed using ENNOV Clinical data management software. Patients were randomly assigned 1:1 to a routine follow-up (control arm) or web-mediated follow-up (experimental arm). Stratification was conducted at inclusion according to the center, performance status, autologous stem cell transplantation history, relapse, and lymphoma subtype.

Patients were included no later than 3 months at the end of their last treatment. Follow-up was 24 months after enrollment. A medical consultation and a biological assessment were performed every 3 months. In the control arm, CT scans were performed every 6 months. In the experimental arm, scans were performed when medically necessary. Quality of life (QoL) was assessed by 2 questionnaires every 3 months (European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire of Cancer Patients [QLQ-C30] and Patient Health Questionnaire-9 [PHQ-9]) [16,17]. Patient satisfaction with the application was evaluated by an internal questionnaire for patients in the experimental group during the 6-month visit from inclusion.

The Moovcare patient-reported outcome (PRO) system is a class 1 medical device registered by Sivan Innovation, Ltd., with Conformité Européenne marking obtained in July 2017. Versions 1.7 (from July 2017 to October 2019) and 1.8 (from November 2019 to April 2020) were used in this study. “The reimbursed indication of the MOOVCARE Lung device is the early detection of recurrences or complications for patients over the age of 16 with nonprogressive lung cancer after the last medical treatment, regardless of the histological type of the tumor”—an excerpt from the user guide [18]. The indication was validated by the data of the Sentinel Lung study (published in 2019), which demonstrated a survival benefit of 9 months for patients monitored by the application compared to standard monitoring (P=.005) [13].

A scientific committee adapted the questionnaire for patients who were being monitored for lymphoma, whereas the technical monitoring and the decision algorithm remained the same.

PRO data were collected using questionnaires sent by the application directly to the patient through a clickable link to their email address. Patients were asked to complete a 16-question self-assessment every 14 days for 24 months after being randomized to the experimental arm (smartphone or email). They were also able to report an event in the web application between the 2 questionnaires. The study coordinators provided them with a short training session on the application. If the patient failed to complete the questionnaire, a reminder was sent after 24 hours, and the health care team contacted the patient as necessary.

The questionnaire included 4 items, namely weight, LDH level (optional), hemoglobin level (optional), and a free comment (for other symptoms or remarks), and 12 following questions:

Patients were assigned a score based on their symptoms as follows: 0=no problem, 1=mild problem, 2=moderate problem, and 3=severe problem. An alert was triggered in the event of weight loss greater than 2 kg over 1 month, in the event of symptoms rated 3, the presence of fever or night sweats on 2 consecutive occasions, or elevation of serum LDH above 2-fold the normal level, or anemia indicated by hemoglobin levels of <10 g/dL.

In the event of an alert triggered by the application, an email was sent to the care team, with a reminder every 24 hours if there was no response (Figure 1). Patients could also report an event by writing a free text.

The primary outcome was to demonstrate that follow-up via a web application could detect more significant events (including relapses) occurring between 2 routine follow-up consultations with the specialist in patients with lymphoma who were at high risk of relapse compared with standard follow-up. Secondary outcomes were overall and progression-free survival at 2 years, relapse rate at 2 years, QoL or patients in both arms, and compliance and satisfaction for the experimental arm.

An adverse event was defined as any symptom reported by the patient either during the protocol consultation in the control arm or, through the application in the experimental arm. An event was considered significant if, the grade was greater (≥2) based on the Common Terminology Criteria for Adverse Events v4.02 or, if it prompted an imaging examination, treatment (of any kind), supportive care, unscheduled consultation, or emergency hospitalization.

A total of 53 patients were included between July 12, 2017, and April 7, 2020, from 14 centers in France: Le Mans (16/52, 31%), Besançon (9/52, 17%), Nantes (6/52, 11%), Bordeaux Bergonié (6/52,11%), Bordeaux Nord (6/52, 11%), Mont de Marson (4/52, 8%), Dijon (1/52, 2%), Grenoble (1/52, 2%), Paris (1/52, 2%), Strasbourg (1/52, 2%), and Vannes (1/52, 2%). One patient withdrew consent before randomization, and 27 patients were randomized to the experimental arm and 25 to the control arm. The median follow-up time was 21.3 (range 1.3‐25.6) months. In total, 26 patients were evaluated at the primary end point in the experimental arm and 24 in the control arm (Figure 2). The Consolidated Standards of Reporting Trials (CONSORT) guideline (Checklist 1) was used to present the results.

The median age of the entire population was 65.6 (range 20.3‐87.8) years; the median age was 64.3 (range 20.3‐87.8) years in the experimental arm and 69.2 (range 23.9‐84.4) years in the control arm (P=.99). Further, 80% of patients (40/50) had large diffuse B-cell lymphoma, 10% (5/50) had T-cell lymphoma, and 10% (5/50) had Hodgkin lymphoma. The 2 arms were well balanced with respect to age and histological subtype (Table 1).

The median time from initial diagnosis to randomization in the study was 7.7 (range 5.0‐170.2) months. The median was 7.5 (range 5.3‐58.2) months for the experimental arm and 7.8 (range 5.0‐170.2) months for the control arm (P=.43).

Rituximab, cyclophosphamide, doxorubicin, vincristine, and prednisone were the primary first-line chemotherapeutic regimens. In the experimental arm, 17/26 (65%) patients received a single line of chemotherapy, and 9/26 (35%) received 2 lines. In the control arm, 16/24 (67%) patients received a line of chemotherapy, and 8/24 (33%) received a second line.

In the experimental arm, 25/26 patients (96.1%) were still included in the study at the 6-month follow-up, 20/26 (76.9%) at 12 months, and 9/26 (34.6%) at 24 months. In the control arm, 22/24 (91.6%) were still being followed at 6 months, 21/24 (87.5%) at 12 months, and 12/24 (50%) at 24 months (Figure 3).

The primary reasons for the loss of follow-up were the planned end of the protocol for 20 patients and the premature termination of the study by the sponsor (19 patients; Table 2).

During the study period, 119 events were reported (Table 3). Most occurred in the experimental arm (83/119, 69.7%) versus 36/119 (30.2%) in the control arm, with a median number of events per patient of 3.5 (range 1-8) in the experimental arm and 1.8 (range 1-6) in the control arm (P=.004). In the experimental arm, 47/83 (56.6%) events were reported directly by the medical team after a scheduled consultation, whereas 36/83 (43.3%) were reported through the web application.

In the control arm, 19/36 (52.7%) events were detected during a scheduled consultation, 2/36 (5.6%) during an unscheduled consultation, 9/36 (25%) during a consultation with another specialist, 3/36 (8.3%) during hospitalization, and 3/36 (8.3%) during a patient call.

Progression and infection were the most frequently reported events. 19 patients relapsed during follow-up, with a median follow-up time of 7 (range 1.9‐20.6) months; 6 in the web experimental arm and 13 in the control arm (P<.001), with a median follow-up of 7.7 (range 2.8‐20.6) months and 6.7 (range 1.9‐16.4) months (P=.94), respectively. 13 patients were treated for relapse; 11 by chemotherapy and 2 by radiotherapy.

30 patients were infected (23 in the experimental arm and 7 in the control arm; P=.59): 5 patients had an influenza infection, and the infectious agent was not reported for the others. 14 patients received treatment; 13 with antibiotics, 1 with antivirals, and 1 with antibiotics and antivirals. The grade of adverse events was not available in 60 out of 119 (50.4%) cases, which limited data interpretation.

In this study, there is no difference in the occurrence of significant events between the 2 arms (median number per patient of 3.5 in the experimental arm and 1.8 in the control arm; P=.004). Progression, infection, and pain were the most frequently reported events. Patient satisfaction was very high and the patients felt reassured to have electronic monitoring. The patients included in the experimental arm underwent fewer scans compared with those in the control arm, without impacting overall survival, despite a short follow-up (P<.001).

First, the primary outcome of a 30% superiority of reporting significant events in the experimental arm has been overly optimistic. Thus, reducing the end point would have led to a substantial increase in the number of included patients. The number of events was probably not the best criterion for evaluating the effectiveness of remote monitoring. An improvement in QoL or a reduction in the risk of relapse would likely have been more relevant [13,15].

Second, technical problems with the web application occurred (electronic questionnaires not received, with major biases in event reporting). The incidents were not expected because of the experience of the software developer (Moovcare, Sivan Innovation, Ltd); however, there was a change in the technical team between this publication on lung cancer and the start of our study [11-13]. The blocking of automatic questionnaires required 42 direct interventions by clinical study investigators with calls to the patient (firewalls and spam). IT support did not correct these recurring anomalies, despite the changes to the application in November 2019 (5 patients were included in the experimental arm after this date). These operational problems resulted in 15 meetings without resolution of the problems, with an average response time of 4.6 months from technical support (frequent changes to contact persons). As a result, the events in the experimental arm were not reported correctly, leading to study bias. The final report has been sent to the Agence Nationale de Sécurité du Médicament on June 29, 2021.

Finally, only 43% of the events were declared by the application in the experimental arm, which would indicate a problem with patient training.

PROs are underestimated in clinical practice and trials for patients with lymphoma and have most often consisted of paper-based QoL questionnaires [21]. The measurement of PROs via electronic questionnaires has subsequently been evaluated in randomized trials with a low representation of patients with lymphoid malignancies [22]. However, Maguire et al [23] demonstrated that real-time electronic monitoring of symptoms was feasible during an initial chemotherapy cycle in patients with solid tumors and lymphoma, with a reduction in the intensity of side effects and anxiety, compared with a control group.

Proposals for the future include improving the study design by limiting the patient population to a single type of lymphoma, defining the objective to demonstrate an improvement in morbidity and possibly reduce cost, and guaranteeing the reliability of the electronic application. PROMs require standardization of analysis for comparative purposes. Therefore, it is necessary to regulate the use of health care applications to avoid malfunction and abuse [24]. Denis and Krakowski [25] defined 20 criteria of effectiveness, safety, and functionality that should govern the development of ePROMs. Telemonitoring applications should strive to improve patient compliance and prevent patients from dropping out due to a lack of understanding or receiving excessive notifications [26]. Telemonitoring applications must evolve with therapeutic innovations and be regularly reevaluated to demonstrate their long-term benefits on a larger scale [27]. Finally, guidelines are recommended for the design of clinical trials to evaluate the effectiveness of electronic solutions [28-30].

Sentinel Lymphoma is the first randomized phase 3 trial to evaluate the effect of remote monitoring on the detection of significant events in patients with hematological malignancies. Progression, infection, and pain were the most frequently reported events. Despite a high number of events (83 in the experimental arm against 36 in the control arm), the difference was not significant. A more targeted population, a more precise objective, and better security for remote surveillance solutions are recommended for subsequent projects.