This paper is in the following e-collection/theme issue:

Innovations and Technology in Cancer Care (420) Ecological Momentary Assessment (EMA) (307) Innovations and Technology in Cancer Research (33) Mobile Apps for Cancer Care and Cancer Prevention and Screening (205) Questionnaires and Research Instruments for Cancer Patients (17) Research Instruments, Questionnaires, and Tools (563)

Published on in Vol 10 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/57510, first published .
Uncovering the Daily Experiences of People Living With Advanced Cancer Using an Experience Sampling Method Questionnaire: Development, Content Validation, and Optimization Study

Uncovering the Daily Experiences of People Living With Advanced Cancer Using an Experience Sampling Method Questionnaire: Development, Content Validation, and Optimization Study

Uncovering the Daily Experiences of People Living With Advanced Cancer Using an Experience Sampling Method Questionnaire: Development, Content Validation, and Optimization Study

Authors of this article:

Joran Geeraerts1 Author Orcid Image ;   Lara Pivodic1 Author Orcid Image ;   Lise Rosquin1 Author Orcid Image ;   Eline Naert2 Author Orcid Image ;   Geert Crombez3 Author Orcid Image ;   Mark De Ridder4 Author Orcid Image ;   Lieve Van den Block1 Author Orcid Image
Article Authors Cited by Tweetations (1) Metrics

    Original Paper

    1End-of-Life Care Research Group, Vrije Universiteit Brussel, Brussels, Belgium

    2Department of Medical Oncology, Universitair Ziekenhuis Gent, Ghent, Belgium

    3Department of Experimental Clinical and Health Psychology, Universiteit Gent, Ghent, Belgium

    4Translational Radiation Oncology, Physics and Supportive Care Research Group, Vrije Universiteit Brussel, Brussels, Belgium

    Corresponding Author:

    Joran Geeraerts, MSc

    End-of-Life Care Research Group

    Vrije Universiteit Brussel

    Laarbeeklaan 103

    Brussels, 1090

    Belgium

    Phone: 32 2 477 47 57

    Email: joran.geeraerts@vub.be


    Background: The experience sampling method (ESM), a self-report method that typically uses multiple assessments per day, can provide detailed knowledge of the daily experiences of people with cancer, potentially informing oncological care. The use of the ESM among people with advanced cancer is limited, and no validated ESM questionnaires have been developed specifically for oncology.

    Objective: This study aims to develop, content validate, and optimize the digital Experience Sampling Method for People Living With Advanced Cancer (ESM-AC) questionnaire, covering multidimensional domains and contextual factors.

    Methods: A 3-round mixed methods study was designed in accordance with the Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) and the European Organization for Research and Treatment of Cancer guidelines. The study included semistructured interviews with 43 people with stage IV breast cancer or stage III to IV lung cancer and 8 health care professionals. Round 1 assessed the appropriateness, relative importance, relevance, and comprehensiveness of an initial set of ESM items that were developed based on the existing questionnaires. Round 2 tested the comprehensibility of ESM items. Round 3 tested the usability of the digital ESM-AC questionnaire using the m-Path app. Analyses included descriptive statistics and qualitative content analysis.

    Results: Following the first round, we developed an initial core set of 68 items (to be used with all patients) and a supplementary set (optional; patients select items), both covering physical, psychological, social, spiritual-existential, and global well-being domains and concurrent contexts in which experiences occur. We categorized items to be assessed multiple times per day as momentary items (eg, “At this moment, I feel tired”), once a day in the morning as morning items (eg, “Last night, I slept well”), or once a day in the evening as evening items (eg, “Today, I felt hopeful”). We used participants’ evaluations to optimize the questionnaire items, the digital app, and its onboarding manual. This resulted in the ESM-AC questionnaire, which comprised a digital core questionnaire containing 31 momentary items, 2 morning items, and 7 evening items and a supplementary set containing 39 items. Participants largely rated the digital questionnaire as “easy to use,” with an average score of 4.5 (SD 0.5) on a scale from 1 (“completely disagree”) to 5 (“completely agree”).

    Conclusions: We developed the ESM-AC questionnaire, a content-validated digital questionnaire for people with advanced breast or lung cancer. It showed good usability when administered on smartphone devices. Future research should evaluate the potential of this ESM tool to uncover daily experiences of people with advanced breast or lung cancer, explore its clinical utility, and extend its validation to other populations with advanced diseases.

    JMIR Cancer 2024;10:e57510

    doi:10.2196/57510

    Keywords

    cancer; quality of life; ecological momentary assessment; experience sampling method; telemedicine; mHealth; eHealth; patient outcome assessment; validated instruments 


    Background

    Quality of life assessment among people with cancer often relies on retrospective patient-reported outcome measures (PROMs), which typically require patients to aggregate their experience over several days or weeks into 1 score (eg, “During the past week, were you tired?”) [1-3]. This precludes temporally fine-grained knowledge on how cancer-related experiences such as physical or psychological symptoms and concerns change within and across days and the mechanisms underlying these changes. Moreover, studies found that retrospective PROMs often over- or underestimate in-the-moment somatic and psychological experiences across various populations, indicating a need for more fine-grained measures [4,5]. From a research and clinical perspective, this detailed knowledge on in-the-moment experiences is critical for improving patient symptom management and psychosocial support, such as by identifying novel intervention targets.

    To bridge this gap, the experience sampling method (ESM) [6], also called ecological momentary assessments [7], may be suitable. The ESM or ecologic momentary assessments involve repeatedly gathering self-reported data from participants in the context of their daily lives, often multiple times per day for several consecutive days through mobile devices such as smartphones [7-9]. Contrary to traditional PROMs, the ESM mitigates retrospective biases and improves ecological validity of findings by asking questions about momentary experiences in their natural environment (eg, “At this moment, I feel...”) [7]. Moreover, the ESM provides the opportunity to study affect over time (ie, experiences of feelings or emotions) as an important indicator of emotional functioning and psychological well-being [9-11] and to investigate patients’ experiences together with concurrent contexts, such as the social environment [12]. Including contextual items can facilitate the identification of situations that alleviate or exacerbate certain experiences, thereby informing future psychosocial interventions.

    Despite the ESM’s potential to provide novel insights into the daily experiences of people with cancer, its use in oncology research remains limited, especially among people with advanced (ie, metastatic) cancer [9,12,13]. Nevertheless, compared to people in the earlier stages of cancer, people with advanced cancer have a higher likelihood of experiencing symptoms and concerns that negatively impact their quality of life [14,15]. A possible explanation for the limited use of these methods among people with advanced cancer is that researchers may avoid them to prevent placing additional burden on patients through repeated assessments. However, to develop and improve interventions to alleviate these high levels of symptoms and distress, gaining a more detailed understanding of the well-being of people with advanced cancer in the context of their daily life (ie, its fluctuations, mechanisms, determinants, and consequences) is imperative; for this purpose, the use of the ESM is recommended [16,17]. The limited number of ESM studies among people with advanced cancer have investigated a range of symptoms, concerns, and measures of well-being across quality of life domains and provided evidence for the dynamic nature and associations thereof [18-30]. For example, Badr et al [21] found that greater pain in the morning was associated with feeling less aroused mood (eg, more tiredness and less peppy) during the rest of the day for women with metastatic breast cancer, with pain and low arousal mood being associated with romantic relationship interference.

    There is currently no validated ESM questionnaire designed specifically for people with advanced cancer [9,13]. Validity, especially content validity, is a crucial indicator of whether the content of an instrument is an adequate reflection of the construct being measured [31,32]. However, it is often overlooked in ESM research as a whole, leading to recent calls for more content validation of ESM questionnaires [9,13,32,33].

    By reporting the development, content validation, and optimization of an ESM questionnaire, this study is the first step of a larger project in which we aim to test the feasibility of the ESM and use it to obtain novel insights into the daily experiences of people with advanced cancer. Because symptoms can vary across different advanced cancer diagnoses and our aim was to develop a questionnaire that is highly relevant to the specific experiences of intended users, our project’s scope is narrowed to people living with advanced breast or lung cancer. We selected these diagnoses as they are among the most prevalent cancer diagnoses with high mortality rates [34-36] and are associated with considerable risk for experiencing serious symptom burden [37-41].

    Objectives

    In this study, we aimed to develop, validate, and optimize the Experience Sampling Method for People Living With Advanced Cancer (ESM-AC) questionnaire. The digital ESM questionnaire aims to comprehensively assess relevant daily experiences (ie, symptoms, concerns, and well-being) of people with advanced breast or lung cancer and the context in which these experiences occur; it collects these data multiple times per day for several consecutive days.


    Study Design

    We conducted a 3-round interview study with patients and health care professionals using a mixed methods research design (summarized in Figure 1 [42]). To develop and validate the ESM questionnaire in the first 2 interview rounds, we based our design on the guidelines of PROMs [31,43] because no specific guidelines for ESM questionnaires were available [32]. Specifically, the Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) methodology [31] guided the assessment of the content validity of our initial set of items in the first 2 rounds (ie, covering relevance, comprehensibility, and comprehensiveness; refer to Textbox 1 for an overview of key psychometric concepts used in this study). In the first round, the item set was shortened and categorized into a core and supplementary item set based on content validity, appropriateness, and relative importance, following the European Organization for Research and Treatment of Cancer (EORTC) guidelines for module development [43-45]. The second round focused on the comprehensibility of all items and on the relevance and appropriateness of the items added after round 1. In the third round, we optimized the digital (core) ESM questionnaire by assessing barriers related to its usability for patients using the dedicated ESM smartphone app (ie, m-Path; KU Leuven [46]).

    ‎
    Figure 1. Flowchart of the development and validation procedure [42]. EORTC QLQ-C30: European Organization for Research and Treatment of Cancer Core Quality of Life Questionnaire; ESM: experience sampling method; FACIT-Pal: Functional Assessment of Chronic Illness Therapy—Palliative Care; IPOS: Integrated Palliative Care Outcome Scale.
    Textbox 1. Key concepts with their respective definitions.
    • Content validity: the extent to which the content of an instrument is an adequate reflection of the construct to be measured. This includes relevance, comprehensiveness, and comprehensibility [31].
    • Relevance: the extent to which a questionnaire item is relevant for the construct of interest within a specific population and context of use [31].
    • Comprehensiveness: the extent to which all key aspects of the construct are included in the questionnaire [31].
    • Comprehensibility: the extent to which a questionnaire item is understood by patients as intended [31].
    • Appropriateness: the extent to which a questionnaire item is perceived as appropriate and not upsetting [43].
    • Relative importance: the extent to which a questionnaire item is deemed more important for the questionnaire’s context of use than other items in the same content domain.
    • Usability: the extent to which a system, product, or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use [47].

    Ethical Considerations

    This study was approved by the central ethics committee of university hospital Brussels (Belgian Unique Numbers: 1432021000533 and 1432023000043) and by the local committee of general hospital Aalst, Belgium. All participants provided written informed consent before study participation. Patients did not receive any compensation. Health care professionals received a €25 (US $27.06) gift card. Data were treated confidentially and were strictly analyzed in a deidentified form.

    Participants and Setting

    For the first 2 rounds, we planned to interview 32 patients and 8 health care professionals from 1 university hospital and 1 regional hospital in Belgium. These sample sizes adhere to the COSMIN and EORTC guidelines [31,43]. In the third round, we aimed to include 8 patients from the former university hospital [48] and 4 additional patients if, after the previous usability interviews, large changes would be made that would require further testing. JG and the hospital staff identified eligible patients through clinic appointment lists, and JG invited patients to participate via telephone or in-person communication during hospital visits. Health care professionals were identified through the research team’s professional networks and contacted via email.

    Inclusion criteria included the following: (1) a diagnosis of stage III or IV lung cancer or stage IV breast cancer; (2) patient aged ≥18 years; (3) patient spoke and understood the Dutch language; and (4) patient assigned an Eastern Cooperative Oncology Group performance status of 0, 1, or 2, based on the assessment by their treating physician.

    Exclusion criteria included the following: (1) patient having major communication difficulties or insufficient cognitive abilities to take part in a semistructured interview (as judged by their treating physician); (2) patient having any psychiatric disorder that, in the opinion of their treating physician, might hinder participation due to expected burden or unreliable responses; (3) patient having uncorrectable hearing or poor vision; or (4) patient had participated in a previous part of this study.

    We aimed to include 4 equally sized subgroups based on the primary tumor site (breast or lung cancer) and age (<70 years or ≥70 years) [49,50].

    As for health care professionals, we aimed to include a specialist in respiratory oncology, an oncologist specialized in breast cancer, a radiotherapy specialist, an oncology nurse, an onco-psychologist, a health sciences researcher, and 2 specialist palliative care providers (ie, a physician and a nurse affiliated with a palliative home care team).

    Measurement Instruments and Procedures

    Initial Item Set

    The questionnaire aimed to comprehensively measure and evaluate daily experiences of people with advanced cancer and the context in which they occur. More specifically, we conceptualized daily experiences as symptoms, concerns, and well-being across physical (including physical symptoms and functioning), psychological (including positive and negative affect, psychological symptoms, and cognitive concerns), social, spiritual-existential, and global well-being domains (Figure 2). Context was conceptualized as the person’s current location, activity, social company, substantial events, medication use, and sleep quality.

    We created an initial item set capturing in-the-moment experiences based on (1) the items of questionnaires identified in the 2018 review of PROMs in patients with advanced cancer by van Roij et al [1] and (2) an existing ESM item repository from the field of mental health sciences [42]. From the review by van Roij et al [1], we selected 3 questionnaires: the European Organization for Research and Treatment of Cancer Core Quality of Life Questionnaire (EORTC QLQ-C30), Functional Assessment of Chronic Illness Therapy—Palliative Care (FACIT-Pal), and the Integrated Palliative Care Outcome Scale [51-53], as they relate to our target population, have sufficient content validity, and have a comprehensive symptom coverage (ie, did not focus on one specific symptom or experience). On the basis of the consensus achieved through discussion among the authors, we excluded overlapping items and items with low expected intraday variability (eg, “I have family members who will take on my responsibilities”) and retained 43 items suitable for the measurement of symptoms, concerns, and well-being across various subdomains (Figure 2). When consensus was required for adding, changing, or removing items, the content was first discussed primarily among JG, LP, and LVdB, who are all trained psychologists. LP and LVdB have >10 and 20 years of experience as end-of-life researchers, respectively. JG had 1 year of prior expertise in ESM mental health research. If further discussion or advice was needed, other authors were consulted, including a research assistant (LR; no prior expertise), a medical oncologist (EN; 7 years of experience), a health psychology researcher with experience in ESM research (GC; ≥30 years of experience), and a radiation oncologist (MDR; ≥20 years of experience).

    ‎
    Figure 2. Subdomains that the Experience Sampling Method for People Living With Advanced Cancer (ESM-AC) questionnaire intended to cover. Note that the between-bracket numbers after each domain name indicate the approximate number of items that we aimed to retain per domain and the number of most important items that participants had to choose for each right-most subdomain.

    From the ESM item repository, we purposively selected 12 items measuring affect, spanning across the valence and arousal dimensions [54] (ie, levels of pleasantness and physiological activation, respectively), and 13 items measuring context. We additionally selected items to measure the patient’s experience while completing the ESM questionnaire (ie, meta-experience items). We obtained official Dutch translations for all items and rephrased them to reflect in-the-moment experiences (eg, changing “During the past 7 days, I felt...” to “In this moment, I feel...” or “Since the last beep, I felt...” with “beep” referring to the assessment prompt). For less frequent experiences or events, such as, for the item “I have had diarrhea,” we used the phrase “Since the last beep” instead of “In this moment.” One item measuring sleep quality was adapted from the FACIT-Pal questionnaire [52] and used for the first assessment of the day (ie, “Last night, I slept well”). All English translations of items presented in this paper are phrased analogous to their existing PROM counterparts, or if no such counterparts were available, we provided translations of the Dutch versions used in this study.

    Content Validity and Usability Assessments

    In all study rounds, we conducted individual semistructured interviews with patients with advanced breast or lung cancer. One round also included interviews with health care professionals, as outlined in Figure 1. These interviews served to assess content validity, to shorten the initial item list and divide it into a core and supplementary set, and to optimize the digital ESM questionnaire based on its usability. At the start of all interviews, the patients completed a baseline questionnaire on sociodemographic and clinical characteristics (age, gender, living situation, marital status, education level, employment status, religious denomination, and received treatments). In round 3, the baseline questionnaire additionally assessed cognitive concerns [55] and smartphone use [56,57]. We conducted all interviews in person, either at patients’ homes or in quiet hospital rooms. Patients’ friends and relatives were allowed to be present during the interviews. Across rounds, we introduced the ESM to participants as a digital diary on a smartphone device that uses 10 assessments per day for several consecutive days to study people’s symptoms, concerns, well-being, and daily situations as well as their fluctuations within and across days.

    During round 1, JG interviewed patients and health care professionals to evaluate the relevance and comprehensiveness of symptoms, concerns, and well-being items. We aimed to create a core item set of 33 items, which was the foreseen number of items needed to cover all subdomains, and a supplementary set with no item limit and aimed to improve its comprehensiveness by adding items deemed relevant but missing by the participants. Participants were asked to verbally rate each item’s relevance (“not at all,” “a little,” “quite a bit,” and “very much”), select the most important items for each subdomain (Figure 2 displays the number of items to select per subdomain, as instructed by the interviewer), suggest missing concepts, and mark inappropriate items. Participants were prompted for reasons for categorizing items as inappropriate or “not at all” or “a little” relevant.

    In round 2, JG interviewed patients on the comprehensibility of items resulting from the first round (as the last part of content validation), the relevance and appropriateness of newly added items, and the appropriateness and comprehensiveness of context and meta-experience items and their response options (assessed analogous to round 1). To assess comprehensibility, patients completed a pen-and-paper questionnaire while thinking out loud [58].

    In round 3, JG and LR conducted interviews to assess and optimize the ESM questionnaire’s usability by letting patients respond to it in the m-Path app [46]. m-Path is a web-based platform that provides “an intuitive and flexible framework to conduct smartphone-based ecological momentary assessment and intervention studies...” [46]. Patients were each provided with a Motorola E20 smartphone device (Motorola Mobility LLC) with the digital ESM questionnaire available in the m-Path app. They were instructed on how to use the app and asked to complete the digital questionnaire on the provided device while thinking out loud. The researcher prompted patients when difficulties were observed (eg, difficulties answering certain ESM questions). Afterward, a brief semistructured interview assessed the usability of the questionnaire through an adapted version of the System Usability Scale (5-point Likert scale; 1=totally do not agree and 5=totally agree) [59,60]. Usability outcomes included readability, comprehensibility, ease of use, reasons for encountered difficulties, and expected burden of receiving 10 assessments per day for 6 days. Finally, patients completed the digital ESM questionnaire a second time without thinking out loud to estimate completion times. All interviews were recorded and transcribed verbatim. More details on procedure and instruments for this round have been reported in the study protocol [61].

    Data Analyses and Continuous Adaptations of the Questionnaire

    Following the EORTC guidelines for module development, as applied by Sprangers et al [43] and Groenvold et al [45], we transformed item relevance ratings into a 0 to 100 scale, with “not at all” corresponding to 0 and “very much” to 100. We calculated mean relevance scores and SDs per item. In addition, we calculated the percentages of respondents who rated an item as inappropriate or upsetting, who listed an item among the top n most important items per subdomain (n was the approximated number of items to retain in the final questionnaire for each subdomain; Figure 2), and who found an item incomprehensible. We calculated descriptive statistics for usability.

    Using conventional content analysis [62] on the interview transcriptions, we inductively developed content categories for participants’ reasons of lack of item relevance (provided by participants who judged an item as “not at all” or “a little” relevant), inappropriateness, problems with comprehensibility, and themes of novel items to add [62]. We added items to the list if at least 2 participants suggested adding it to the questionnaire. Furthermore, we developed content categories for difficulties or conveniences in the user experience or comprehension of the digital questionnaire.

    The questionnaire was adapted after each of the 3 rounds. After round 1, we used descriptive statistics of relevance, importance, and appropriateness ratings from the patients and health care professionals to guide item exclusion and categorization into core and supplementary sets (refer to Multimedia Appendix 1 for an overview of the categorizations). We assigned items to the core item set if they ranked among the top n most important per subdomain (refer to Figure 2 for n values), were judged “quite a bit” or “very much” relevant by half of the participants (50%), and were deemed appropriate (or amenable to rewording). For the removal of items, the authors discussed the participants’ reasons for low relevance of items that were rated as “not at all” or “a little” relevant by at least half of the participants, or of items for which the participants provided recurring reasons for lack of relevance or the inappropriateness of items and the item could not be appropriately reworded or changed to resolve those reasons. Items that were not removed or categorized into the core set were assigned to the supplementary set. Note that the decision to use the core and supplementary sets was made after analysis of round 1.

    After round 2, we made necessary and feasible item revisions based on the descriptive statistics of comprehensibility and inappropriateness and on the content categories for reasons of items’ low comprehensibility and inappropriateness.

    After round 3, we used descriptive statistics of usability outcomes and content categories of difficulties when using the digital questionnaire to improve the usability of the questionnaire in m-Path. Following general recommendations in ESM research [16,63], we used a mean questionnaire completion time threshold of 3 minutes to determine whether the questionnaire was considered too long.


    Participant Characteristics

    In round 1, a total of 15 patients and 8 health care professionals participated; in round 2, a total of 18 new patients participated; and in round 3, a total of 10 new patients participated (Table 1). The overall mean age was 67.3 (SD 10.3) years. Overall, 23 (53%) of the 43 patients had a stage III or IV lung cancer diagnosis, and the remaining 20 patients (47%) had a stage IV breast cancer diagnosis. Close others were present during 4 interviews in round 1, seven in round 2, and seven in round 3.

    Table 1. Sociodemographic and clinical characteristics of patients per interview round (N=43).
    CharacteristicsRound 1 (n=15)aRound 2 (n=18)bRound 3 (n=10)
    Age (years)

    		Mean (SD)68.0 (8.5)68.7 (11.3)63.8 (11.1)

    		Range56-7844-8645-78
    Gender (female), n %11 (73)14 (78)6 (60)
    Living situation, n

    		Living alone at home242

    		Living with a partner/children/others at home13148
    Marital status, n

    		Married138—c

    		Living together but not married06—

    		Widowed11—

    		Divorced13—
    Educational level, n

    		Primary201

    		Secondary8104

    		Tertiary585
    Employment status, n

    		Professionally active211

    		Not professionally active13179
    Religious denomination, n

    		Catholic Christian686

    		Not religious594

    		Not specified410
    Cancer diagnosis, n

    		Stage III or IV lung cancer7106

    		Stage IV breast cancer884
    Treatment or treatments received, as reported by the patient, n

    		Chemotherapy14139

    		Radiotherapy13105

    		Surgery1237

    		Hormonal therapy452

    		Immunotherapy694
    EORTC QLQ-C30d concentration problems, n

    		Not at all——7

    		A little——2

    		Quite a bit——1

    		Very much——0
    EORTC QLQ-C30 memory problems, n

    		Not at all——5

    		A little——3

    		Quite a bit——2

    		Very much——0
    Smartphone ownership in years, mean (SD)——10.2 (4.4)
    Daily time spent on smartphone in hours, mean (SD)——3.2 (2.8)
    Confidence using smartphone (1=“not at all confident,” 5=“very confident”), mean (SD)——4.1 (0.7)

    aDue to an oversight, we did not collect participation rates and reasons for nonparticipation in this round.

    bOut of 25 invited patients. Reasons for nonparticipation included no interest, as indicated by patient or partner (n=5), inability to find an appropriate interview location (n=1), experiencing distress (n=1), or no reasons provided (n=1).

    cNot measured.

    dEORTC QLQ-C30: European Organization for Research and Treatment of Cancer Core Quality of Life Questionnaire.

    The following sections present the results per interview round and relevant adaptations made to the ESM questionnaire based on these findings.

    Interview Round 1

    Relevance

    Most items received positive relevance ratings, with no unanimous low relevance ratings across all participants (Multimedia Appendix 2). The most frequent reasons for considering an item lacking in relevance were overlapping content with other items, not experiencing the measured construct, not perceiving the measured construct as bothersome, and thinking the item could be phrased better. After discussion among the research team, we removed 12 items that at least half of the participants rated as having “a little” relevance or less or that participants noted had considerable overlapping content with other items. For instance, we removed the item “At this moment, I feel sick” due to overlap with specific symptoms such as nausea and removed the item “At this moment, I feel capable of making decisions” due to low reported relevance because patients reported not having to make decisions.

    Some items were considered irrelevant by the participants because they measured stable constructs within a day. To address this, we deviated from the planned approach to develop in-the-moment items only and instead developed several items for designated morning and evening assessments. We dedicated 1 item of the initial item list to morning assessments and 11 to evening assessments. For instance, the in-the-moment item “At this moment, I feel moral support by my close ones” was revised to the evening item “Today I felt supported by others.” Items excluded before round 1 based on little expected within-day variability were reconsidered for inclusion in the once-daily questionnaires. Hence, we added 8 initially removed items to the evening list for further testing in round 2 (eg, “Today, I was able to openly discuss my concerns with my close ones”).

    Appropriateness

    Out of 55 items, 22 (40%) were deemed inappropriate by between 1 and 5 participants (Multimedia Appendix 2), with 12 (22%) items deemed inappropriate by at least 2 participants. Reasons included privacy concerns, content overlap, confronting questions, infrequent experiences, question formulation, clinical utility, and bad subdomain fit (Multimedia Appendix 3). We removed the most inappropriate item “At this moment, I feel enthusiastic” as 4 patients and 1 health care professional marked it as inappropriate due to content overlap and patients not experiencing this feeling.

    Comprehensiveness

    Participants suggested adding several constructs to improve comprehensiveness, leading to the addition of 13 items to the item list (Multimedia Appendix 4). Among these, 2 were conditional items administered only if certain responses are given during the same assessment, such as reporting moderate pain levels or poor sleep. These questions included “The pain is located in these parts of the body: ...” and “I think I didn’t sleep so well, because: ... .” Examples of other added items included “At this moment, I feel capable of working” and “At this moment, I have negative thoughts or feelings.” In addition, we included 3 items in the questionnaire as the research team thought them to be necessary for comprehensive measurement of the psychological domain (“At this moment, I feel restless” and “At this moment, I feel depressed”) and an open question concerning other contextual factors (“If there is anything else you want to mention about the period since last beep, you can do that here:”).

    Relative Importance

    We assigned 46 items with the highest relative importance of their subdomain to the core questionnaire and 38 items to the supplementary list (refer to Multimedia Appendix 2 for the proportions of how many times items were chosen as among the top most important).

    Interview Round 2

    Comprehensibility

    Between 1 and 5 participants provided remarks for 31 (39%) out of 79 items (Multimedia Appendix 5). Reasons for marking items as incomprehensible included unclear word meanings, different interpretations from the intended meaning, situational content, response options misalignment, and other issues. In response to this feedback, we changed the wording of some items and response options and removed some items (Multimedia Appendix 6). For instance, we replaced the response option “On the move” under the item “What am I doing?” to “En route (eg, on the bus)” for clarity. Another example is the core questionnaire item “Today I felt supported by others,” which we changed to “Today I received the support I needed from my loved one(s)” because some patients indicated not needing or seeking support all the time.

    Relevance of Added Items

    On average, most added items were rated as at least “a little” relevant, with mean ratings typically exceeding “quite a bit” relevant (Multimedia Appendix 2).

    Appropriateness of Added Items

    No items were considered as inappropriate by the participants.

    Additional Findings and Changes Made

    Three patients reported frequently experiencing muscle cramps, leading to the addition of the item “Since the last beep, I had muscle cramps” to the supplementary list. On the basis of research team consensus, we improved the comprehensiveness of the “Where am I?” item by adding an “outside” response option. Figure 3 displays the resulting questionnaire in the m-Path app.

    ‎
    Figure 3. Screenshots of the Experience Sampling Method for People Living With Advanced Cancer questionnaire in the m-Path app. Left: receiving a notification, middle: example of the slider response scale; right: example of the multiple-choice response scale.

    Interviews Round 3

    Usability

    On a scale ranging from 0=“completely disagree” to 5=“completely agree,” participants generally expressed positive sentiments about using the ESM-AC questionnaire in their daily lives (mean 3.6, SD 0.8), finding it easy to use (mean 4.5, SD 0.5), and expecting no need for support with the questionnaire or the smartphone device in their daily lives (mean 1.6, SD 0.7 and mean 1.5, SD 0.7, respectively). They also indicated that there was no inconsistency in the questionnaire (mean 1.6, SD 0.7). They expected that most people would quickly learn to use the questionnaire (mean 4.0, SD 1.1), felt confident using it (mean 4.2, SD 1), did not require a lot of knowledge to complete it (mean 1.3, SD 0.5), items and response options were clear (mean 4.3, SD 0.5 and mean 4.0, SD 0.9; respectively), the response options were comprehensive (mean 4.1, SD 1), and the lay-out was satisfying (mean 4.2, SD 0.6). Moreover, participants did not experience it as burdensome to complete the questionnaire (mean 1.5, SD 0.7) and did not think it was too long (mean 1.9, SD 0.9). However, as reflected by neutral mean scores with higher variance, participants were more divided regarding the simplicity of item phrasings (mean 2.2, SD 1.2) and the readability of items (mean 3.9, SD 1.4). Moreover, most participants anticipated that completing the questionnaire 10 times per day on 6 consecutive days would be burdensome (mean 3.7, SD 1.1).

    Perceived Difficulties

    Participants reported various barriers with using the digital ESM-AC questionnaire and device, and we observed some difficulties when participants used the questionnaire. For some patients, response formats and the option to skip open-ended items were initially not clear, the momentariness of items (ie, “At this moment, I feel...”) required further instructions (eg, participants would give higher pain scores due to previous pain episodes, when currently not experiencing pain), interpretations of some complex items were unintended (eg, concentration problems were interpreted as wider cognitive problems), the purpose of the intensive assessment schedule of the ESM study and of specific questionnaire content domains were unclear (eg, context items), and the device went into standby mode during the interview. All the changes made to the ESM-AC questionnaire, smartphone device settings, and onboarding instructions are reported in Table 2. Refer to Multimedia Appendix 7 for the resulting core ESM questionnaire. We also created a manual for researchers to provide patients with instructions where needed (Multimedia Appendix 8).

    Table 2. Changes made to different ESM-ACa questionnaire properties after the usability interviews of round 3.
    Property and observed or reported barriersChanges made
    ESMb questionnaire

    		Momentariness of item unclearThe phrasing “at the moment the beep went off” was added to the multiple-choice context items. For example: “Who am I with?” was replaced with “Who was with me at the moment of the beep?”

    		Momentariness of item unclearIn-the-moment phrasings were added to items that did not previously include it. For example: “I’m in bed or on the couch” was replaced with “I was in bed or sofa when the beep went off.”

    		Meaning of “place I was at” wrongly associated with bed or sofa“I was happy with the place I was at” was reordered to be between “Where was I at the moment of the beep?” and “I was in bed or sofa when the beep went off.”

    		Unclear what was measured with substance item“Since last beep, I have used the following” was replaced with “Since last beep, I have used the following substance(s)”; the response option “Other” was changed to “Other substance(s).”

    		Need for additional open-ended items when participants used the “Other” response optionAn m-Path app feature was selected for the multiple-choice items that allows participants to directly type new categories when the “Other” option is selected.
    Smartphone device settings

    		Device screen darkened while completing the questionnaireThe time-to-standby settings on the devices was changed from 30 seconds to 60 seconds.
    Onboarding instructions

    		Response formats and option to skip open-ended items were not initially clear, momentariness of items required instructions, unintended interpretations of some complex items, purpose of the intensive assessment schedule of the ESM study and of some study domains (eg, context items) was unclear, reported expectations of missing assessments, and difficulty unlocking the smartphoneA formal interview guide was developed for the training at the onboarding session, which included instructions on how to explain the different response option formats and how to use them, skipping open-ended items, temporality of questions (ie, in-the-moment or since the last beep), content of more complex items (eg, concentration as separate from memory problems), the purpose of the intensive assessment schedule of the ESM study and of some question domains, acceptability of missing assessments, and unlocking the smartphone.

    aESM-AC: Experience Sampling Method for People Living With Advanced Cancer.

    bESM: experience sampling method.

    Completion Times

    During the second time of filling in the digital ESM-AC questionnaire (ie, without thinking out loud), it took participants on average 3.8 (SD 1.1) minutes to complete the questionnaire of 25 to 31 items (depending on the number of triggered conditional items).


    Principal Findings

    We developed, content validated, and optimized the ESM-AC questionnaire, a digital ESM questionnaire covering multidimensional domains to capture the experiences of people with advanced breast or lung cancer. Overall, the patients found the questionnaire items comprehensible and appropriate and had positive views toward using the questionnaire in the m-Path app. As all items in the initial set were relevant to at least some patients, we primarily used the perceived importance of the items to categorize them into a core questionnaire for use with all patients and a supplementary item set from which patients can select items to tailor the ESM questionnaire to their needs and experiences.

    As a novel and promising tool to assess patients’ symptoms, concerns, and overall well-being, the ESM-AC questionnaire supplements the existing measurement methods in oncology, a field that has traditionally relied on retrospective PROMs [1-3]. The ESM uniquely allows for the measurement of experiences in real time within the patient’s everyday life [16]. By using multiple assessments per day, it enables the investigation of how these experiences change and unfold over time, including their correlations and temporal relationships [16]. The repeated within-day assessments of the ESM can also supplement more traditional daily diary measures in oncology that assess patients once per day to uncover fine-grained fluctuations of symptoms. This can be important to better understand the complexity and dynamics of patient experiences from a research perspective. Moreover, from a clinical perspective, the ESM can be used to improve understanding of symptoms or concerns of individual patients identified using traditional once-daily or weekly administered PROMs.

    To the best of our knowledge, the ESM-AC questionnaire is the first of its kind in oncology in several respects. First, the limited number of ESM studies in populations with cancer have never determined the content validity of their questionnaire items to be assessed in a repeated in-the-moment context [9,13]. Second, in cancer ESM research, the ESM-AC questionnaire is among the first to incorporate items on context and context appraisal [9,12]. By including items on concurrent location, activity, and social company, it will be possible to better understand fluctuating symptoms and their interactions with contextual factors. ESM research in other fields has shown how different contexts such as social company, concurrent activities, and location can influence patients’ mental and physical experiences [64-66]. Third, by dividing items into a core and supplementary list, item selection can be adapted or tailored to a particular patient or a population of patients, that is, by adding relevant supplementary items such as “At this moment, I feel capable of working.” This makes our ESM measurement highly relevant for people with advanced breast or lung cancer.

    Using the m-Path app [46], results showed that the ESM-AC questionnaire was easy to use for all patients, and the patients had positive views toward the questionnaire presented on the device. This is crucial because it is important to minimize the potential burden of frequent daily assessments. This is especially true when working with populations that may be more likely to experience increased symptoms and reduced physical functioning related to cancer and related treatments. In addition, although the questionnaire took, on average, longer than the generally recommended 3 minutes’ completion time in ESM research [16,63], participants indicated that it was not too long. Therefore, we deviated from our initial 3-minute threshold and did not further shorten the questionnaire [61]. As we purposively sampled people aged >70 years and <70 years (mean 63.8, SD 11.1; range 45-78 years), we were able to conclude that the system questionnaire was usable for older age groups (ie, those aged ≤78 years) that are typically thought to have less smartphone experience, as indicated by their positive views on usability of the system.

    Implications for Future Research

    The next step in the development of the ESM-AC questionnaire is to evaluate it in a detailed pilot ESM study. Such a study needs to evaluate the optimal number of daily assessments among people with advanced lung cancer or advanced breast cancer. As most participants indicated that they expected 10 assessments per day for 6 consecutive days, as is often used in ESM research [16], to be potentially burdensome, the burden of completing such an intensive assessment schedule should be carefully investigated in real life. This burden needs to be weighed against the necessary resolution to measure change in the construct of interest. In addition, further research is needed regarding the acceptability of the questionnaire length and clarity of the instructions, items, and response options if researcher help is not immediately available. If further research confirms the feasibility and optimal features for a larger-scale ESM study, this will pave the way toward a substantial improvement of our knowledge of how symptoms, concerns, and well-being across multiple domains fluctuate in the everyday life of people with advanced breast or lung cancer.

    Researchers aspiring to apply similar methods to other populations with cancer or serious illness are encouraged to further adapt the methods to their target population. We recommend the ESM-AC questionnaire as a starting point for adaptations toward the target population and context. The core ESM questionnaire can be used in its entirety or researchers can select the domains of interest, possibly supplemented by items selected from the supplementary item set. Determining the questionnaire’s content validity through semistructured interviews will help to optimize and ensure its relevance, comprehensiveness, and comprehensibility for intended research.

    Furthermore, ESM data can be compared to retrospective patient-reported outcome data to confirm and obtain more evidence on the added value of the ESM and the different experiences it captures and to investigate the ecological validity of such data. Another important area of future ESM research in oncology can be to explore its clinical value and utility, for instance, by providing clinicians with time-series visualizations of their patients and comparing these with information gathered through traditional consultations.

    Strengths and Limitations

    This study is among the first studies to test the content validity of an ESM questionnaire in any scientific field and has resulted in the first content-valid ESM questionnaire in the field of oncology, thereby answering to recent calls for more questionnaire validation in ESM research [9,12,13]. This study has several strengths. First, it involved close collaboration with people with cancer and health care professionals in multiple phases of questionnaire development, ensuring its relevance for the target population. Second, relevance was further ensured by adapting items from existing validated PROMs [51-53]. Moreover, unlike many quantitatively focused questionnaires in ESM research, the use of a free-text response item “If there is anything else you want to mention about the period since last beep, you can do that here:” allows us to study any relevant experiences that are currently missing in the core questionnaire. Third, we included an equal number of patients aged <70 years and >70 years, ensuring the inclusion of the latter as an often underrepresented group in cancer studies. Finally, this study’s relatively good participation rate reduces the risk of selection bias.

    Several limitations should be noted. First, the study was limited to Dutch-speaking patients from 2 study sites, possibly limiting the extent to which the ESM-AC questionnaire’s content validity can be generalized to patients with sociodemographic characteristics different from our sample. However, the ESM questionnaire will be further tested among new patients recruited from different hospitals. Second, the relatively high functional status of patients in our sample (ie, Eastern Cooperative Oncology Group scores between 0 and 2) may lead to limited generalizability of the results to patients with advanced cancer who have more functional limitations. Third, as no people aged >78 years participated, the usability of our ESM is unknown for older populations. Fourth, we did not record whether patients were actively receiving treatment, thereby preventing more detailed insight into the sample’s current perspectives and experiences. Finally, due to the study design, we were not able to test how health care professionals viewed the relevance and how patients and health care professionals viewed the relative importance of evening assessment items that that were initially removed by the authors based on their low expected within-day variability.

    Conclusions

    We successfully developed the ESM-AC questionnaire, the first content-valid digital ESM questionnaire in oncology to study the daily experiences of people with advanced breast or lung cancer in their everyday environments. If the method proves feasible in future research on advanced cancer and in other patient groups, it paves the way toward gaining novel insights into the daily lives of patients with cancer, possibly informing and facilitating patient-centered care.

    Acknowledgments

    This work is part of a wider study funded by the Research Foundation—Flanders (Fonds Wetenschappelijk Onderzoek) under grant agreement G0C8120N. LP is a senior postdoctoral fellow of the Research Foundation–Flanders. LVdB holds the Collen-Francqui Research Professor Chair “Aging, dementia and palliative care” (2020 to 2023). The authors sincerely thank all the participants. We used generative artificial intelligence (ChatGPT, using GPT-3.5 [67]) throughout the manuscript as guidance to shorten the original draft (no new information or ideas were generated).

    Authors' Contributions

    The study was conceptualized by LVdB, MDR, LP, and JG. Data were curated by JG and LR. Formal analysis, software, and visualization were performed by JG. LVdB, MDR, and LP were involved in funding acquisition. Investigation was conducted by JG and LR. Methodology was conducted by JG, LP, EN, GC, MDR, and LVdB. LP, MDR, and LVdB provided supervision and validation. The original draft was written by JG, LP, and LVdB, and reviewing and editing were done by LR, EN, GC, and MDR.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    A figure about the criteria for categorization into the core questionnaire, supplementary set, or items to be removed.

    DOCX File , 22 KB
    Multimedia Appendix 2

    Inappropriateness frequencies, relevance means, and proportions of relative importance ratings of experience sampling method items.

    DOCX File , 33 KB
    Multimedia Appendix 3

    Frequency table of the categorized reasons for deeming an item inappropriate.

    DOCX File , 22 KB
    Multimedia Appendix 4

    Content categories of patient and health care professional responses to the open-ended question on what content was missing from the item sets.

    DOCX File , 23 KB
    Multimedia Appendix 5

    Proportions of participants who had no difficulties with comprehensibility of item per item, ordered by subdomain.

    DOCX File , 21 KB
    Multimedia Appendix 6

    Resulting Dutch item versions before and after the first 2 interview rounds.

    DOCX File , 34 KB
    Multimedia Appendix 7

    Final core the ESM-AC (Experience Sampling Method for People Living With Advanced Cancer) questionnaire.

    DOCX File , 25 KB
    Multimedia Appendix 8

    Dutch onboarding session manual created after interview round 3.

    DOCX File , 19 KB
    1. van Roij J, Fransen H, van de Poll-Franse L, Zijlstra M, Raijmakers N. Measuring health-related quality of life in patients with advanced cancer: a systematic review of self-administered measurement instruments. Qual Life Res. Aug 2018;27(8):1937-1955. [FREE Full text] [CrossRef] [Medline]
    2. Miranda R, Raemdonck E, Deliens L, Kaasa S, Zimmermann C, Rodin G, et al. Do cancer centres and palliative care wards routinely measure patients' quality of life? An international cross-sectional survey study. Support Care Cancer. Jul 31, 2023;31(8):499. [FREE Full text] [CrossRef] [Medline]
    3. Osoba D. Health-related quality of life and cancer clinical trials. Ther Adv Med Oncol. Mar 2011;3(2):57-71. [FREE Full text] [CrossRef] [Medline]
    4. Van den Bergh O, Walentynowicz M. Accuracy and bias in retrospective symptom reporting. Curr Opin Psychiatry. Sep 2016;29(5):302-308. [CrossRef] [Medline]
    5. Arizmendi C, Wang S, Kaplan S, Weinfurt K. Evaluating recall periods for patient-reported outcome measures: a systematic review of quantitative methods. Value Health. Apr 2024;27(4):518-526. [FREE Full text] [CrossRef] [Medline]
    6. Larson R, Csikszentmihalyi M. The experience sampling method. In: Csikszentmihalyi M, editor. Flow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi. Cham, Switzerland. Springer; 2014:21-34.
    7. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol. 2008;4:1-32. [FREE Full text] [CrossRef] [Medline]
    8. Wrzus C, Neubauer AB. Ecological momentary assessment: a meta-analysis on designs, samples, and compliance across research fields. Assessment. Apr 2023;30(3):825-846. [FREE Full text] [CrossRef] [Medline]
    9. Geeraerts J, de Nooijer K, Pivodic L, De Ridder M, Van den Block L. Intensive longitudinal methods among adults with breast or lung cancer: scoping review. J Med Internet Res. Jun 12, 2024;26:e50224. [FREE Full text] [CrossRef] [Medline]
    10. Houben M, Van Den Noortgate W, Kuppens P. The relation between short-term emotion dynamics and psychological well-being: a meta-analysis. Psychol Bull. Jul 2015;141(4):901-930. [FREE Full text] [CrossRef] [Medline]
    11. Kuppens P, Verduyn P. Emotion dynamics. Curr Opin Psychol. Oct 2017;17:22-26. [FREE Full text] [CrossRef] [Medline]
    12. Kampshoff CS, Verdonck-de Leeuw IM, van Oijen MG, Sprangers MA, Buffart LM. Ecological momentary assessments among patients with cancer: a scoping review. Eur J Cancer Care (Engl). May 14, 2019;28(3):e13095. [FREE Full text] [CrossRef] [Medline]
    13. Paterson C, Armitage L, Turner M. Current landscape of ecological momentary assessment (real-time data) methodology in cancer research: a systematic review. Semin Oncol Nurs. Dec 2023;39(6):151514. [FREE Full text] [CrossRef] [Medline]
    14. Henson LA, Maddocks M, Evans C, Davidson M, Hicks S, Higginson IJ. Palliative care and the management of common distressing symptoms in advanced cancer: pain, breathlessness, nausea and vomiting, and fatigue. J Clin Oncol. Mar 20, 2020;38(9):905-914. [FREE Full text] [CrossRef]
    15. Beernaert K, Pardon K, Van den Block L, Devroey D, De Laat M, Geboes K, et al. Palliative care needs at different phases in the illness trajectory: a survey study in patients with cancer. Eur J Cancer Care (Engl). Jul 2016;25(4):534-543. [FREE Full text] [CrossRef] [Medline]
    16. Myin-Germeys I, Kuppens P. The Open Handbook of Experience Sampling Methodology: A Step-by-Step Guide to Designing, Conducting, and Analyzing ESM Studies. Leuven, Belgium. Center for Research on Experience Sampling and Ambulatory Methods Leuven (REAL); 2021.
    17. Thong MS, Chan RJ, van den Hurk C, Fessele K, Tan W, Poprawski D, et al. MASCC EMA working group. Going beyond (electronic) patient-reported outcomes: harnessing the benefits of smart technology and ecological momentary assessment in cancer survivorship research. Support Care Cancer. Jan 2021;29(1):7-10. [FREE Full text] [CrossRef] [Medline]
    18. Steffen LE, Cheavens JS, Vowles KE, Gabbard J, Nguyen H, Gan GN, et al. Hope-related goal cognitions and daily experiences of fatigue, pain, and functional concern among lung cancer patients. Support Care Cancer. Feb 2020;28(2):827-835. [FREE Full text] [CrossRef] [Medline]
    19. Hacker ED, Kim I, Park C, Peters T. Real-time fatigue and free-living physical activity in hematopoietic stem cell transplantation cancer survivors and healthy controls: a preliminary examination of the temporal, dynamic relationship. Cancer Nurs. 2017;40(4):259-268. [FREE Full text] [CrossRef] [Medline]
    20. Vehling S, Gerstorf D, Schulz-Kindermann F, Oechsle K, Philipp R, Scheffold K, et al. The daily dynamics of loss orientation and life engagement in advanced cancer: a pilot study to characterise patterns of adaptation at the end of life. Eur J Cancer Care (Engl). Jul 2018;27(4):e12842. [FREE Full text] [CrossRef] [Medline]
    21. Badr H, Laurenceau JP, Schart L, Basen-Engquist K, Turk D. The daily impact of pain from metastatic breast cancer on spousal relationships: a dyadic electronic diary study. Pain. Dec 2010;151(3):644-654. [FREE Full text] [CrossRef] [Medline]
    22. Campbell GB, Belcher SM, Lee YJ, Courtney-Brooks M, Bovbjerg DH, Bizhanova Z, et al. Intensive daily symptom and function monitoring is feasible and acceptable to women undergoing first-line chemotherapy for gynecologic cancer. Cancer Nurs. Jan 13, 2022;45(5):369-377. [FREE Full text] [CrossRef] [Medline]
    23. Buck R, Morley S. A daily process design study of attentional pain control strategies in the self-management of cancer pain. Eur J Pain. Jul 2006;10(5):385-398. [FREE Full text] [CrossRef] [Medline]
    24. Badr H, Pasipanodya EC, Laurenceau JP. An electronic diary study of the effects of patient avoidance and partner social constraints on patient momentary affect in metastatic breast cancer. Ann Behav Med. Apr 2013;45(2):192-202. [FREE Full text] [CrossRef] [Medline]
    25. Stephenson E, DeLongis A, Bruel B, Badr H. Outpatient pain medication use: an electronic daily diary study in metastatic breast cancer. J Pain Symptom Manage. Apr 2018;55(4):1131-1137. [FREE Full text] [CrossRef] [Medline]
    26. Carson JW, Carson K, Olsen M, Sanders L, Westbrook K, Keefe FJ, et al. Yoga practice predicts improvements in day-to-day pain in women with metastatic breast cancer. J Pain Symptom Manage. Jun 2021;61(6):1227-1233. [FREE Full text] [CrossRef] [Medline]
    27. LeBaron V, Homdee N, Ogunjirin E, Patel N, Blackhall L, Lach J. Describing and visualizing the patient and caregiver experience of cancer pain in the home context using ecological momentary assessments. Digit Health. 2023;9:20552076231194936. [FREE Full text] [CrossRef] [Medline]
    28. Lim E, Boyle F, Okera M, Loi S, Goksu SS, van Hal G, et al. An open label, randomized phase 2 trial assessing the impact of food on the tolerability of abemaciclib in patients with advanced breast cancer. Breast Cancer Res Treat. Oct 2022;195(3):275-287. [FREE Full text] [CrossRef] [Medline]
    29. Schuler T, King C, Matsveru T, Back M, Clark K, Chin D, et al. Wearable-triggered ecological momentary assessments are feasible in people with advanced cancer and their family caregivers: feasibility study from an outpatient palliative care clinic at a cancer center. J Palliat Med. Jul 2023;26(7):980-985. [FREE Full text] [CrossRef] [Medline]
    30. Weaver A, Love S, Larsen M, Shanyinde M, Waters R, Grainger L, et al. A pilot study: dose adaptation of capecitabine using mobile phone toxicity monitoring - supporting patients in their homes. Support Care Cancer. Oct 2014;22(10):2677-2685. [FREE Full text] [CrossRef] [Medline]
    31. Terwee CB, Prinsen CA, Chiarotto A, Westerman M, Patrick D, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. May 2018;27(5):1159-1170. [FREE Full text] [CrossRef] [Medline]
    32. Degroote L, DeSmet A, De Bourdeaudhuij I, Van Dyck D, Crombez G. Content validity and methodological considerations in ecological momentary assessment studies on physical activity and sedentary behaviour: a systematic review. Int J Behav Nutr Phys Act. Mar 10, 2020;17(1):35. [FREE Full text] [CrossRef] [Medline]
    33. Stone AA, Schneider S, Smyth JM. Evaluation of pressing issues in ecological momentary assessment. Annu Rev Clin Psychol. May 09, 2023;19:107-131. [FREE Full text] [CrossRef] [Medline]
    34. Riihimäki M, Thomsen H, Sundquist K, Sundquist J, Hemminki K. Clinical landscape of cancer metastases. Cancer Med. Nov 2018;7(11):5534-5542. [FREE Full text] [CrossRef] [Medline]
    35. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global Cancer Statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. May 2021;71(3):209-249. [FREE Full text] [CrossRef] [Medline]
    36. Gallicchio L, Devasia TP, Tonorezos E, Mollica MA, Mariotto A. Estimation of the number of individuals living with metastatic cancer in the United States. J Natl Cancer Inst. Nov 14, 2022;114(11):1476-1483. [FREE Full text] [CrossRef] [Medline]
    37. Hamer J, McDonald R, Zhang L, Verma S, Leahey A, Ecclestone C, et al. Quality of life (QOL) and symptom burden (SB) in patients with breast cancer. Support Care Cancer. Feb 2017;25(2):409-419. [FREE Full text] [CrossRef] [Medline]
    38. Su Z, Zhou Y, Han X, Pang Y, He S, Tang L. Symptom burden in advanced breast cancer patients and its association between death anxiety and psychological distress. Chin J Cancer Res. Jun 30, 2022;34(3):298-308. [FREE Full text] [CrossRef] [Medline]
    39. Sung MR, Patel MV, Djalalov S, Le LW, Shepherd FA, Burkes RL, et al. Evolution of symptom burden of advanced lung cancer over a decade. Clin Lung Cancer. May 2017;18(3):274-80.e6. [FREE Full text] [CrossRef] [Medline]
    40. Morrison EJ, Novotny PJ, Sloan JA, Yang P, Patten CA, Ruddy KJ, et al. Emotional problems, quality of life, and symptom burden in patients with lung cancer. Clin Lung Cancer. Sep 2017;18(5):497-503. [FREE Full text] [CrossRef] [Medline]
    41. Chen Z, He G, Zhao Y, Han C, Xu L, Jian H, et al. Symptom burden and emotional distress in advanced lung cancer: the moderating effects of physicians' communication skills and patients' disease understanding. Support Care Cancer. Nov 2022;30(11):9497-9505. [FREE Full text] [CrossRef] [Medline]
    42. Kirtley O, Hiekkaranta AP, Kunkels Y, Verhoeven D, Van NM, Myin-Germeys I. The experience sampling method. ESM Item Repository. 2019. URL: https://www.esmitemrepository.com/ [accessed 2024-04-29]
    43. Sprangers MA, Cull A, Bjordal K, Groenvold M, Aaronson NK. The European Organization for Research and treatment of cancer approach to quality of life assessment: guidelines for developing questionnaire modules. Qual Life Res. Aug 1993;2(4):287-295. [CrossRef]
    44. Zeng L, Bedard G, Cella D, Thavarajah N, Chen E, Zhang L, et al. Preliminary results of the generation of a shortened quality-of-life assessment for patients with advanced cancer: the FACIT-Pal-14. J Palliat Med. May 2013;16(5):509-515. [CrossRef] [Medline]
    45. Groenvold M, Petersen MA, Aaronson NK, Arraras JI, Blazeby JM, Bottomley A, et al. EORTC Quality of Life Group. The development of the EORTC QLQ-C15-PAL: a shortened questionnaire for cancer patients in palliative care. Eur J Cancer. Jan 2006;42(1):55-64. [FREE Full text] [CrossRef] [Medline]
    46. Mestdagh M, Verdonck S, Piot M, Niemeijer K, Kilani G, Tuerlinckx F, et al. m-Path: an easy-to-use and highly tailorable platform for ecological momentary assessment and intervention in behavioral research and clinical practice. Front Digit Health. 2023;5:1182175. [FREE Full text] [CrossRef] [Medline]
    47. ISO 9241-11:2018: ergonomics of human-system interaction: part 11: usability: definitions and concepts. ISO. 2018. URL: https://www.iso.org/standard/63500.html [accessed 2024-04-29]
    48. Hwang W, Salvendy G. Number of people required for usability evaluation: the 10±2 rule. Commun ACM. May 2010;53(5):130-133. [FREE Full text] [CrossRef]
    49. Pallis A, Ring A, Fortpied C, Penninckx B, Van Nes MC, Wedding U, et al. European Organisation for Research and Treatment of Cancer Elderly Task Force. EORTC workshop on clinical trial methodology in older individuals with a diagnosis of solid tumors. Ann Oncol. Aug 2011;22(8):1922-1926. [FREE Full text] [CrossRef] [Medline]
    50. Balducci L. Geriatric oncology: challenges for the new century. Eur J Cancer. Sep 2000;36(14):1741-1754. [CrossRef] [Medline]
    51. Aaronson NK, Ahmedzai S, Bergman B, Bullinger M, Cull A, Duez NJ, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst. Mar 03, 1993;85(5):365-376. [FREE Full text] [CrossRef] [Medline]
    52. Lyons KD, Bakitas M, Hegel MT, Hanscom B, Hull J, Ahles TA. Reliability and validity of the Functional Assessment of Chronic Illness Therapy-Palliative care (FACIT-Pal) scale. J Pain Symptom Manage. Jan 2009;37(1):23-32. [FREE Full text] [CrossRef] [Medline]
    53. Murtagh FE, Ramsenthaler C, Firth A, Groeneveld EI, Lovell N, Simon ST, et al. A brief, patient- and proxy-reported outcome measure in advanced illness: validity, reliability and responsiveness of the Integrated Palliative care Outcome Scale (IPOS). Palliat Med. Sep 12, 2019;33(8):1045-1057. [FREE Full text] [CrossRef] [Medline]
    54. Russell JA. A circumplex model of affect. J Pers Soc Psychol. Dec 1980;39(6):1161-1178. [FREE Full text] [CrossRef]
    55. Fayers P, Bottomley A, EORTC Quality of Life Group, Quality of Life Unit. Quality of life research within the EORTC-the EORTC QLQ-C30. European Organisation for Research and Treatment of Cancer. Eur J Cancer. Mar 2002;38 Suppl 4:S125-S133. [FREE Full text] [CrossRef] [Medline]
    56. Reynoldson C, Stones C, Allsop M, Gardner P, Bennett MI, Closs SJ, et al. Assessing the quality and usability of smartphone apps for pain self-management. Pain Med. Jun 2014;15(6):898-909. [FREE Full text] [CrossRef] [Medline]
    57. Busch PA, Hausvik GI, Ropstad OK, Pettersen D. Smartphone usage among older adults. Comput Human Behav. Aug 2021;121:106783. [FREE Full text] [CrossRef]
    58. Jobe JB, Mingay DJ. Cognitive research improves questionnaires. Am J Public Health. Aug 1989;79(8):1053-1055. [FREE Full text] [CrossRef] [Medline]
    59. Brooke J. SUS: A 'Quick and Dirty' Usability Scale. Boca Raton, FL. CRC Press; 1996.
    60. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact. Jul 30, 2008;24(6):574-594. [FREE Full text] [CrossRef]
    61. Geeraerts J, Pivodic L, De Nooijer K, Naert E, Crombez G, De Ridder M, et al. Investigating experiences of people with advanced breast or lung cancer in their natural context: protocol for an experience sampling study. BMJ Open. Feb 14, 2024;14(2):e075752. [FREE Full text] [CrossRef] [Medline]
    62. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. Nov 2005;15(9):1277-1288. [FREE Full text] [CrossRef] [Medline]
    63. Kimhy D, Myin-Germeys I, Palmier-Claus J, Swendsen J. Mobile assessment guide for research in schizophrenia and severe mental disorders. Schizophr Bull. May 2012;38(3):386-395. [FREE Full text] [CrossRef] [Medline]
    64. Zhaoyang R, Sliwinski MJ, Martire LM, Smyth JM. Social interactions and physical symptoms in daily life: quality matters for older adults, quantity matters for younger adults. Psychol Health. Jul 2019;34(7):867-885. [FREE Full text] [CrossRef] [Medline]
    65. van Os J, Verhagen S, Marsman A, Peeters F, Bak M, Marcelis M, ESM-MERGE Investigators PhD, et al. The experience sampling method as an mHealth tool to support self-monitoring, self-insight, and personalized health care in clinical practice. Depress Anxiety. Jun 2017;34(6):481-493. [FREE Full text] [CrossRef] [Medline]
    66. de Vries LP, Baselmans BM, Bartels M. Smartphone-based ecological momentary assessment of well-being: a systematic review and recommendations for future studies. J Happiness Stud. 2021;22(5):2361-2408. [FREE Full text] [CrossRef] [Medline]
    67. ChatGPT. URL: https://chatgpt.com/ [accessed 2024-04-29]

    ‎
    COSMIN: Consensus-Based Standards for the Selection of Health Measurement Instruments
    EORTC: European Organization for Research and Treatment of Cancer
    EORTC QLQ-C30: European Organization for Research and Treatment of Cancer Core Quality of Life Questionnaire
    ESM: experience sampling method
    ESM-AC: Experience Sampling Method for People Living With Advanced Cancer
    FACIT-Pal: Functional Assessment of Chronic Illness Therapy–Palliative Care
    PROM: patient-reported outcome measure

    Edited by N Cahill; submitted 19.02.24; peer-reviewed by C Low, S Wang, A Hassan; comments to author 24.05.24; revised version received 12.06.24; accepted 02.10.24; published 05.11.24.

    Copyright

    ©Joran Geeraerts, Lara Pivodic, Lise Rosquin, Eline Naert, Geert Crombez, Mark De Ridder, Lieve Van den Block. Originally published in JMIR Cancer (https://cancer.jmir.org), 05.11.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on https://cancer.jmir.org/, as well as this copyright and license information must be included.