This paper is in the following e-collection/theme issue:

Viewpoints on Innovations in Cancer Care and Research (10) Clinical Information and Decision Making (1339) Decision Support for Health Professionals (1145) Innovations and Technology in Cancer Care (405)

Published on in Vol 8, No 2 (2022): Apr-Jun

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/31461, first published .
Physicians’ Perceptions of and Satisfaction With Artificial Intelligence in Cancer Treatment: A Clinical Decision Support System Experience and Implications for Low-Middle–Income Countries

Physicians’ Perceptions of and Satisfaction With Artificial Intelligence in Cancer Treatment: A Clinical Decision Support System Experience and Implications for Low-Middle–Income Countries

Physicians’ Perceptions of and Satisfaction With Artificial Intelligence in Cancer Treatment: A Clinical Decision Support System Experience and Implications for Low-Middle–Income Countries

Authors of this article:

Srinivas Emani1, 2 Author Orcid Image ;   Angela Rui1 Author Orcid Image ;   Hermano Alexandre Lima Rocha3, 4 Author Orcid Image ;   Rubina F Rizvi5 Author Orcid Image ;   Sergio Ferreira Juaçaba4, 6 Author Orcid Image ;   Gretchen Purcell Jackson7, 8 Author Orcid Image ;   David W Bates1, 9 Author Orcid Image
Article Authors Cited by (5) Tweetations (4) Metrics

    Viewpoint

    1Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States

    2Department of Behavioral, Social, and Health Education Sciences, Emory University, Atlanta, GA, United States

    3Department of Community Health, Federal University of Cearrá, Fortaleza, CE, Brazil

    4Instituto do Câncer do Ceará, Fortaleza, CE, Brazil

    5IBM Watson Health, IBM, Cambridge, MA, United States

    6Rodolfo Teofilo College, Fortaleza CE, Brazil

    7Intuitive Surgical, Sunnyvale, CA, United States

    8Departments of Pediatric Surgery, Pediatrics, and Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States

    9Department of Healthcare Policy and Management, Harvard School of Public Health, Boston, MA, United States

    *these authors contributed equally

    Corresponding Author:

    Srinivas Emani, MA, PhD

    Division of General Internal Medicine and Primary Care

    Brigham and Women's Hospital

    Harvard Medical School

    1620 Tremont Street, OBC-3

    Boston, MA, 02120

    United States

    Phone: 1 6177327063

    Email: semani1@partners.org


    As technology continues to improve, health care systems have the opportunity to use a variety of innovative tools for decision-making, including artificial intelligence (AI) applications. However, there has been little research on the feasibility and efficacy of integrating AI systems into real-world clinical practice, especially from the perspectives of clinicians who use such tools. In this paper, we review physicians’ perceptions of and satisfaction with an AI tool, Watson for Oncology, which is used for the treatment of cancer. Watson for Oncology has been implemented in several different settings, including Brazil, China, India, South Korea, and Mexico. By focusing on the implementation of an AI-based clinical decision support system for oncology, we aim to demonstrate how AI can be both beneficial and challenging for cancer management globally and particularly for low-middle–income countries. By doing so, we hope to highlight the need for additional research on user experience and the unique social, cultural, and political barriers to the successful implementation of AI in low-middle–income countries for cancer care.

    JMIR Cancer 2022;8(2):e31461

    doi:10.2196/31461

    Keywords

    artificial intelligence; cancer; low-middle–income countries; physicians; perceptions; Watson for Oncology; implementation; local context 


    The last several decades have witnessed the rapid growth of artificial intelligence (AI) applications in health care. AI is considered to comprise areas like machine learning, natural language processing, expert systems, and image and signal processing [1]. One group, who cited a study from Global Market Insights, noted that the use of AI in health care was expected to grow annually from 2016 to 2024, with expenditures increasing from US $760 million in 2016 to over US $10 billion in 2024 [2]. In a 2020 study, Global Market Insights noted that the AI in the health care market exceeded US $4 billion in 2020 and would grow at a compound annual growth rate of 33.7% between 2021 and 2027, with an expenditure of US $34.5 billion in 2027 [3]. This market growth has been accompanied by both national initiatives for AI and the rapid growth of academic literature on the use of AI in health care. For example, in India, an “AI for All” policy was established along with NITI (National Institution for Transforming India) Aayog—a Government of India think tank for formulating a national strategy for AI [4]. A bibliometric analysis of the literature reported in the Journal of Medical Internet Research found a growth rate of 45.15% in publications from 2014 to 2019, with 70.67% of all publications occurring in the same period [5]. This analysis also found the following top five health problems in the publications (in order of frequency): cancer, depression, Alzheimer disease, heart failure, and diabetes. Another review of AI applications in health care found the following areas of focus in the applications: sepsis, breast cancer, diabetic retinopathy, and polyps and adenomas [6]. Additionally, this review noted that the implementation of AI applications in real-world clinical settings is not widespread. Another recent review with a focus on patient safety outcomes also noted the lack of AI applications in real-world settings [7]. These articles, and others in the Journal of Medical Internet Research and elsewhere, have started to capture the use and role of AI in health care [8-11].

    In this viewpoint, we contribute to this growing literature by detailing physicians’ experiences with an AI application—Watson for Oncology (WfO)—in the treatment of cancer. Physicians’ experiences with WfO are especially relevant, as the application has been implemented in diverse, real-world social and cultural settings. Our summary of physicians’ experiences with WfO relies on the extensive, published literature on this topic. After we describe physicians’ experiences with WfO, we comment about the opportunities and challenges associated with using AI for cancer care in low-middle–income countries (LMICs).


    WfO is a therapeutic oncology clinical decision support system (CDSS) that was trained by experts from the Memorial Sloan Kettering Cancer Center [12]. WfO uses both natural language processing and machine learning to process structured and unstructured data about patients with cancer and generate therapeutic options based on available evidence [13]. WfO provides 3 categories of therapeutic options: “recommended” treatments are those that adhere to the preferred training approach of the Memorial Sloan Kettering Cancer Center, treatments “for consideration” refer to alternative treatments based on evidence, and “not recommended” treatments refer to those that are not appropriate for certain patients [14]. Many early adopters of WfO measured the degree to which WfO therapeutic options were concordant with either clinical practice or the decisions of a multidisciplinary tumor board. WfO concordance rates varied widely across countries for many reasons, including differences in standard treatment guidelines, resource availability, and physician or patient preferences [15]. It is well recognized that concordance studies do not measure system accuracy but instead assess agreement with decisions made in practice, which may or may not reflect evidence-based decisions [16].

    In this viewpoint, we focus on physicians’ perceptions of and satisfaction with WfO. We believe that an evaluation of physicians’ perceptions of this AI tool will provide valuable insights for the successful implementation of AI-based CDSSs for cancer treatment, especially in LMICs. Additionally, little is known about how physicians perceive the use of AI tools for cancer treatment. We present physicians’ perceptions of the advantages of, as well as the disadvantages and concerns with, AI in a real-world setting. Our summary relies on published literature on physicians’ perceptions of WfO implementation in a number of countries, including China, India, Mexico, South Korea, and Thailand. Multimedia Appendix 1 provides a comprehensive list of the studies on WfO [13-74].


    The positive perceptions of WfO relate to the system’s ability to aid clinicians during the therapeutic decision-making process by quickly providing relevant scientific evidence. In China, a satisfaction survey, which was completed by 51 oncologists who used WfO, found that 86.3% of oncologists approved the quality of WfO and 88.2% approved the comprehensibility of WfO’s treatment options, justifications, and external literature [17]. The clinicians rated WfO highly in terms of its ability to provide evidence-based medicine medical education (score: 8.1/10) and literature assistance (score: 7.7/10), assist in medical care quality control (score: 7.3/10), act as a second-opinion consultation resource (score: 7.0/10), perform case reviews with a tumor board (score: 6.9/10), and provide decision support (score: 6.4/10). Overall, the oncologists recommended using WfO as a CDSS to other clinicians (score: 7.3/10). At Shanghai Tenth People’s Hospital, the multiple disciplinary team (MDT) also used WfO and found that their treatment plans became “more standardized, reasonable, and personalized” [18].

    WfO’s ability to compare treatment options was tested in Mexico, where it was used for a total of 100 patient cases involving lung, breast, gastric, colon, and rectal cancers diagnosed within the last 5 years [19]. In terms of perceived utility, oncologists found WfO to be “very useful” in comparing treatment options. They reported that WfO might be especially valuable for individuals, such as medical students and residents who lack oncology experience, as well as clinics that do not have enough subspecialists. Several implementations of WfO in China indicate the role of WfO in enhancing the learning experience and efficiency of physicians, particularly junior physicians, and the facilitation of better diagnoses and treatment recommendations [20,21]. This perspective was also substantiated by students from Taipei Medical University Hospital in Taiwan who had limited clinical experience; by using WfO, they performed better on their colon cancer learning assessment than their peers who used traditional search methods and were more clinically experienced [22]. The study also found that students with less clinical experience felt that WfO was “clearer and more understandable” than information found through traditional methods.

    WfO’s links to recent and relevant scientific information may provide treatment information that clinicians may not know. In India, an MDT changed their treatment recommendations for 136 of 1000 cases of breast, lung, colon, and rectal cancers because of the data provided by WfO [23]. For 55% of those cases, WfO provided recent evidence of newer treatments. For 30% of the cases, WfO provided new information about genotypic and phenotypic data. For 15% of the cases, WfO provided information on evolving clinical experiences, which influenced the MDT to change their treatment decisions. These results demonstrate the potential of WfO to positively impact cancer outcomes by providing scientific evidence and up-to-date information on clinical guidelines. In a separate study that focused on adjuvant systemic therapy for breast carcinoma, treatment decisions were changed for 4 of 11 patients after the MDT reviewed WfO’s recommendations and EndoPredict (Myriad Genetics Inc) test reports [24]. WfO was able to aid clinicians in providing personalized cancer care while addressing the difficulties of staying informed on evolving cancer guidelines and studies.

    Another aspect that must be considered is whether WfO can be useful as a CDSS. At the Instituto Câncer do Ceará in Brazil, a majority of oncologists chose the “agree” or “strongly agree” option for statements that were used to confirm if WfO meets the “CDS Five Rights” criteria [25]. The “CDS Five Rights” contain clinical quality criteria for determining if a CDSS offers benefits that are optimal for a given setting [75]. In the study, 6 of the 7 oncologists at the Instituto Câncer do Ceará believed that WfO provided relevant information that resulted in action being taken and presented the information in a manner that positively aligned with their individual workflows. Further, 5 oncologists agreed that the additional details for each treatment option were easily comprehensible, and 4 oncologists agreed that WfO exceeded their expectations as a CDSS tool for patient management.


    Although WfO appears to be useful for displaying information in a succinct and timely manner, there are concerns regarding the system’s usability and integration into clinician workflows. First, at sites without integrated patient record systems, some users found manual data entry to be a burdensome process [13,26]. At Manipal Hospital in India, it was observed that acclimation to the system reduced the time needed for each patient case [27]. The mean time needed to collect and enter data for nonmetastatic diseases was 20 minutes. This was reduced to 12 minutes after an increased acquaintance of 10 cases with WfO. In comparison, the time needed to collect and enter data for metastatic diseases was 5 to 7 minutes longer than that for localized diseases. On average, WfO took a median of 40 seconds to capture, analyze, and provide treatment recommendations. For physicians with a high patient load, the time needed to enter information into the system may be an issue. Users also want WfO to provide an explanation of its process for scoring and ranking treatment options [26]. In doing so, users would feel more comfortable with trusting the information and recommendations provided by WfO.

    A second important concern that has been identified in studies is localizing WfO’s treatment recommendations to the country of implementation. In the previously mentioned satisfaction study conducted in China, 66.7% of physicians recommended that WfO should integrate data on locally available treatments to improve the system [17]. For example, WfO did not take into consideration whether the immunotherapy drugs it recommended had been approved by the China Food and Drug Administration. Physicians also chose chemotherapy instead of WfO’s recommended medication because the medication was too expensive for patients. Similar challenges were found for WfO users in Mexico and Thailand [19,28]. In Mexico, clinicians deviated from WfO’s recommendations due to the high costs associated with them and the fact that they did not adhere to Mexican cancer treatment guidelines [19]. In Thailand, oncologists preferred basing their treatment recommendations on other countries’ guidelines instead of US guidelines [28].


    In 2012, 65% of all cancer deaths worldwide occurred in LMICs, and the projection for 2030 is that this will increase to 75% [76]. LMICs may also be experiencing an even higher burden from cancer than that experienced by high-income countries (HICs) for several reasons. LMICs have restrained funding and often lack optimal cancer registries and surveillance data; thus, they are unable to implement evidence-based cancer control programs [76]. Treatment modalities are also more limited in LMICs than in HICs; radiotherapy and chemotherapy are available in 43% to 51% of LMICs but are available in 94% of HICs [77]. However, there is a high demand for such therapies, as 5 million new people annually are estimated to need radiation therapy in LMICs [78]. LMICs also lack specialized medical personnel, such as oncologists and oncology nurses, who are needed to address those affected by cancer in LMICs [79]. According to a World Health Organization report, LMICs have the lowest density of health care workers in comparison to HICs, where the density of health care workers is significantly higher [80]. A lack of health care workers for serving the population makes offering high-quality, personalized care a difficult task.

    Oncologists also often require the expertise of their colleagues and additional literary resources to determine a course of treatment for unique cancer cases. Gaining access to high-quality medical information is key for creating an appropriate treatment plan, but oncologists may need additional help with sorting information that is both relevant to their patients and viable in terms of what resources are available. AI-based platforms such as WfO may be able to address the growing challenges of providing cancer treatment plans in LMICs. AI can address issues of access to knowledge bases in a comprehensive and easy-to-access manner. The ability of AI tools to quickly provide evidence-based cancer treatment options would be especially helpful in low-resource settings where the lack of time, expertise, and other needed resources can become a barrier to providing care. Using AI in this manner may also promote international partnerships on cancer therapy research and standardize guidelines for certain cancer types. The studies reviewed in this viewpoint demonstrate the potential of AI to reduce the cognitive burdens of less experienced physicians who would benefit from additional medical education resources.

    The experiences with WfO in different settings also reveal a positive perception of AI with regard to its ability to reassure clinicians and confirm their interpretations of data and the potential of such a tool to do so in an LMIC. The ability of AI to act as a second opinion resource and standardize treatments may prove especially useful for cancer care in LMICs where the likelihood of receiving comprehensive care and achieving positive outcomes is lower than that in HICs. A lack of available specialized medical personnel in LMICs, especially in rural regions, is one of the factors contributing to poor cancer outcomes in LMICs [81]. The ability of AI tools, such as WfO, to provide subspecialty treatment information makes such tools a much-needed resource that existing physicians can use to meet population demands, especially in rural areas, as envisioned for the use of AI in an LMIC like India [4,82]. Approximately 60 oncologists serve over 300 million people in West Africa, and only 2000 oncologists are available for 10 million patients in India [83-85]. WfO’s open-access information, which can be used to supplement self-paced learning, would be an ideal resource for cancer physicians in LMICs where medical education resources are lacking [86,87].

    The use of an AI tool such as WfO in LMICs also poses certain challenges. The technological challenges that are unique to LMICs and should be mentioned include access to the internet, technology training, and whether local technology teams would be able to address technical issues [87]. Providing a decision support tool, such as WfO, that is user-friendly and aligns with daily workflows is essential for implementation in LMICs, where physicians’ experiences with technology can vary [87]. Additionally, there is concern about whether AI tools would exacerbate the divide in health care access and use, especially with respect to socioeconomic status. There is a fear that AI would recommend treatment options that patients cannot afford or that only high-income and educated patients who are aware of AI tools may benefit from the use of AI in LMICs [4,19,28] Another important concern—one that applies to a tool such as WfO—is that the data and training of AI tools may not incorporate patient characteristics into treatment recommendations. For example, in China, local patient characteristics such as gene mutations and the weaker physiques of Chinese patients, which can influence treatment recommendations, were not accounted for in WfO recommendations [20]. Similarly, the need to consider the presence of multiple ethnic groups in countries like India during the implementation of AI tools developed by Western countries will be an important factor to address in LMICs [88].


    It is undeniable that oncology physicians in LMICs need as much additional support as possible. The implementation of AI tools, such as WfO, in different settings has revealed that access to a second opinion CDSS resource, concise scientific evidence, and international clinical guidelines can help physicians feel more confident in their final treatment decisions. To improve the clinical utility of AI tools such as WfO, it is necessary that the experiences and satisfaction of physicians who use such tools are explored more in-depth, especially those of physicians in LMICs. These perspectives are especially key to tailoring AI systems for use in real-world clinical settings [6,7]. Such perspectives are of course embedded in the local social, cultural, and political LMIC contexts within which AI is implemented and the ways in which local contexts can shape the use of AI. We are gaining experience with respect to the implementation of AI tools, such as WfO, in real-world settings for the treatment of cancer. However, we still need to address some of the challenges in the “last mile” stage of implementation, specifically those related to local contexts [89].

    Acknowledgments

    This work was supported by IBM Watson Health (Cambridge, Massachusetts), which is not responsible for the content or recommendations made.

    Conflicts of Interest

    SE, AR, and DWB received salary support from a grant funded by IBM Watson Health. DWB has received research support from and consults for EarlySense, which makes patient safety monitoring systems. He receives cash compensation from the Center for Digital Innovation (Negev), which is a not-for-profit incubator for health information technology start-ups. He receives equity from ValeraHealth, which makes software to help patients with chronic diseases; Clew, which makes software to support clinical decision-making in intensive care; and MDClone, which takes clinical data and produces deidentified versions of them. He consults for and receives equity from AESOP, which makes software to reduce medication error rates, and FeelBetter. He has received research support from MedAware. RFR is employed by IBM Watson Health. GPJ was employed by IBM Watson Health at the time of manuscript submission and GPJ's compensation included salary and equity. All other authors declare no conflicts of interests.

    ‎
    Multimedia Appendix 1

    Studies on Watson for Oncology (WfO).

    DOCX File , 23 KB
    1. Wahl B, Cossy-Gantner A, Germann S, Schwalbe NR. Artificial intelligence (AI) and global health: how can AI contribute to health in resource-poor settings? BMJ Glob Health 2018 Aug 29;3(4):e000798 [FREE Full text] [CrossRef] [Medline]
    2. Healthcare artificial intelligence outlook: Benefits, projected growth and challenges. The SSI Group.   URL: https:/​/thessigroup.​com/​healthcare-artificial-intelligence-outlook-benefits-projected-growth-challenges/​ [accessed 2022-01-04]
    3. Artificial intelligence in healthcare market size by application (medical imaging and diagnosis, drug discovery, therapy planning, hospital workflow, wearables, virtual assistants), COVID-19 impact analysis, regional outlook, application potential, price trends, competitive market share and forecast, 2021 - 2027. Global Market Insights. 2020.   URL: https://www.gminsights.com/industry-analysis/healthcare-artificial-intelligence-market [accessed 2022-01-04]
    4. Pradhan K, John P, Sandhu N. Use of artificial intelligence in healthcare delivery in India. J Hosp Manag Health Policy 2021 Sep;5:1-10 [FREE Full text] [CrossRef]
    5. Guo Y, Hao Z, Zhao S, Gong J, Yang F. Artificial intelligence in health care: Bibliometric analysis. J Med Internet Res 2020 Jul 29;22(7):e18228 [FREE Full text] [CrossRef] [Medline]
    6. Yin J, Ngiam KY, Teo HH. Role of artificial intelligence applications in real-life clinical practice: Systematic review. J Med Internet Res 2021 Apr 22;23(4):e25759 [FREE Full text] [CrossRef] [Medline]
    7. Choudhury A, Asan O. Role of artificial intelligence in patient safety outcomes: Systematic literature review. JMIR Med Inform 2020 Jul 24;8(7):e18599 [FREE Full text] [CrossRef] [Medline]
    8. Iqbal U, Celi LA, Li YCJ. How can artificial intelligence make medicine more preemptive? J Med Internet Res 2020 Aug 11;22(8):e17211 [FREE Full text] [CrossRef] [Medline]
    9. Lee J. Is artificial intelligence better than human clinicians in predicting patient outcomes? J Med Internet Res 2020 Aug 26;22(8):e19918 [FREE Full text] [CrossRef] [Medline]
    10. Vuong QH, Ho MT, Vuong TT, La VP, Ho MT, Nghiem KCP, et al. Artificial intelligence vs. natural stupidity: Evaluating AI readiness for the Vietnamese Medical Information System. J Clin Med 2019 Feb 01;8(2):168 [FREE Full text] [CrossRef] [Medline]
    11. Hosny A, Aerts HJWL. Artificial intelligence for global health. Science 2019 Nov 22;366(6468):955-956 [FREE Full text] [CrossRef] [Medline]
    12. Luxton DD. Should Watson be consulted for a second opinion? AMA J Ethics 2019 Feb 01;21(2):E131-E137 [FREE Full text] [CrossRef] [Medline]
    13. Keikes L, Medlock S, van de Berg DJ, Zhang S, Guicherit OR, Punt CJA, et al. The first steps in the evaluation of a "black-box" decision support tool: a protocol and feasibility study for the evaluation of Watson for Oncology. J Clin Transl Res 2018 Jul 27;3(Suppl 3):411-423 [FREE Full text] [Medline]
    14. Lee WS, Ahn SM, Chung JW, Kim KO, Kwon KA, Kim Y, et al. Assessing concordance with Watson for Oncology, a cognitive computing decision support system for colon cancer treatment in Korea. JCO Clin Cancer Inform 2018 Dec;2:1-8 [FREE Full text] [CrossRef] [Medline]
    15. Liang J, Li T, Zhang SS, Chen C, VanHouten C, Preininger A, et al. Reasons for discordance in treatment approaches between oncology practice and clinical decision support in China. J Clin Oncol 2019 May 20;37(15_suppl):6555. [CrossRef]
    16. Suwanvecho S, Suwanrusme H, Jirakulaporn T, Issarachai S, Taechakraichana N, Lungchukiet P, et al. Comparison of an oncology clinical decision-support system's recommendations with actual treatment decisions. J Am Med Inform Assoc 2021 Mar 18;28(4):832-838 [FREE Full text] [CrossRef] [Medline]
    17. Li T, Chen C, Zhang SS, Dankwa-Mullan I, Chen A, Preininger A, et al. Deployment and integration of a cognitive technology in China: Experiences and lessons learned. J Clin Oncol 2019 May 20;37(15_suppl):6538. [CrossRef]
    18. Fang J, Zhu Z, Wang H, Hu F, Liu Z, Guo X, et al. The establishment of a new medical model for tumor treatment combined with Watson for Oncology, MDT and patient involvement. J Clin Oncol 2018 Jun 01;36(15_suppl):e18504. [CrossRef]
    19. Sarre-Lazcano C, Alonso AA, Melendez FDH, Arrieta O, Norden AD, Urman A, et al. Cognitive computing in oncology: A qualitative assessment of IBM Watson for Oncology in Mexico. J Clin Oncol 2017 May 30;35(15_suppl):e18166. [CrossRef]
    20. Liu C, Liu X, Wu F, Xie M, Feng Y, Hu C. Using artificial intelligence (Watson for Oncology) for treatment recommendations amongst Chinese patients with lung cancer: Feasibility study. J Med Internet Res 2018 Sep 25;20(9):e11087 [FREE Full text] [CrossRef] [Medline]
    21. Tian Y, Liu X, Wang Z, Cao S, Liu Z, Ji Q, et al. Concordance between Watson for Oncology and a multidisciplinary clinical decision-making team for gastric cancer and the prognostic implications: Retrospective study. J Med Internet Res 2020 Feb 20;22(2):e14122 [FREE Full text] [CrossRef] [Medline]
    22. Chen CY, Hung HC, Chiu HY, Wei PL, Kuo PL, Chiou JF, et al. Enhancing evidence-based medicine skills in oncology training with cognitive technology. J Clin Oncol 2019 May 26;37(15_suppl):10532. [CrossRef]
    23. Somashekhar SP, Sepúlveda MJ, Shortliffe EH, C RK, Rauthan A, Patil P, et al. A prospective blinded study of 1000 cases analyzing the role of artificial intelligence: Watson for oncology and change in decision making of a Multidisciplinary Tumor Board (MDT) from a tertiary care cancer center. J Clin Oncol 2019 May 26;37(15_suppl):6533. [CrossRef]
    24. Somashekhar SP, Yethadka R, C RK, Rajgopal AK, Rauthan A, Patil P. Triple blinded prospective study assessing the impact of genomics: EndoPredict and artificial intelligence Watson for Oncology (WFO) on MDT’s decision of adjuvant systemic therapy for hormone receptor positive early breast carcinoma. J Clin Oncol 2019 May 26;37(15_suppl):e18013. [CrossRef]
    25. Rocha HAL, Dankwa-Mullan I, Juacaba SF, Preininger A, Felix W, Thompson JV, et al. An evaluation of artificial intelligence-based clinical decision supports use in Brazil. J Clin Oncol 2019 May 26;37(15_suppl):e18081. [CrossRef]
    26. Zauderer MG, Gucalp A, Epstein AS, Seidman AD, Caroline A, Granovsky S, et al. Piloting IBM Watson Oncology within Memorial Sloan Kettering’s regional network. J Clin Oncol 2014 May 20;32(15_suppl):e17653. [CrossRef]
    27. Somashekhar SP, Kumarc R, Rauthan A, Arun KR, Ramya YE. Abstract S6-07: Double blinded validation study to assess performance of IBM artificial intelligence platform, Watson for oncology in comparison with Manipal multidisciplinary tumour board – First study of 638 breast cancer cases. Cancer Res 2017;77(4_Supplement):S6-07. [CrossRef]
    28. Suwanvecho S, Suwanrusme H, Sangtian M, Norden AD, Urman A, Hicks A, et al. Concordance assessment of a cognitive computing system in Thailand. J Clin Oncol 2017 May 30;35(15_suppl):6589. [CrossRef]
    29. Zhang XC, Zhou N, Zhang CT, Lv HY, Li TJ, Zhu JJ, et al. Concordance study between IBM Watson for Oncology (WFO) and clinical practice for breast and lung cancer patients in China. Ann Oncol 2017 Nov 01;28(Supplement 10):x170. [CrossRef]
    30. Bach P, Zauderer MG, Gucalp A, Epstein AS, Norton L, Seidman AD, et al. Beyond Jeopardy!: Harnessing IBM's Watson to improve oncology decision making. J Clin Oncol 2013 May 20;31(15_suppl):6508. [CrossRef]
    31. Epstein AS, Zauderer MG, Gucalp A, Seidman AD, Caroline A, Fu J, et al. Next steps for IBM Watson Oncology: Scalability to additional malignancies. J Clin Oncol 2014 May 20;32(15_suppl):6618. [CrossRef]
    32. Fu J, Gucalp A, Zauderer MG, Epstein AS, Kris MG, Keesing J, et al. Steps in developing Watson for Oncology, a decision support system to assist physicians choosing first-line metastatic breast cancer (MBC) therapies: Improved performance with machine learning. J Clin Oncol 2015 May 20;33(15_suppl):566. [CrossRef]
    33. Kris MG, Gucalp A, Epstein AS, Seidman AD, Fu J, Keesing J, et al. Assessing the performance of Watson for oncology, a decision support system, using actual contemporary clinical cases. J Clin Oncol 2015 May 20;33(15_suppl):8023. [CrossRef]
    34. Seidman AD, Pilewskie ML, Robson ME, Kelvin JF, Zauderer MG, Epstein AS, et al. Integration of multi-modality treatment planning for early stage breast cancer (BC) into Watson for Oncology, a Decision Support System: Seeing the forest and the trees. J Clin Oncol 2015 May 20;33(15_suppl):e12042. [CrossRef]
    35. Herath DH, Wilson-Ing D, Ramos E, Morstyn G. Assessing the natural language processing capabilities of IBM Watson for oncology using real Australian lung cancer cases. J Clin Oncol 2016 May 20;34(15_suppl):e18229. [CrossRef]
    36. Somashekhar SSP, Kumar R, Kumar A, Patil P, Rauthan A. 551PD Validation study to assess performance of IBM cognitive computing system Watson for oncology with Manipal multidisciplinary tumour board for 1000 consecutive cases: An Indian experience. Ann Oncol 2016 Dec;27:ix179. [CrossRef]
    37. Zhou N, Lv H, Zhang C, Li T, Zhu J, Jiang M, et al. P1.01-069 Clinical experience with IBM Watson for Oncology (WFO) cognitive system for lung cancer treatment in China. J Thorac Oncol 2017 Nov;12(11):S1921. [CrossRef]
    38. Yue L, Yang L. Clinical experience with IBM Watson for Oncology (WFO) for multiple types of cancer patients in China. Ann Oncol 2017 Nov;28:x162. [CrossRef]
    39. Somashekhar S, Sepúlveda M, Norden AD, Rauthan A, Arun K, Patil P, et al. Early experience with IBM Watson for Oncology (WFO) cognitive computing system for lung and colorectal cancer treatment. J Clin Oncol 2017 May 30;35(15_suppl):8527. [CrossRef]
    40. Baek JH, Ahn SM, Urman A, Kim YS, Ahn HK, Won PS, et al. Use of a cognitive computing system for treatment of colon and gastric cancer in South Korea. J Clin Oncol 2017 May 30;35(15_suppl):e18204. [CrossRef]
    41. Lim S, Lee KB. Use of a cognitive computing system for treatment of cervical cancer. J Gynecol Oncol 2017 Sep;28(5):e67 [FREE Full text] [CrossRef] [Medline]
    42. Zhou N, Zhang CT, Lv HY, Hao CX, Li TJ, Zhu JJ, et al. Concordance study between IBM Watson for Oncology and clinical practice for patients with cancer in China. Oncologist 2019 Jun;24(6):812-819 [FREE Full text] [CrossRef] [Medline]
    43. Chen PJ, Sun TT, Li TI, Dankwa-Mullan I, Urman A, Wang CK, et al. Can AI technology augment tumor board treatment decisions for stage II colon cancer care? J Clin Oncol 2018 Jun 01;36(15_suppl):e18582. [CrossRef]
    44. Wu A, Chen PJ, Li TI, Dankwa-Mullan I, Sun TT, Rhee K. Real world survival outcomes in patients with high risk stage II colon cancer at a Beijing Cancer Hospital. J Clin Oncol 2018 May 20;36(15_suppl):e15670-e15670. [CrossRef]
    45. Jiang Z, Xu F, Sepúlveda M, Li J, Wang H, Liu Z, et al. Concordance, decision impact and guidelines adherence using artificial intelligence in high-risk breast cancer. J Clin Oncol 2018 Jun 01;36(15_suppl):e18566. [CrossRef]
    46. Yuwen D, Zhang W, Wu J, Zhang J, Shen Y, Shi J, et al. Concordance evaluation of an artificial intelligence technology with a multidisciplinary tumor board in gastric cancer. J Clin Oncol 2018 Jun 01;36(15_suppl):e18569. [CrossRef]
    47. Somashekhar SP, Sepúlveda MJ, Puglielli S, Norden AD, Shortliffe EH, Rohit Kumar C, et al. Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board. Ann Oncol 2018 Feb 01;29(2):418-423 [FREE Full text] [CrossRef] [Medline]
    48. Kim YY, Oh SJ, Chun YS, Lee WK, Park HK. Gene expression assay and Watson for Oncology for optimization of treatment in ER-positive, HER2-negative breast cancer. PLoS One 2018 Jul 06;13(7):e0200100. [CrossRef] [Medline]
    49. Suwanrusme H, Issarachai S, Umsawasdi T, Suwanvecho S, Decha W, Dankwa-Mullan I, et al. Concordance assessment of a clinical decision support software in patients with solid tumors. J Clin Oncol 2018 Jun 01;36(15_suppl):e18584. [CrossRef]
    50. Simon G, DiNardo CD, Takahashi K, Cascone T, Powers C, Stevens R, et al. Applying artificial intelligence to address the knowledge gaps in cancer care. Oncologist 2019 Jun;24(6):772-782 [FREE Full text] [CrossRef] [Medline]
    51. Saiz FJS, Urman A, Sanders C, Britt MW, Nielsen R, Stevens RJ. IBM Watson Evidence Service (WES): A system for retrieval, summation and insight generation of relevant clinical evidence for personalized oncology. J Clin Oncol 2018 Jun 01;36(15_suppl):e18588. [CrossRef]
    52. Graham DMA, McNamara DM, Waintraub SE, Goldberg SL, Norden AD, Hervey J, et al. Are treatment recommendations provided by cognitive computing supported by real world data (Watson for Oncology with Cota RWE) concordant with expert opinions? Ann Oncol 2018 Oct;29:viii571. [CrossRef]
    53. Waintraub SE, McNamara DM, Graham DMA, Goldberg SL, Norden AD, Hervey J, et al. Can the cognitive computing system Watson for Oncology with Cota RWE help oncologists deliver subspecialist level care? 2018 Presented at: 36th Annual Chemotherapy Foundation Symposium (CFS); November 7-9, 2018; New York, New York.
    54. Rocha HAL, Dankwa-Mullan I, Juacaba SF, Willis V, Arriaga YE, Jackson GP, et al. Shared-decision making in prostate cancer with clinical decision-support. J Clin Oncol 2019 May 26;37(15_suppl):e16576. [CrossRef]
    55. Wang Z, Yu Z, Zhang X. Artificial intelligence-based clinical decision-support system improves cancer treatment and patient satisfaction. J Clin Oncol 2019 May 26;37(15_suppl):e18303. [CrossRef]
    56. Yu Z, Wang Z, Ren X, Lou D, Li X, Liu H, et al. Practical exploration and research of Watson for oncology clinical decision support system in real-world and localized practice. J Clin Oncol 2019 May 26;37(15_suppl):e18304. [CrossRef]
    57. Chen ZB, Chen SL, Liang RM, Peng ZW, Shen JX, Zhu WJ, et al. Can artificial intelligence support the clinical decision making for Barcelona clinic liver cancer stage 0/a hepatocellular carcinoma in China? J Clin Oncol 2019 May 26;37(15_suppl):e15634. [CrossRef]
    58. Chen S, Chen Z, Xiao H, Peng Z, Peng S, Kuang M. IDDF2019-ABS-0095 Can artificial intelligence support the clinical decision making for hepatocellular carcinoma? Gut 2019;68:A134. [CrossRef]
    59. Liang J, Tang C, Wang X, Guo Z, Ni J, Xu W, et al. Impact of decision-support system and guideline treatment concordance on response rate in advanced lung cancer. J Clin Oncol 2019 May 26;37(15_suppl):e20006. [CrossRef]
    60. Liang J, Dankwa-Mullan I, Ren Y, Chen A, Willis V, Jackson G, et al. Employing an oncology decision-support system to quantify treatment variation. J Clin Oncol 2019 May 26;37(15_suppl):e18067. [CrossRef]
    61. Fang J, Guo X, Zhu Z, Wang H, Hu F, Chen J, et al. Quality control system of Watson for oncology: Artificial intelligence for supporting clinical decisions in oncology. J Clin Oncol 2019 May 26;37(15_suppl):6616. [CrossRef]
    62. Fang J, Guo X, Zhu Z, Wang H, Hu F, Chen J, et al. Watson for Oncology applied to teaching and remote consulting model. J Clin Oncol 2019 May 26;37(15_suppl):6545. [CrossRef]
    63. Choi YI, Chung JW, Kim KO, Kwon KA, Kim YJ, Park DK, et al. Concordance rate between clinicians and Watson for Oncology among patients with advanced gastric cancer: Early, real-world experience in Korea. Can J Gastroenterol Hepatol 2019 Feb 03;2019:8072928 [FREE Full text] [CrossRef] [Medline]
    64. Kim EJ, Woo HS, Cho JH, Sym SJ, Baek JH, Lee WS, et al. Early experience with Watson for oncology in Korean patients with colorectal cancer. PLoS One 2019 Mar 25;14(3):e0213640. [CrossRef] [Medline]
    65. Kim D, Kim YY, Lee JH, Chung YS, Choi S, Kang JM, et al. A comparative study of Watson for Oncology and tumor boards in breast cancer treatment. Korean Journal of Clinical Oncology 2019 Jun 30;15(1):3-6 [FREE Full text] [CrossRef]
    66. Kim M, Kim BH, Kim JM, Kim EH, Kim K, Pak K, et al. Concordance in postsurgical radioactive iodine therapy recommendations between Watson for Oncology and clinical practice in patients with differentiated thyroid carcinoma. Cancer 2019 Aug 15;125(16):2803-2809 [FREE Full text] [CrossRef] [Medline]
    67. Suwanvecho S, Shortliffe EH, Suwanrusme H, Issarachai S, Jirakulaporn T, Taechakraichana N, et al. A blinded evaluation of a clinical decision-support system at a regional cancer care center. J Clin Oncol 2019 May 26;37(15_suppl):6553. [CrossRef]
    68. Saiz FJS, Sanders C, Stevens RJ, Nielsen R, Britt MW, Preininger A, et al. Use of machine learning to identify relevant research publications in clinical oncology. J Clin Oncol 2019 May 26;37(15_suppl):6558. [CrossRef]
    69. Waintraub SC, Pecora AL. Cognitive computing and real world data in a breast cancer clinic. 2019 Presented at: Machine Learning & AI for Healthcare: A HIMSS Event; Orlando, Florida; February 11, 2019   URL: https://365.himss.org/sites/himss365/files/365/handouts/552672001/handout-MLAI09.pdf
    70. Hamilton JG, Garzon MG, Westerman JS, Shuk E, Hay JL, Walters C, et al. "A Tool, Not a Crutch": Patient perspectives about IBM Watson for Oncology trained by Memorial Sloan Kettering. J Oncol Pract 2019 Apr;15(4):e277-e288 [FREE Full text] [CrossRef] [Medline]
    71. Saiz FS, Sanders C, Stevens R, Nielsen R, Britt M, Yuravlivker L, et al. Artificial intelligence clinical evidence engine for automatic identification, prioritization, and extraction of relevant clinical oncology research. JCO Clin Cancer Inform 2021 Jan;5:102-111 [FREE Full text] [CrossRef] [Medline]
    72. Zou FW, Tang YF, Liu CY, Ma JA, Hu CH. Concordance study between IBM Watson for Oncology and real clinical practice for cervical cancer patients in China: A retrospective analysis. Front Genet 2020 Mar 24;11:200 [FREE Full text] [CrossRef] [Medline]
    73. Soochit A, Zhang C, Li T, Dankwa-Mullan I, Liu J. Concordance assessment of an artificial intelligence decision support tool for primary and recurrent cervical cancer at an academic cancer center. In: Int J Gynecol Cancer. 2018 Presented at: 17th Biennial Meeting of the International Gynecologic Cancer Society; September 14-16, 2018; Kyoto, Japan.
    74. Zhang C, Soochit A, Dankwa-Mullan I, Li T, Liu J. Evaluation of personalized treatment with artificial intelligence decision-support tools for patients with ovarian cancer. In: Int J Gynecol Cancer. 2018 Presented at: 17th Biennial Meeting of the International Gynecologic Cancer Society; September 14-16, 2018; Kyoto, Japan.
    75. Clinical decision support: More than just 'alerts' tipsheet. Centers for Medicare & Medicaid Services. 2014.   URL: https:/​/www.​cms.gov/​Regulations-and-Guidance/​Legislation/​EHRIncentivePrograms/​Downloads/​ClinicalDecisionSupport_Tipsheet-.​pdf [accessed 2022-03-15]
    76. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018 Nov;68(6):394-424 [FREE Full text] [CrossRef] [Medline]
    77. Bellanger M, Zeinomar N, Tehranifar P, Terry MB. Are global breast cancer incidence and mortality patterns related to country-specific economic development and prevention strategies? J Glob Oncol 2018 Jul;4:1-16 [FREE Full text] [CrossRef] [Medline]
    78. Shah SC, Kayamba V, Peek RMJ, Heimburger D. Cancer control in low- and middle-income countries: Is it time to consider screening? J Glob Oncol 2019 Mar;5:1-8 [FREE Full text] [CrossRef] [Medline]
    79. Institute of Medicine (US) Committee on Cancer Control in Low- and Middle-Income Countries. In: Sloan FA, Gelband H, editors. Cancer Control Opportunities in Low- and Middle-Income Countries. Washington (DC): National Academies Press (US); 2007.
    80. World Health Statistics 2019: Monitoring health for the SDGs, sustainable development goals. World Health Organization. 2019.   URL: https://apps.who.int/iris/bitstream/handle/10665/324835/9789241565707-eng.pdf?sequence=9&isAllowed=y [accessed 2022-03-15]
    81. Mathew A. Global survey of clinical oncology workforce. J Glob Oncol 2018 Sep;4:1-12 [FREE Full text] [CrossRef] [Medline]
    82. Paul Y, Hickok E, Sinha A, Tiwari U. Artificial intelligence in the healthcare industry in India. The Centre for Internet and Society.   URL: https://cis-india.org/internet-governance/ai-and-healthcare-report [accessed 2022-01-07]
    83. Omeje C. Nigeria has fewer than 50 specialists for cancer treatment, says Chidebe. International Centre for Investigative Reporting. 2018 Aug 14.   URL: https://www.icirnigeria.org/nigeria-has-fewer-than-50-specialists-for-cancer-treatment-says-chidibe/ [accessed 2022-03-15]
    84. Obinna C. Shortage of oncologists threatens cancer care in W-Africa. Vanguard. 2014 Jun 30.   URL: https://www.vanguardngr.com/2014/06/shortage-oncologists-threatens-cancer-care-w-africa/ [accessed 2022-03-15]
    85. Mallath MK, Taylor DG, Badwe RA, Rath GK, Shanta V, Pramesh CS, et al. The growing burden of cancer in India: epidemiology and social context. Lancet Oncol 2014 May;15(6):e205-e212. [CrossRef] [Medline]
    86. Ma X, Vervoort D. Leveraging e-learning for medical education in low- and middle-income countries. Cardiol Young 2020 Jun;30(6):903-904. [CrossRef] [Medline]
    87. Barteit S, Guzek D, Jahn A, Bärnighausen T, Jorge MM, Neuhann F. Evaluation of e-learning for medical education in low- and middle-income countries: A systematic review. Comput Educ 2020 Feb;145:103726 [FREE Full text] [CrossRef] [Medline]
    88. Pinninti R, Rajappa S. Artificial intelligence in health-care: How long to go? Cancer Res Stat Treat 2020;3(1):133-134. [CrossRef]
    89. Coiera E. The last mile: Where artificial intelligence meets reality. J Med Internet Res 2019 Nov 08;21(11):e16323 [FREE Full text] [CrossRef] [Medline]

    ‎
    AI: artificial intelligence
    CDSS: clinical decision support system
    HIC: high-income country
    LMIC: low-middle–income country
    MDT: multiple disciplinary team
    NITI: National Institution for Transforming India
    WfO: Watson for Oncology

    Edited by A Mavragani; submitted 22.06.21; peer-reviewed by G Kolostoumpis, N Benda, P Dattathreya, D Ghosh, S Azadnajafabad; comments to author 10.08.21; revised version received 21.01.22; accepted 08.02.22; published 07.04.22

    Copyright

    ©Srinivas Emani, Angela Rui, Hermano Alexandre Lima Rocha, Rubina F Rizvi, Sergio Ferreira Juaçaba, Gretchen Purcell Jackson, David W Bates. Originally published in JMIR Cancer (https://cancer.jmir.org), 07.04.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on https://cancer.jmir.org/, as well as this copyright and license information must be included.