This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on https://cancer.jmir.org/, as well as this copyright and license information must be included.
As technology continues to improve, health care systems have the opportunity to use a variety of innovative tools for decision-making, including artificial intelligence (AI) applications. However, there has been little research on the feasibility and efficacy of integrating AI systems into real-world clinical practice, especially from the perspectives of clinicians who use such tools. In this paper, we review physicians’ perceptions of and satisfaction with an AI tool, Watson for Oncology, which is used for the treatment of cancer. Watson for Oncology has been implemented in several different settings, including Brazil, China, India, South Korea, and Mexico. By focusing on the implementation of an AI-based clinical decision support system for oncology, we aim to demonstrate how AI can be both beneficial and challenging for cancer management globally and particularly for low-middle–income countries. By doing so, we hope to highlight the need for additional research on user experience and the unique social, cultural, and political barriers to the successful implementation of AI in low-middle–income countries for cancer care.
artificial intelligencecancerlow-middle–income countriesphysiciansperceptionsWatson for Oncologyimplementationlocal contextIntroduction
The last several decades have witnessed the rapid growth of artificial intelligence (AI) applications in health care. AI is considered to comprise areas like machine learning, natural language processing, expert systems, and image and signal processing [1]. One group, who cited a study from Global Market Insights, noted that the use of AI in health care was expected to grow annually from 2016 to 2024, with expenditures increasing from US $760 million in 2016 to over US $10 billion in 2024 [2]. In a 2020 study, Global Market Insights noted that the AI in the health care market exceeded US $4 billion in 2020 and would grow at a compound annual growth rate of 33.7% between 2021 and 2027, with an expenditure of US $34.5 billion in 2027 [3]. This market growth has been accompanied by both national initiatives for AI and the rapid growth of academic literature on the use of AI in health care. For example, in India, an “AI for All” policy was established along with NITI (National Institution for Transforming India) Aayog—a Government of India think tank for formulating a national strategy for AI [4]. A bibliometric analysis of the literature reported in the Journal of Medical Internet Research found a growth rate of 45.15% in publications from 2014 to 2019, with 70.67% of all publications occurring in the same period [5]. This analysis also found the following top five health problems in the publications (in order of frequency): cancer, depression, Alzheimer disease, heart failure, and diabetes. Another review of AI applications in health care found the following areas of focus in the applications: sepsis, breast cancer, diabetic retinopathy, and polyps and adenomas [6]. Additionally, this review noted that the implementation of AI applications in real-world clinical settings is not widespread. Another recent review with a focus on patient safety outcomes also noted the lack of AI applications in real-world settings [7]. These articles, and others in the Journal of Medical Internet Research and elsewhere, have started to capture the use and role of AI in health care [8-11].
In this viewpoint, we contribute to this growing literature by detailing physicians’ experiences with an AI application—Watson for Oncology (WfO)—in the treatment of cancer. Physicians’ experiences with WfO are especially relevant, as the application has been implemented in diverse, real-world social and cultural settings. Our summary of physicians’ experiences with WfO relies on the extensive, published literature on this topic. After we describe physicians’ experiences with WfO, we comment about the opportunities and challenges associated with using AI for cancer care in low-middle–income countries (LMICs).
The WfO Clinical Decision Support System Tool
WfO is a therapeutic oncology clinical decision support system (CDSS) that was trained by experts from the Memorial Sloan Kettering Cancer Center [12]. WfO uses both natural language processing and machine learning to process structured and unstructured data about patients with cancer and generate therapeutic options based on available evidence [13]. WfO provides 3 categories of therapeutic options: “recommended” treatments are those that adhere to the preferred training approach of the Memorial Sloan Kettering Cancer Center, treatments “for consideration” refer to alternative treatments based on evidence, and “not recommended” treatments refer to those that are not appropriate for certain patients [14]. Many early adopters of WfO measured the degree to which WfO therapeutic options were concordant with either clinical practice or the decisions of a multidisciplinary tumor board. WfO concordance rates varied widely across countries for many reasons, including differences in standard treatment guidelines, resource availability, and physician or patient preferences [15]. It is well recognized that concordance studies do not measure system accuracy but instead assess agreement with decisions made in practice, which may or may not reflect evidence-based decisions [16].
In this viewpoint, we focus on physicians’ perceptions of and satisfaction with WfO. We believe that an evaluation of physicians’ perceptions of this AI tool will provide valuable insights for the successful implementation of AI-based CDSSs for cancer treatment, especially in LMICs. Additionally, little is known about how physicians perceive the use of AI tools for cancer treatment. We present physicians’ perceptions of the advantages of, as well as the disadvantages and concerns with, AI in a real-world setting. Our summary relies on published literature on physicians’ perceptions of WfO implementation in a number of countries, including China, India, Mexico, South Korea, and Thailand. Multimedia Appendix 1 provides a comprehensive list of the studies on WfO [13-74].
Advantages
The positive perceptions of WfO relate to the system’s ability to aid clinicians during the therapeutic decision-making process by quickly providing relevant scientific evidence. In China, a satisfaction survey, which was completed by 51 oncologists who used WfO, found that 86.3% of oncologists approved the quality of WfO and 88.2% approved the comprehensibility of WfO’s treatment options, justifications, and external literature [17]. The clinicians rated WfO highly in terms of its ability to provide evidence-based medicine medical education (score: 8.1/10) and literature assistance (score: 7.7/10), assist in medical care quality control (score: 7.3/10), act as a second-opinion consultation resource (score: 7.0/10), perform case reviews with a tumor board (score: 6.9/10), and provide decision support (score: 6.4/10). Overall, the oncologists recommended using WfO as a CDSS to other clinicians (score: 7.3/10). At Shanghai Tenth People’s Hospital, the multiple disciplinary team (MDT) also used WfO and found that their treatment plans became “more standardized, reasonable, and personalized” [18].
WfO’s ability to compare treatment options was tested in Mexico, where it was used for a total of 100 patient cases involving lung, breast, gastric, colon, and rectal cancers diagnosed within the last 5 years [19]. In terms of perceived utility, oncologists found WfO to be “very useful” in comparing treatment options. They reported that WfO might be especially valuable for individuals, such as medical students and residents who lack oncology experience, as well as clinics that do not have enough subspecialists. Several implementations of WfO in China indicate the role of WfO in enhancing the learning experience and efficiency of physicians, particularly junior physicians, and the facilitation of better diagnoses and treatment recommendations [20,21]. This perspective was also substantiated by students from Taipei Medical University Hospital in Taiwan who had limited clinical experience; by using WfO, they performed better on their colon cancer learning assessment than their peers who used traditional search methods and were more clinically experienced [22]. The study also found that students with less clinical experience felt that WfO was “clearer and more understandable” than information found through traditional methods.
WfO’s links to recent and relevant scientific information may provide treatment information that clinicians may not know. In India, an MDT changed their treatment recommendations for 136 of 1000 cases of breast, lung, colon, and rectal cancers because of the data provided by WfO [23]. For 55% of those cases, WfO provided recent evidence of newer treatments. For 30% of the cases, WfO provided new information about genotypic and phenotypic data. For 15% of the cases, WfO provided information on evolving clinical experiences, which influenced the MDT to change their treatment decisions. These results demonstrate the potential of WfO to positively impact cancer outcomes by providing scientific evidence and up-to-date information on clinical guidelines. In a separate study that focused on adjuvant systemic therapy for breast carcinoma, treatment decisions were changed for 4 of 11 patients after the MDT reviewed WfO’s recommendations and EndoPredict (Myriad Genetics Inc) test reports [24]. WfO was able to aid clinicians in providing personalized cancer care while addressing the difficulties of staying informed on evolving cancer guidelines and studies.
Another aspect that must be considered is whether WfO can be useful as a CDSS. At the Instituto Câncer do Ceará in Brazil, a majority of oncologists chose the “agree” or “strongly agree” option for statements that were used to confirm if WfO meets the “CDS Five Rights” criteria [25]. The “CDS Five Rights” contain clinical quality criteria for determining if a CDSS offers benefits that are optimal for a given setting [75]. In the study, 6 of the 7 oncologists at the Instituto Câncer do Ceará believed that WfO provided relevant information that resulted in action being taken and presented the information in a manner that positively aligned with their individual workflows. Further, 5 oncologists agreed that the additional details for each treatment option were easily comprehensible, and 4 oncologists agreed that WfO exceeded their expectations as a CDSS tool for patient management.
Disadvantages and Concerns
Although WfO appears to be useful for displaying information in a succinct and timely manner, there are concerns regarding the system’s usability and integration into clinician workflows. First, at sites without integrated patient record systems, some users found manual data entry to be a burdensome process [13,26]. At Manipal Hospital in India, it was observed that acclimation to the system reduced the time needed for each patient case [27]. The mean time needed to collect and enter data for nonmetastatic diseases was 20 minutes. This was reduced to 12 minutes after an increased acquaintance of 10 cases with WfO. In comparison, the time needed to collect and enter data for metastatic diseases was 5 to 7 minutes longer than that for localized diseases. On average, WfO took a median of 40 seconds to capture, analyze, and provide treatment recommendations. For physicians with a high patient load, the time needed to enter information into the system may be an issue. Users also want WfO to provide an explanation of its process for scoring and ranking treatment options [26]. In doing so, users would feel more comfortable with trusting the information and recommendations provided by WfO.
A second important concern that has been identified in studies is localizing WfO’s treatment recommendations to the country of implementation. In the previously mentioned satisfaction study conducted in China, 66.7% of physicians recommended that WfO should integrate data on locally available treatments to improve the system [17]. For example, WfO did not take into consideration whether the immunotherapy drugs it recommended had been approved by the China Food and Drug Administration. Physicians also chose chemotherapy instead of WfO’s recommended medication because the medication was too expensive for patients. Similar challenges were found for WfO users in Mexico and Thailand [19,28]. In Mexico, clinicians deviated from WfO’s recommendations due to the high costs associated with them and the fact that they did not adhere to Mexican cancer treatment guidelines [19]. In Thailand, oncologists preferred basing their treatment recommendations on other countries’ guidelines instead of US guidelines [28].
Implications for LMICs
In 2012, 65% of all cancer deaths worldwide occurred in LMICs, and the projection for 2030 is that this will increase to 75% [76]. LMICs may also be experiencing an even higher burden from cancer than that experienced by high-income countries (HICs) for several reasons. LMICs have restrained funding and often lack optimal cancer registries and surveillance data; thus, they are unable to implement evidence-based cancer control programs [76]. Treatment modalities are also more limited in LMICs than in HICs; radiotherapy and chemotherapy are available in 43% to 51% of LMICs but are available in 94% of HICs [77]. However, there is a high demand for such therapies, as 5 million new people annually are estimated to need radiation therapy in LMICs [78]. LMICs also lack specialized medical personnel, such as oncologists and oncology nurses, who are needed to address those affected by cancer in LMICs [79]. According to a World Health Organization report, LMICs have the lowest density of health care workers in comparison to HICs, where the density of health care workers is significantly higher [80]. A lack of health care workers for serving the population makes offering high-quality, personalized care a difficult task.
Oncologists also often require the expertise of their colleagues and additional literary resources to determine a course of treatment for unique cancer cases. Gaining access to high-quality medical information is key for creating an appropriate treatment plan, but oncologists may need additional help with sorting information that is both relevant to their patients and viable in terms of what resources are available. AI-based platforms such as WfO may be able to address the growing challenges of providing cancer treatment plans in LMICs. AI can address issues of access to knowledge bases in a comprehensive and easy-to-access manner. The ability of AI tools to quickly provide evidence-based cancer treatment options would be especially helpful in low-resource settings where the lack of time, expertise, and other needed resources can become a barrier to providing care. Using AI in this manner may also promote international partnerships on cancer therapy research and standardize guidelines for certain cancer types. The studies reviewed in this viewpoint demonstrate the potential of AI to reduce the cognitive burdens of less experienced physicians who would benefit from additional medical education resources.
The experiences with WfO in different settings also reveal a positive perception of AI with regard to its ability to reassure clinicians and confirm their interpretations of data and the potential of such a tool to do so in an LMIC. The ability of AI to act as a second opinion resource and standardize treatments may prove especially useful for cancer care in LMICs where the likelihood of receiving comprehensive care and achieving positive outcomes is lower than that in HICs. A lack of available specialized medical personnel in LMICs, especially in rural regions, is one of the factors contributing to poor cancer outcomes in LMICs [81]. The ability of AI tools, such as WfO, to provide subspecialty treatment information makes such tools a much-needed resource that existing physicians can use to meet population demands, especially in rural areas, as envisioned for the use of AI in an LMIC like India [4,82]. Approximately 60 oncologists serve over 300 million people in West Africa, and only 2000 oncologists are available for 10 million patients in India [83-85]. WfO’s open-access information, which can be used to supplement self-paced learning, would be an ideal resource for cancer physicians in LMICs where medical education resources are lacking [86,87].
The use of an AI tool such as WfO in LMICs also poses certain challenges. The technological challenges that are unique to LMICs and should be mentioned include access to the internet, technology training, and whether local technology teams would be able to address technical issues [87]. Providing a decision support tool, such as WfO, that is user-friendly and aligns with daily workflows is essential for implementation in LMICs, where physicians’ experiences with technology can vary [87]. Additionally, there is concern about whether AI tools would exacerbate the divide in health care access and use, especially with respect to socioeconomic status. There is a fear that AI would recommend treatment options that patients cannot afford or that only high-income and educated patients who are aware of AI tools may benefit from the use of AI in LMICs [4,19,28] Another important concern—one that applies to a tool such as WfO—is that the data and training of AI tools may not incorporate patient characteristics into treatment recommendations. For example, in China, local patient characteristics such as gene mutations and the weaker physiques of Chinese patients, which can influence treatment recommendations, were not accounted for in WfO recommendations [20]. Similarly, the need to consider the presence of multiple ethnic groups in countries like India during the implementation of AI tools developed by Western countries will be an important factor to address in LMICs [88].
Conclusion
It is undeniable that oncology physicians in LMICs need as much additional support as possible. The implementation of AI tools, such as WfO, in different settings has revealed that access to a second opinion CDSS resource, concise scientific evidence, and international clinical guidelines can help physicians feel more confident in their final treatment decisions. To improve the clinical utility of AI tools such as WfO, it is necessary that the experiences and satisfaction of physicians who use such tools are explored more in-depth, especially those of physicians in LMICs. These perspectives are especially key to tailoring AI systems for use in real-world clinical settings [6,7]. Such perspectives are of course embedded in the local social, cultural, and political LMIC contexts within which AI is implemented and the ways in which local contexts can shape the use of AI. We are gaining experience with respect to the implementation of AI tools, such as WfO, in real-world settings for the treatment of cancer. However, we still need to address some of the challenges in the “last mile” stage of implementation, specifically those related to local contexts [89].
Studies on Watson for Oncology (WfO).
AbbreviationsAI
artificial intelligence
CDSS
clinical decision support system
HIC
high-income country
LMIC
low-middle–income country
MDT
multiple disciplinary team
NITI
National Institution for Transforming India
WfO
Watson for Oncology
This work was supported by IBM Watson Health (Cambridge, Massachusetts), which is not responsible for the content or recommendations made.
SE, AR, and DWB received salary support from a grant funded by IBM Watson Health. DWB has received research support from and consults for EarlySense, which makes patient safety monitoring systems. He receives cash compensation from the Center for Digital Innovation (Negev), which is a not-for-profit incubator for health information technology start-ups. He receives equity from ValeraHealth, which makes software to help patients with chronic diseases; Clew, which makes software to support clinical decision-making in intensive care; and MDClone, which takes clinical data and produces deidentified versions of them. He consults for and receives equity from AESOP, which makes software to reduce medication error rates, and FeelBetter. He has received research support from MedAware. RFR is employed by IBM Watson Health. GPJ was employed by IBM Watson Health at the time of manuscript submission and GPJ's compensation included salary and equity. All other authors declare no conflicts of interests.
WahlBCossy-GantnerAGermannSSchwalbeNRArtificial intelligence (AI) and global health: how can AI contribute to health in resource-poor settings?2018082934e00079810.1136/bmjgh-2018-00079830233828bmjgh-2018-000798PMC6135465Healthcare artificial intelligence outlook: Benefits, projected growth and challenges2022-01-04https://thessigroup.com/healthcare-artificial-intelligence-outlook-benefits-projected-growth-challenges/Artificial intelligence in healthcare market size by application (medical imaging and diagnosis, drug discovery, therapy planning, hospital workflow, wearables, virtual assistants), COVID-19 impact analysis, regional outlook, application potential, price trends, competitive market share and forecast, 2021 - 202720202022-01-04https://www.gminsights.com/industry-analysis/healthcare-artificial-intelligence-marketPradhanKJohnPSandhuNUse of artificial intelligence in healthcare delivery in India20219511010.21037/jhmhp-20-126GuoYHaoZZhaoSGongJYangFArtificial intelligence in health care: Bibliometric analysis20200729227e1822810.2196/1822832723713v22i7e18228PMC7424481YinJNgiamKYTeoHHRole of artificial intelligence applications in real-life clinical practice: Systematic review20210422234e2575910.2196/2575933885365v23i4e25759PMC8103304ChoudhuryAAsanORole of artificial intelligence in patient safety outcomes: Systematic literature review2020072487e1859910.2196/1859932706688v8i7e18599PMC7414411IqbalUCeliLALiYCJHow can artificial intelligence make medicine more preemptive?20200811228e1721110.2196/1721132780024v22i8e17211PMC7448175LeeJIs artificial intelligence better than human clinicians in predicting patient outcomes?20200826228e1991810.2196/1991832845249v22i8e19918PMC7481865VuongQHHoMTVuongTTLaVPHoMTNghiemKCPTranBXGiangHHGiangTVLatkinCNguyenHKTHoCSHHoRCMArtificial intelligence vs. natural stupidity: Evaluating AI readiness for the Vietnamese Medical Information System201902018216810.3390/jcm802016830717268jcm8020168PMC6406313HosnyAAertsHJWLArtificial intelligence for global health20191122366646895595610.1126/science.aay518931753987366/6468/955PMC7790340LuxtonDDShould Watson be consulted for a second opinion?20190201212E131E13710.1001/amajethics.2019.13130794122amajethics.2019.131KeikesLMedlockSvan de BergDJZhangSGuicheritORPuntCJAvan OijenMGHThe first steps in the evaluation of a "black-box" decision support tool: a protocol and feasibility study for the evaluation of Watson for Oncology201807273Suppl 341142330873490PMC6412599LeeWSAhnSMChungJWKimKOKwonKAKimYSymSShinDParkILeeUBaekJHAssessing concordance with Watson for Oncology, a cognitive computing decision support system for colon cancer treatment in Korea20181221810.1200/CCI.17.0010930652564LiangJLiTZhangSSChenCVanHoutenCPreiningerADankwa-MullanIJacksonGPReasons for discordance in treatment approaches between oncology practice and clinical decision support in China201905203715_suppl655510.1200/jco.2019.37.15_suppl.6555SuwanvechoSSuwanrusmeHJirakulapornTIssarachaiSTaechakraichanaNLungchukietPDechaWBoonpakdeeWThanakarnNWongrattananonPPreiningerAMSolomonMWangSHekmatRDankwa-MullanIShortliffeEPatelVLArriagaYJacksonGPKiatikajornthadaNComparison of an oncology clinical decision-support system's recommendations with actual treatment decisions2021031828483283810.1093/jamia/ocaa334335173896124728PMC7973455LiTChenCZhangSSDankwa-MullanIChenAPreiningerAJacksonGPLiangJDeployment and integration of a cognitive technology in China: Experiences and lessons learned201905203715_suppl653810.1200/jco.2019.37.15_suppl.6538FangJZhuZWangHHuFLiuZGuoXChenJLiCShenYXuQThe establishment of a new medical model for tumor treatment combined with Watson for Oncology, MDT and patient involvement201806013615_supple1850410.1200/jco.2018.36.15_suppl.e18504Sarre-LazcanoCAlonsoAAMelendezFDHArrietaONordenADUrmanAPerroniMLandis-McgrathAMedina-FrancoHCognitive computing in oncology: A qualitative assessment of IBM Watson for Oncology in Mexico201705303515_supple1816610.1200/jco.2017.35.15_suppl.e18166LiuCLiuXWuFXieMFengYHuCUsing artificial intelligence (Watson for Oncology) for treatment recommendations amongst Chinese patients with lung cancer: Feasibility study20180925209e1108710.2196/1108730257820v20i9e11087PMC6231834TianYLiuXWangZCaoSLiuZJiQLiZSunYZhouXWangDZhouYConcordance between Watson for Oncology and a multidisciplinary clinical decision-making team for gastric cancer and the prognostic implications: Retrospective study20200220222e1412210.2196/1412232130123v22i2e14122PMC7059081ChenCYHungHCChiuHYWeiPLKuoPLChiouJFPreiningerADankwa-MullanIKefayatiSSolomonMJacksonGRheeKYenYEnhancing evidence-based medicine skills in oncology training with cognitive technology201905263715_suppl1053210.1200/jco.2019.37.15_suppl.10532SomashekharSPSepúlvedaMJShortliffeEHCRKRauthanAPatilPYethadkaRA prospective blinded study of 1000 cases analyzing the role of artificial intelligence: Watson for oncology and change in decision making of a Multidisciplinary Tumor Board (MDT) from a tertiary care cancer center201905263715_suppl653310.1200/jco.2019.37.15_suppl.6533SomashekharSPYethadkaRCRKRajgopalAKRauthanAPatilPTriple blinded prospective study assessing the impact of genomics: EndoPredict and artificial intelligence Watson for Oncology (WFO) on MDT’s decision of adjuvant systemic therapy for hormone receptor positive early breast carcinoma201905263715_supple1801310.1200/jco.2019.37.15_suppl.e18013RochaHALDankwa-MullanIJuacabaSFPreiningerAFelixWThompsonJVBrightTJacksonGPMeneleuPAn evaluation of artificial intelligence-based clinical decision supports use in Brazil201905263715_supple1808110.1200/jco.2019.37.15_suppl.e18081ZaudererMGGucalpAEpsteinASSeidmanADCarolineAGranovskySFuJKeesingJLewisSCoHPetriJMegerianMEggebraatenTBachPKrisMGPiloting IBM Watson Oncology within Memorial Sloan Kettering’s regional network201405203215_supple1765310.1200/jco.2014.32.15_suppl.e17653SomashekharSPKumarcRRauthanAArunKRRamyaYEAbstract S6-07: Double blinded validation study to assess performance of IBM artificial intelligence platform, Watson for oncology in comparison with Manipal multidisciplinary tumour board – First study of 638 breast cancer cases2017774_SupplementS60710.1158/1538-7445.sabcs16-s6-07SuwanvechoSSuwanrusmeHSangtianMNordenADUrmanAHicksADankwa-MullanIRheeKKiatikajornthadaNConcordance assessment of a cognitive computing system in Thailand201705303515_suppl658910.1200/JCO.2017.35.15_suppl.6589ZhangXCZhouNZhangCTLvHYLiTJZhuJJJiangMHouHLLiuDLiAQLiuGZhangGQConcordance study between IBM Watson for Oncology (WFO) and clinical practice for breast and lung cancer patients in China2017110128Supplement 10x17010.1093/annonc/mdx678.001BachPZaudererMGGucalpAEpsteinASNortonLSeidmanADCarolineAGrigorenkoABartashnikAWagnerIKeesingJKohnMHsiaoFMegerianMStevensRJMalinJWhitneyJKrisMGBeyond Jeopardy!: Harnessing IBM's Watson to improve oncology decision making201305203115_suppl650810.1200/jco.2013.31.15_suppl.6508EpsteinASZaudererMGGucalpASeidmanADCarolineAFuJKeesingJHsiaoFMegerianMEggebraatenTDeLimaRSetnesMBachPKrisMGNext steps for IBM Watson Oncology: Scalability to additional malignancies201405203215_suppl661810.1200/jco.2014.32.15_suppl.6618FuJGucalpAZaudererMGEpsteinASKrisMGKeesingJCarolineAMegerianMEggebraatenTDeLimaRSetnesMWildtPSeidmanADSteps in developing Watson for Oncology, a decision support system to assist physicians choosing first-line metastatic breast cancer (MBC) therapies: Improved performance with machine learning201505203315_suppl56610.1200/jco.2015.33.15_suppl.566KrisMGGucalpAEpsteinASSeidmanADFuJKeesingJCarolineAMegerianMEggebraatenTDeLimaRSetnesMZaudererMGAssessing the performance of Watson for oncology, a decision support system, using actual contemporary clinical cases201505203315_suppl802310.1200/jco.2015.33.15_suppl.8023SeidmanADPilewskieMLRobsonMEKelvinJFZaudererMGEpsteinASKrisMGFuJKeesingJCarolineAMegerianMEggebraatenTDeLimaRSetnesMBarkerKGucalpAIntegration of multi-modality treatment planning for early stage breast cancer (BC) into Watson for Oncology, a Decision Support System: Seeing the forest and the trees201505203315_supple1204210.1200/jco.2015.33.15_suppl.e12042HerathDHWilson-IngDRamosEMorstynGAssessing the natural language processing capabilities of IBM Watson for oncology using real Australian lung cancer cases201605203415_supple1822910.1200/JCO.2016.34.15_suppl.e18229SomashekharSSPKumarRKumarAPatilPRauthanA551PD Validation study to assess performance of IBM cognitive computing system Watson for oncology with Manipal multidisciplinary tumour board for 1000 consecutive cases: An Indian experience20161227ix17910.1016/S0923-7534(21)00709-2ZhouNLvHZhangCLiTZhuJJiangMHouHLiuDLiALiuGLiuKZhangGZhangXP1.01-069 Clinical experience with IBM Watson for Oncology (WFO) cognitive system for lung cancer treatment in China2017111211S192110.1016/j.jtho.2017.09.723YueLYangLClinical experience with IBM Watson for Oncology (WFO) for multiple types of cancer patients in China20171128x16210.1093/annonc/mdx676.024SomashekharSSepúlvedaMNordenADRauthanAArunKPatilPEthadkaRYKumarRCEarly experience with IBM Watson for Oncology (WFO) cognitive computing system for lung and colorectal cancer treatment201705303515_suppl852710.1200/JCO.2017.35.15_suppl.8527BaekJHAhnSMUrmanAKimYSAhnHKWonPSLeeWSSymSJParkHKChunYSChoEKLeeWKShinDKimYBDankwa-MullanINordenADRheeKLeeUUse of a cognitive computing system for treatment of colon and gastric cancer in South Korea201705303515_supple1820410.1200/JCO.2017.35.15_suppl.e18204LimSLeeKBUse of a cognitive computing system for treatment of cervical cancer201709285e6710.3802/jgo.2017.28.e672865722828.e67PMC5540726ZhouNZhangCTLvHYHaoCXLiTJZhuJJZhuHJiangMLiuKWHouHLLiuDLiAQZhangGQTianZBZhangZCConcordance study between IBM Watson for Oncology and clinical practice for patients with cancer in China20190624681281910.1634/theoncologist.2018-025530181315theoncologist.2018-0255PMC6656482ChenPJSunTTLiTIDankwa-MullanIUrmanAWangCKZhangYYaoYFHeGIRheeKWuACan AI technology augment tumor board treatment decisions for stage II colon cancer care?201806013615_supple1858210.1200/JCO.2018.36.15_suppl.e18582WuAChenPJLiTIDankwa-MullanISunTTRheeKReal world survival outcomes in patients with high risk stage II colon cancer at a Beijing Cancer Hospital201805203615_supple15670e1567010.1200/JCO.2018.36.15_suppl.e15670JiangZXuFSepúlvedaMLiJWangHLiuZYinYYanMSongYGuoJRoebuckMCGengCTangJConcordance, decision impact and guidelines adherence using artificial intelligence in high-risk breast cancer201806013615_supple1856610.1200/JCO.2018.36.15_suppl.e18566YuwenDZhangWWuJZhangJShenYShiJTangCDankwa-MullanIRheeKChenJConcordance evaluation of an artificial intelligence technology with a multidisciplinary tumor board in gastric cancer201806013615_supple1856910.1200/JCO.2018.36.15_suppl.e18569SomashekharSPSepúlvedaMJPuglielliSNordenADShortliffeEHRohit KumarCRauthanAArun KumarNPatilPRheeKRamyaYWatson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board2018020129241842310.1093/annonc/mdx78129324970S0923-7534(19)35072-0KimYYOhSJChunYSLeeWKParkHKGene expression assay and Watson for Oncology for optimization of treatment in ER-positive, HER2-negative breast cancer20180706137e020010010.1371/journal.pone.020010029979736PONE-D-18-07239PMC6034851SuwanrusmeHIssarachaiSUmsawasdiTSuwanvechoSDechaWDankwa-MullanIWangCKUrmanAKiatikajornthadaNConcordance assessment of a clinical decision support software in patients with solid tumors201806013615_supple1858410.1200/JCO.2018.36.15_suppl.e18584SimonGDiNardoCDTakahashiKCasconeTPowersCStevensRAllenJAntonoffMBGomezDKeanePSuarez SaizFNguyenQRoartyEPierceSZhangJBarnhillEHLakhaniKShawKSmithBSwisherSHighRFutrealPAHeymachJChinLApplying artificial intelligence to address the knowledge gaps in cancer care20190624677278210.1634/theoncologist.2018-025730446581theoncologist.2018-0257PMC6656515SaizFJSUrmanASandersCBrittMWNielsenRStevensRJIBM Watson Evidence Service (WES): A system for retrieval, summation and insight generation of relevant clinical evidence for personalized oncology201806013615_supple1858810.1200/JCO.2018.36.15_suppl.e18588GrahamDMAMcNamaraDMWaintraubSEGoldbergSLNordenADHerveyJPecoraALLandstromCSnowdonJLFrancisPMJungbluthNWangCKLattsLAre treatment recommendations provided by cognitive computing supported by real world data (Watson for Oncology with Cota RWE) concordant with expert opinions?20181029viii57110.1093/annonc/mdy297.031WaintraubSEMcNamaraDMGrahamDMAGoldbergSLNordenADHerveyJPecoraALLandstromCSnowdonJLSnowdonPMJungbluthNWangCKLattsLCan the cognitive computing system Watson for Oncology with Cota RWE help oncologists deliver subspecialist level care?201836th Annual Chemotherapy Foundation Symposium (CFS)November 7-9, 2018New York, New YorkRochaHALDankwa-MullanIJuacabaSFWillisVArriagaYEJacksonGPMeneleuPShared-decision making in prostate cancer with clinical decision-support201905263715_supple1657610.1200/JCO.2019.37.15_suppl.e16576WangZYuZZhangXArtificial intelligence-based clinical decision-support system improves cancer treatment and patient satisfaction201905263715_supple1830310.1200/JCO.2019.37.15_suppl.e18303YuZWangZRenXLouDLiXLiuHZhangXPractical exploration and research of Watson for oncology clinical decision support system in real-world and localized practice201905263715_supple1830410.1200/JCO.2019.37.15_suppl.e18304ChenZBChenSLLiangRMPengZWShenJXZhuWJLiBPengSKuangMCan artificial intelligence support the clinical decision making for Barcelona clinic liver cancer stage 0/a hepatocellular carcinoma in China?201905263715_supple1563410.1200/JCO.2019.37.15_suppl.e15634ChenSChenZXiaoHPengZPengSKuangMIDDF2019-ABS-0095 Can artificial intelligence support the clinical decision making for hepatocellular carcinoma?201968A13410.1136/gutjnl-2019-IDDFAbstracts.263LiangJTangCWangXGuoZNiJXuWDankwa-MullanIWillisVChenAJacksonGPRheeKLiTImpact of decision-support system and guideline treatment concordance on response rate in advanced lung cancer201905263715_supple2000610.1200/JCO.2019.37.15_suppl.e20006LiangJDankwa-MullanIRenYChenAWillisVJacksonGLevitanNRheeKLiTIEmploying an oncology decision-support system to quantify treatment variation201905263715_supple1806710.1200/JCO.2019.37.15_suppl.e18067FangJGuoXZhuZWangHHuFChenJYuanMZhaoYWuJGaoSXuQQuality control system of Watson for oncology: Artificial intelligence for supporting clinical decisions in oncology201905263715_suppl661610.1200/JCO.2019.37.15_suppl.6616FangJGuoXZhuZWangHHuFChenJGaoSGuoLZhaoYYuanMXuQWatson for Oncology applied to teaching and remote consulting model201905263715_suppl654510.1200/JCO.2019.37.15_suppl.6545ChoiYIChungJWKimKOKwonKAKimYJParkDKAhnSMParkSHSymSJShinDBKimYSSungKHBaekJHLeeUConcordance rate between clinicians and Watson for Oncology among patients with advanced gastric cancer: Early, real-world experience in Korea201902032019807292810.1155/2019/807292830854352PMC6377977KimEJWooHSChoJHSymSJBaekJHLeeWSKwonKAKimKOChungJWParkDKKimYJEarly experience with Watson for oncology in Korean patients with colorectal cancer20190325143e021364010.1371/journal.pone.021364030908530PONE-D-17-43118PMC6433269KimDKimYYLeeJHChungYSChoiSKangJMParkHKChunYSA comparative study of Watson for Oncology and tumor boards in breast cancer treatment201906301513610.14216/kjco.19002KimMKimBHKimJMKimEHKimKPakKJeonYKKimSSParkHKangTLeeBJKimIJConcordance in postsurgical radioactive iodine therapy recommendations between Watson for Oncology and clinical practice in patients with differentiated thyroid carcinoma20190815125162803280910.1002/cncr.3216631216369SuwanvechoSShortliffeEHSuwanrusmeHIssarachaiSJirakulapornTTaechakraichanaNLungchukietPThanakarnNDechaWBoonpakdeeWWongrattananonPPreiningerAKefayatiSDankwa-MullanIEsquivelJPatelVLJacksonGKiatikajornthadaNA blinded evaluation of a clinical decision-support system at a regional cancer care center201905263715_suppl655310.1200/JCO.2019.37.15_suppl.6553SaizFJSSandersCStevensRJNielsenRBrittMWPreiningerAJacksonGUse of machine learning to identify relevant research publications in clinical oncology201905263715_suppl655810.1200/JCO.2019.37.15_suppl.6558WaintraubSCPecoraALCognitive computing and real world data in a breast cancer clinic2019Machine Learning & AI for Healthcare: A HIMSS EventOrlando, FloridaFebruary 11, 2019HamiltonJGGarzonMGWestermanJSShukEHayJLWaltersCElkinEBertelsenCChoJDalyBGucalpASeidmanADZaudererMGEpsteinASKrisMG"A Tool, Not a Crutch": Patient perspectives about IBM Watson for Oncology trained by Memorial Sloan Kettering201904154e277e28810.1200/JOP.18.0041730689492PMC6494242SaizFSSandersCStevensRNielsenRBrittMYuravlivkerLPreiningerAMJacksonGPArtificial intelligence clinical evidence engine for automatic identification, prioritization, and extraction of relevant clinical oncology research202101510211110.1200/CCI.20.0008733439724PMC8140792ZouFWTangYFLiuCYMaJAHuCHConcordance study between IBM Watson for Oncology and real clinical practice for cervical cancer patients in China: A retrospective analysis202003241120010.3389/fgene.2020.0020032265980PMC7105853SoochitAZhangCLiTDankwa-MullanILiuJConcordance assessment of an artificial intelligence decision support tool for primary and recurrent cervical cancer at an academic cancer center201817th Biennial Meeting of the International Gynecologic Cancer SocietySeptember 14-16, 2018Kyoto, JapanZhangCSoochitADankwa-MullanILiTLiuJEvaluation of personalized treatment with artificial intelligence decision-support tools for patients with ovarian cancer201817th Biennial Meeting of the International Gynecologic Cancer SocietySeptember 14-16, 2018Kyoto, JapanClinical decision support: More than just 'alerts' tipsheet20142022-03-15https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/ClinicalDecisionSupport_Tipsheet-.pdfBrayFFerlayJSoerjomataramISiegelRLTorreLAJemalAGlobal cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries20181168639442410.3322/caac.2149230207593BellangerMZeinomarNTehranifarPTerryMBAre global breast cancer incidence and mortality patterns related to country-specific economic development and prevention strategies?201807411610.1200/JGO.17.0020730085889PMC6223528ShahSCKayambaVPeekRM JrHeimburgerDCancer control in low- and middle-income countries: Is it time to consider screening?20190351810.1200/JGO.18.0020030908147PMC6452918Institute of Medicine (US) Committee on Cancer Control in Low- and Middle-Income CountriesSloanFAGelbandH2007Washington (DC)National Academies Press (US)World Health Statistics 2019: Monitoring health for the SDGs, sustainable development goals20192022-03-15https://apps.who.int/iris/bitstream/handle/10665/324835/9789241565707-eng.pdf?sequence=9&isAllowed=yMathewAGlobal survey of clinical oncology workforce201809411210.1200/JGO.17.0018830241241PMC6223442PaulYHickokESinhaATiwariUArtificial intelligence in the healthcare industry in India2022-01-07https://cis-india.org/internet-governance/ai-and-healthcare-reportOmejeCNigeria has fewer than 50 specialists for cancer treatment, says Chidebe201808142022-03-15https://www.icirnigeria.org/nigeria-has-fewer-than-50-specialists-for-cancer-treatment-says-chidibe/ObinnaCShortage of oncologists threatens cancer care in W-Africa201406302022-03-15https://www.vanguardngr.com/2014/06/shortage-oncologists-threatens-cancer-care-w-africa/MallathMKTaylorDGBadweRARathGKShantaVPrameshCSDigumartiRSebastianPBorthakurBBKalwarAKapoorSKumarSGillJLKuriakoseMAMalhotraHSharmaSCShuklaSViswanathLChackoRTPautuJLReddyKSSharmaKSPurushothamADSullivanRThe growing burden of cancer in India: epidemiology and social context201405156e205e21210.1016/S1470-2045(14)70115-924731885S1470-2045(14)70115-9MaXVervoortDLeveraging e-learning for medical education in low- and middle-income countries20200630690390410.1017/S104795112000110932389137S1047951120001109BarteitSGuzekDJahnABärnighausenTJorgeMMNeuhannFEvaluation of e-learning for medical education in low- and middle-income countries: A systematic review20200214510372610.1016/j.compedu.2019.10372632565611S0360-1315(19)30279-9PMC7291921PinnintiRRajappaSArtificial intelligence in health-care: How long to go?20203113313410.4103/CRST.CRST_7_20CoieraEThe last mile: Where artificial intelligence meets reality201911082111e1632310.2196/1632331702559v21i11e16323PMC7351266