This was a comparative, cross-sectional evaluation of an AI-based LLM (GPT-4; OpenAI) and a standardized internet search engine (Google; Alphabet Inc) using predefined gynecologic oncology clinical scenarios and patient-centered questions (Figure 1). Methods were structured and reported in accordance with the CHART (Chatbot Health Advice Reporting Tool) guideline for chatbot health advice studies [12].

Ethics approval was not required for this study because it did not involve human participants, patient data, or animal subjects. The study evaluated outputs from publicly accessible AI tools and internet search engines and therefore did not constitute human subjects research.

ChatGPT responses were generated using the GPT-4 model via the paid ChatGPT Plus subscription (OpenAI [13]). Google searches were conducted using the Chrome browser.

Six clinical scenarios representing common presentations of gynecologic cancers were developed by a panel of gynecologic oncology providers. For each cancer type (ovarian, endometrial, and cervical), 7 patient-centered questions were created based on commonly asked topics in clinical practice and information available from professional society websites [14], yielding a total of 21 unique questions (Textbox 1). Questions were entered verbatim into both platforms, with each question generating 1 response from GPT-4 and 1 response from Google. A new ChatGPT session was initiated prior to each question to prevent learning across questions or contextual carryover.

Accuracy was assessed using a 6-point Likert scale: 1=“completely incorrect,” 2=“more incorrect than correct,” 3=“approximately equal incorrect and correct,” 4=“more correct than incorrect,” 5=“nearly all correct,” and 6=“correct.” Completeness was assessed using a 3-point Likert scale: 1=“incomplete,” addressing some aspects of the question but with significant parts missing or incomplete; 2=“adequate,” addressing all aspects of the question and providing the minimum information required to be complete; and 3=“comprehensive,” addressing all aspects of the question and providing additional information or context beyond what was expected. Reference quality was assessed using a 3-point Likert scale: 1=“bottom tier,” including any .com sources such as WebMD; 2=“middle tier,” including any .gov or .org sources such as the American Cancer Society and National Institutes of Health; and 3=“top tier,” including peer-reviewed sources or randomized controlled trials such as the National Comprehensive Cancer Network and UpToDate.

Google searches were conducted in private browsing mode with the browser cache cleared. For each query, the top-ranked nonsponsored web page was recorded to allow direct comparison with GPT-4’s single-response output (Multimedia Appendix 1). This approach reflects typical user behavior and aligns with prior studies demonstrating reliance on the first search results [15].

Responses from both platforms were evaluated for accuracy, completeness, and reference quality by 4 gynecologic oncology physicians from 2 institutions (1 academic tertiary referral center and 1 county-based safety-net hospital). Reviewers represented a range of clinical experience, with 2 physicians having <5 years of clinical experience and 2 having >5 years of clinical experience. Reviewers scored responses independently and were blinded to each other’s evaluations. Reviewers were also invited to provide optional free-text comments within the standardized evaluation form. These qualitative comments were summarized descriptively. Comments were reviewed in their entirety and grouped into thematic categories reflecting perceptions of accuracy, completeness, reference quality, clarity, and patient comprehensibility. A total of 76 optional comments were submitted across all reviewers. Representative comments were selected to illustrate common themes across platforms and are presented in Textbox 2. This descriptive approach aligns with conventional qualitative content analysis methods used to contextualize the quantitative findings [16]. Accuracy was defined as concordance with National Comprehensive Cancer Network guidelines and current literature and scored using a 6-point Likert scale. Completeness was scored using a 3-point Likert scale based on the inclusion of major concepts relevant to each question. Reference quality was categorized as low, medium, or high and scored using a 3-point Likert scale.

A total of 21 standardized patient-centered questions were evaluated. Each question generated 1 response from GPT-4 and 1 response from Google, resulting in 21 responses per platform. All responses were independently evaluated by 4 gynecologic oncology physicians, yielding 84 physician evaluations per platform. The number of questions was selected to capture key informational domains, including etiology, prognosis, treatment, and treatment efficacy, while maintaining feasibility for blinded expert review. This approach is consistent with prior studies evaluating the quality of AI-generated and online health information [8,10,11,17-19].

Statistical analyses were conducted using JMP Pro (version 16.0.0; SAS Institute Inc). Descriptive statistics were calculated for all outcomes, and paired comparisons between platforms were performed using Wilcoxon signed-rank tests. Interrater reliability was assessed using intraclass correlation coefficients (ICCs) derived from a 2-way random-effects model with single measures and consistency (ICC[2,1]). Reliability thresholds were interpreted using established criteria [20].

This study found that GPT-4 performed better than Google sources when providing high-quality responses to clinical scenarios involving a new diagnosis of gynecologic cancer. Physician reviewers rated GPT-4 significantly higher than Google in terms of accuracy, completeness, and reference quality. Reviewers were surprised by the quality of the GPT-4 answers and references and saw the potential for GPT-4 to better address patient-specific questions regarding diagnosis, prognosis, and treatment options with both precise and comprehensive information. The most common shortcoming identified in the Google-sourced content was the provision of general information that was not specific to the patient’s cancer diagnosis, stage, or treatment options and therefore was difficult to individually interpret.

Although a growing number of studies have evaluated GPT-4 for cancer-related patient education, relatively few have focused specifically on gynecologic oncology. Initial research on AI in medicine focused on its diagnostic role and potential for clinical decision-making. For example, in otorhinolaryngology, GPT-4 was compared to UpToDate in terms of usefulness and reliability in diagnosing certain common clinical scenarios. UpToDate was found to be more accurate than GPT-4 (P=.009), and the authors concluded that improvements were needed in the usefulness and reliability of GPT-4’s evidence-based knowledge [21]. Similarly, in various clinical cases of thyroid cancer, GPT-4 was found to be reasonably accurate (approximately 76.7%) when providing information about thyroid cancer, but recommendations about evaluation, treatment, and follow-up were deemed general and inadequate [22].

More recently, studies in gynecologic oncology have demonstrated that GPT-4 generally provides guideline-concordant recommendations for gynecologic cancer diagnosis and management, while emphasizing the continued importance of physician oversight for complex clinical decision-making [10,11,23]. Chou et al [17] further reported that GPT-4 generated patient-directed responses regarding ovarian cancer that were comparable to those provided by gynecologic oncology clinicians, supporting its potential role as a patient education tool.

As AI is increasingly explored as a tool for patient-facing communication, several prior studies have evaluated the use of GPT-4 to generate patient information. One study used various LLM platforms to generate patient education materials for both common and rare dermatologic conditions and found that these platforms were able to create accessible, understandable content at an appropriate reading level [18]. Another study focused on the ability of GPT-4 to provide patients with information about screening mammography but documented concerns about the understandability and actionability of the answers, specifically from a health literacy standpoint [19]. Although AI seems to offer an opportunity to enhance patients’ access to health information, there continues to be concern about the possibility of misinformation and the quality of references.

Our findings build upon this growing body of literature. Compared to top-ranked Google search results, GPT-4 consistently outperformed in terms of accuracy, completeness, and reference quality. These findings are consistent with prior studies demonstrating that GPT-4 generally provides accurate, comprehensive, and guideline-concordant information across a variety of cancer-related clinical scenarios [10,11,17,22]. Consistent with the findings of Chou et al [17], we found that GPT-4 generated comprehensive, patient-directed responses that were highly rated by gynecologic oncology physicians. Similarly, our findings align with those of Piazza et al [10] and Finch et al [11], who reported strong agreement between ChatGPT-generated recommendations and established ovarian cancer guidelines. However, although previous studies primarily compared ChatGPT with clinical guidelines or clinician responses, our study directly compared GPT-4 with a conventional Google search strategy, reflecting a common approach used by patients when seeking information after a new cancer diagnosis. Reviewers noted that GPT-4 responses were generally more comprehensive and better aligned with the clinical context, often including relevant treatment information and prognostic details that were lacking in standard search results. Although previous concerns have been raised about the citation quality of previous GPT-4 models, our findings suggest that, in this study, the reference quality was significantly higher than that of Google search results [3,4,8,19,22]. Overall, these results support the potential of GPT-4 to generate personalized, high-quality responses that may enhance patient understanding, support shared decision-making, and improve access to reliable health information.

Strengths of this study include the novel design used to investigate the role of AI in generating informational content related to gynecologic malignancies. By querying GPT-4 and Google with multiple patient scenarios, results were more generalizable to patients with different types and stages of gynecologic cancers. The use of gynecologic oncology physicians, who were blinded to each other’s assessments, enabled the clinical evaluation of response quality by experts who routinely guide patients through diagnosis and treatment. Evaluation of accuracy, completeness, and reference quality offered a comprehensive assessment of the content delivered by each platform. Importantly, we also assessed interrater reliability, which is often omitted in similar studies. Although agreement varied by domain, the inclusion of this analysis strengthens the rigor of our evaluation.

Some limitations of this study include a relatively small number of reviewers, a subjective grading scale, and no direct assessment of the readability of the sources used. Although reviewers were blinded to each other’s ratings, they were aware of the source of each response (GPT-4 or Google), which may have introduced bias. Additionally, all 4 reviewers were from only 2 institutions, which raises the possibility that assessments may have been influenced by local practice patterns or institutional standards, potentially limiting generalizability. Interrater reliability was variable across domains, which likely reflects the inherently subjective nature of evaluating informational content, particularly when responses were largely accurate and complete, resulting in clustering of scores at the higher end of the grading scales. In this context, small differences in individual scoring behavior may disproportionately affect interrater reliability metrics despite overall agreement in comparative performance between platforms. Another important consideration is that the phrasing of questions may influence the quality and clarity of responses generated by LLMs. In this study, the scenarios were designed to reflect the types of questions a well-informed patient might ask after receiving a new cancer diagnosis. In real-world settings, however, patients may not know what questions to ask, may lack the background to phrase them effectively, or may not have the clinical knowledge needed to interpret detailed responses. GPT-4 does not tailor responses based on a user’s health literacy level, and this study did not formally assess the readability or accessibility of the responses. This represents an important limitation, as prior research has shown that the effectiveness of patient education materials depends not only on content accuracy but also on whether they are comprehensible and actionable. Additionally, by evaluating only the top-ranked Google result for each query, the study may not fully capture the breadth of information users might access through additional searches.

Since completing this study, Google has expanded the integration of LLM-generated summaries within its search platform. This underscores the rapidly evolving landscape of online health information and highlights the need for continued evaluation of AI-assisted search tools. Although such integrations may narrow performance gaps between traditional search engines and stand-alone LLMs, our findings remain relevant by illustrating the potential advantages of conversational, context-aware responses over static web page retrieval. Future studies should directly compare AI-generated summaries embedded within search engines with stand-alone LLMs, with a focus on reference transparency, patient personalization, and health literacy.

The results of this study suggest that GPT-4 may be a useful tool for delivering accurate, comprehensive, and well-referenced informational content in gynecologic oncology when compared with top-ranked Google search results within the context of standardized clinical scenarios. However, these findings should be interpreted cautiously given several important methodological limitations, including incomplete blinding of reviewers to response source, variable interrater reliability across evaluation domains, and the restriction of the Google comparator to a single top-ranked result per query. Accordingly, this study should be viewed as a preliminary, hypothesis-generating evaluation rather than a definitive assessment of comparative performance. Although LLMs such as GPT-4 show promise in generating context-aware responses tailored to specific clinical scenarios, further research is needed to clarify their appropriate role in supporting patient education and health care professional–patient communication in gynecologic oncology.