Published on in Vol 7, No 1 (2021): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/25602, first published .
Quality and Readability of Web-Based Information for Patients With Pancreatic Cysts: DISCERN and Readability Test Analysis

Quality and Readability of Web-Based Information for Patients With Pancreatic Cysts: DISCERN and Readability Test Analysis

Quality and Readability of Web-Based Information for Patients With Pancreatic Cysts: DISCERN and Readability Test Analysis

Original Paper

1Division of Hospital Internal Medicine, Mayo Clinic, Jacksonville, FL, United States

2Department of Internal Medicine, Mayo Clinic, Jacksonville, FL, United States

3Department of Radiation Oncology, Mayo Clinic, Rochester, MN, United States

4Division of Gastroenterology and Hepatology, Presbyterian Hospital, Albuquerque, NM, United States

*all authors contributed equally

Corresponding Author:

Sven P Oman, MD

Division of Hospital Internal Medicine

Mayo Clinic

4500 San Pablo Road South

Jacksonville, FL, 32224

United States

Phone: 1 904 956 3032

Fax:1 904 953 2848

Email: oman.sven@mayo.edu


Background: Pancreatic cysts are a complex medical problem with several treatment options. Patients use web-based health information to understand their conditions and to guide treatment choices.

Objective: The goal of this study was to describe the quality and readability of publicly available web-based information on pancreatic cysts and to compare this information across website affiliations.

Methods: A Google search for “pancreatic cysts” was performed and the first 30 websites were evaluated. Website affiliations were classified as academic, media, nonprofit, government, or not disclosed. Information describing cancer risk was recorded. The DISCERN instrument measured the quality of content regarding treatment choices. Four standardized tests were used to measure readability.

Results: Twenty-one websites were included. The majority of the websites (20/21, 95%) described the cancer risk associated with pancreatic cysts. Nearly half of the websites were written by an academic hospital or organization. The average DISCERN score for all websites was 40.4 (range 26-65.5, maximum 80). Websites received low scores due to lack of references, failure to describe the risks of treatment, or lack of details on how treatment choices affect quality of life. The average readability score was 14.74 (range 5.76-23.85, maximum 19+), indicating a college reading level. There were no significant differences across website affiliation groups.

Conclusions: Web-based information for patients with pancreatic cysts is of moderate quality and is written above the reading level of most Americans. Gastroenterological, cancer treatment organizations, and physicians should advocate for improving the available information by providing cancer risk stratification, treatment impact on quality of life, references, and better readability.

JMIR Cancer 2021;7(1):e25602

doi:10.2196/25602

Keywords



The incidence of pancreatic cysts is increasing in developed countries owing to refined abdominal imaging and increasing use of imaging overall [1]. Although mostly benign, some types of pancreatic cysts have malignant potential and may transform into pancreatic cancer. Risk stratification is important to decide which patients should undergo surveillance with serial abdominal imaging (ie, magnetic resonance or endoscopic ultrasound) or which patients should be referred for surgical evaluation. A diagnosis of pancreatic cysts may, therefore, be a cause for concern in patients and may result in health information–seeking behavior. Approximately 70% of Americans use the internet to research health issues [2]. Although reviewing of web-based health information may empower patients to take a more active role in their treatment and improve their relationship, communication, and satisfaction during consultation with their treating physician, it may also introduce cyberchondria, and some websites have information that is of poor quality, is difficult to read, and inconsistent with medical practice guidelines [3-8]. Some of these problems exist because there is no regulation or oversight of web-based health information, which may result in information that is incomplete, unsupported, outdated, biased, or inappropriate for the average reader. Since web-based health information influences patients’ perceived understanding of health issues and how they manage their health, this information must be accurate and readable [9]. Of all the available internet search engines, Google represents 80%-91% of the internet searches and web-based advertising worldwide, with more than 63,000 searches completed every second [10,11]. In this study, we aimed to describe the quality and readability of publicly available web-based information on pancreatic cysts by using the most popular search engine in the world. The secondary objective was to compare the quality and readability among different website affiliations.


A Google website search was performed from November 1, 2019 to December 31, 2019. Two reviewers (SO and HZ) independently rated the first 30 websites retrieved using the search term “pancreatic cysts.” Inclusion criteria were websites intended for the general public with more than 100 words. Websites with associated fees to access content, duplicate websites, publicly modifiable websites, and those with most of the content in audio or video format were excluded. The affiliation of the website was verified using the WHOis.net database (Table 1). For comparison, the same search was performed using Bing and Yahoo search engines on January 8 and 9, 2021, respectively. The quality of the content was measured with the DISCERN instrument (Table 1) [12].

DISCERN was developed by an expert panel of researchers, clinicians, health journalists, and consumers and is funded by The British Library and the NHS Research and Development Program. The instrument is comprised of 16 questions designed for consumers and information providers to assess the quality of written information about available treatment options for any health issue. Each question uses a rating scale from 1 to 5 (1=definite no, 2-4=partially, 5=definite yes) to indicate whether the publication has met certain criteria. The use of DISCERN does not require specialist knowledge or expertise since it is used to judge the reliability of the sources of information and not the scientific quality or evidence. The DISCERN instrument consists of 3 sections. Section 1 addresses the reliability of the publication, specifically, whether the aims were clear, whether these aims were achieved, whether they were relevant, clear, and up-to-date sources of information, whether the information was balanced and unbiased, and whether it referred to any areas of uncertainty. Section 2 evaluates the specific information on the treatment choice(s) presented, specifically, whether they were fully described, whether the benefits and risks and consequences of withholding treatment were mentioned, how the treatment option(s) affect the quality of life, and whether shared decision making was supported. Section 3 rates the overall quality of the source of information (eg, low, moderate, or high). Raters are encouraged to use their independent judgment and their ratings from the proceeding questions, that is, if the majority of questions scored below 2, then the publication would receive low quality; scores in the mid-range would be rated as moderate; and scores mostly rating 4 or above would be rated as high.

Readability was analyzed using 4 standardized tests: Flesch-Kincaid Grade Level, Gunning Fog Index, Simple Measure of Gobbledygook (SMOG) Readability Formula, and Coleman-Liau Index (score range from 5=5th grade level of education to 19 or more=doctorate, Table 1). Scores from each test were averaged. These tests measure the approximate grade level of education needed to understand the written text. Higher scores correspond to a higher grade level of reading. The Flesch-Kincaid Grade Level evaluates word and sentence length; words with more syllables and sentences with more words are rated as more complex and receive a higher score. The Gunning Fog Index works similarly, using the number of words per sentence and the number of complex words (ie, words with 3 or more syllables) to calculate a score. SMOG produces a score based on the number of words with 3 or more syllables in 10 sentence samples. The Coleman-Liau Index assesses the number of characters in a word. The Readable ContentPro Software was used to calculate each of these scores [13].

The websites were grouped into 5 affiliation categories: nonprofit organization, academic, communication/media, government, and affiliation not disclosed. DISCERN scores from the 2 reviewers and the 4 readability tests were summarized as average and median values. Kruskal-Wallis tests compared the differences between the website groups. Interobserver agreement and κ statistic were calculated. Information regarding the cancer risk of pancreatic cysts was also recorded (Table 1).

Table 1. Measuring instruments used for health information websites.
Parameters, question instrumentsDescription
Initial questions

AffiliationNonprofit organization (.org), academic (.edu), communication/media, government (.gov), private/affiliation not disclosed (.com)

Cancer risk explanationMentions risk for pancreatic cancer
Quality of information

DISCERN instrument16 questions × 1-5 points each. Minimum score: 16 points; maximum score: 80 points
Readability

Flesch-Kincaid Grade LevelMinimum score: 5 points, maximum score: 19 points 5 points: 5th grade 6 points: 6th grade 7 points: 7th grade 12 points: 12th grade 13 points: University 1st year 14 points: University 2nd year 15-16 points: University 3rd-4th year 17-18 points: Master’s and professional degree 19+ points: Doctorate

Gunning Fog IndexSame scoring as mentioned for Flesch-Kincaid Grade Level

Simple Measure of Gobbledygook Readability FormulaSame scoring as mentioned for Flesch-Kincaid Grade Level

Coleman-Liau IndexSame scoring as mentioned for Flesch-Kincaid Grade Level

Of the 30 websites examined, 21 met the inclusion criteria. Five of the 21 (23%) were written by nonprofit organizations, 10 (45%) by an academic hospital or organization, 5 (23%) by communication/media websites, and 1 (5%) by an organization without disclosed affiliation. No government websites (.gov) were identified in our sample. Three of the 5 websites published by nonprofit organizations were written by physicians. The Bing and Yahoo searches yielded similar search results as Google. For all 3 search engines, Mayo Clinic’s website on pancreatic cysts was the first appearing search result. Healthline, Memorial Sloan Kettering Cancer Center, The National Pancreas Foundation, and MedicineNet were found in the top 10 websites for all 3 search engines. Hopkins Medicine, Columbia Surgery, and Harvard were common in the top 30 websites for all 3 search engines as well. Pancreatic cancer risk was explained in 20 of the 21 (95%) websites (Table 2).

Table 2. Quality scores for the websites describing pancreatic cysts (DISCERN questionnaire, 1-5 points for each question).
OrderaCancer riskbIs the publication reliable? Questions #1-8 Average (1-5 points)cQuality of information on treatment choices, Questions #9-15 Average (1-5 points)dOverall quality, Q#16 (1-5 points)eTotal score (Q#1-16)fReference
1stYes3.32.13.545.5[14]
2ndYes2.33.23.544.5[15]
3rdYes2.62.0337.5[16]
5thYes2.91.82.538.0[17]
6thYes1.72.8235.0[18]
7thYes2.11.92.532.5[19]
8thYes2.62.02.537.0[20]
9thYes2.42.53.540.0[21]
10thYes2.52.32.538.5[22]
11thYes3.61.11.538.0[23]
12thYes2.63.2346.0[24]
14thYes2.13.23.542.5[25]
15thYes2.02.42.535.0[26]
16thYes4.14.04.565.5[27]
17thYes3.54.44.563.0[28]
18thYes2.22.02.534.0[29]
20thYes2.11.8231.0[30]
22ndYes3.12.02.541.0[31]
24thYes3.32.4245.0[32]
29thYes2.01.92.532.0[33]
30thNo1.61.6226.0[34]

aOrder of appearance. Seven websites did not meet the inclusion criteria: 4th (requires payment); 13th, 23rd, and 27th (not intended for the general public); and 19th, 21st, and 25th (require payment and not intended for the general public).

bDid the website mention the risk of pancreatic cancer associated with pancreatic cysts?

cAverage reliability of the publications=2.3.

dAverage quality of the publications=2.4.

eAverage overall quality=2.8.

fAverage total score=40.4.

The approach and depth to describe cancer risk stratification were diverse. One media website presented a new test for cyst fluid aspirate (monoclonal antibody Das-1) that may predict cancer risk [32]. One nonprofit website provided a complete diagnostic algorithm for patients to discuss with their health care provider, including cancer risk stratification and surgery [27]. One academic website provided a comprehensive 36-page review that included common signs and symptoms of growing cysts, types of pancreatic cysts, treatment options, an agenda for future appointments, and additional references [28]. The average total DISCERN score for all the websites was 40.4 (range 26-65.5, Table 2). The questions that, on average, received the lowest ratings on DISCERN were question #4: “Is it clear what sources of information were used to compile the publication?” (average score 2.14, range 1-5); question #11: “Does it describe the risks of each treatment?” (average score 1.93, range 1-4.5); and question #13: “Does it describe how the treatment choices affect the overall quality of life?” (average score 1.62, range 1-4.5). The median DISCERN score for nonprofit organizations was 33.5 (range 25-74), that for academic hospital or organization was 41 (27-65), that for communication/media was 38 (24-58), and that for an organization without disclosed affiliation was 35 (31-39) (Figure 1). The average total readability score for all the websites was 14.74 (range 5.76-23.9, Table 3).

The median (IQR) readability score for nonprofit organizations was 13.01 (8.64-31.76), that for academic hospital or organization was 15.59 (1.54-20.55), that for communication/media was 15.40 (11.5-19.31), and that for an organization without disclosed affiliation was 14.70 (12.86-16.86) (Figure 1). Scores were similar among website affiliations (DISCERN P=.90 and readability P=.80; Figure 1). Interobserver agreement was adequate (76.2%) but κ was poor (κ= –0.08).

Figure 1. Box plot comparing website quality (DISCERN score, range 16-80 points) and readability (average of 4 instruments, range 5-19 points) by publisher affiliation. Lines represent upper and lower quartiles and dots are outliers.
View this figure
Table 3. Readability scores for websites describing pancreatic cysts.
OrderaAffiliationAverage readabilitybReference
1stAcademic14.26[14]
2ndAcademic16.00[15]
3rdNonprofit23.85[16]
5thMedia14.83[17]
6thPrivate group14.78[18]
7thAcademic12.43[19]
8thMedia15.30[20]
9thAcademic15.49[21]
10thNonprofit12.72[22]
11stMedia13.87[23]
12ndAcademic14.93[24]
14thAcademic17.40[25]
15thAcademic18.04[26]
16thNonprofit11.94[27]
17thAcademic5.76[28]
18thMedia17.30[29]
20thNonprofit10.70[30]
22ndAcademic17.50[31]
24thMedia15.04[32]
29thAcademic14.85[33]
30thNonprofit12.69[34]

aOrder of appearance. Seven websites did not meet the inclusion criteria: 4th (requires payment); 13th, 23rd, and 27th (not intended for the general public); and 19th, 21st, and 25th (require payment and not intended for the general public).

bAverage readability of all websites=14.74.


This study is the first, to our knowledge, to evaluate the quality and readability of web-based information for pancreatic cysts. The findings of this study highlight a substantial gap in the quality and readability of web-based information for patients with pancreatic cysts. Quality was suboptimal due to incomplete descriptions of pancreatic cyst management, lack of clear sources of information, and incomplete descriptions of the risks associated with treatment or how treatment choices affect the overall quality of life. Few websites provided a complete description of the management options (ie, radiological surveillance vs surgical treatment). Similar limitations have been described for other health conditions such as pancreatic cancer and gynecological disorders [4,5,7,35,36]. Despite the overall lack of high-quality web-based health information, a few sources were excellent. The top 5 highest DISCERN-rated websites were The American Gastroenterological Association (DISCERN score 65.5), The University of Michigan Comprehensive Cancer Center, GI Oncology Program (DISCERN score 63), Columbia Surgery (DISCERN score 46), Mayo Clinic (DISCERN score 45.5), and MedicalXpress (DISCERN score 45). We found 2 websites, a publication by the American Gastroenterological Association and a publication by the University of Michigan, which had DISCERN scores greater than 60 [27,28]. Across different health conditions, these high-quality websites represent the minority of cases. Three studies comparing the educational materials for obstetric and pelvic diseases reported that only 5% (3/58) [5], 15% (8/54) [7], and 4% (1/24) [4] of the materials achieved DISCERN scores greater than 60. Encouragingly, in our review, high-quality websites appeared as earlier search hits (our 2 high scorers appeared as the 16th and 17th hits), suggesting that they are more recognizable sources of information for internet users.

We report median readability scores appropriate for a college reading level (range 13-16 points). This is problematic for most readers since only 35% of American citizens complete an undergraduate college education and most Americans read at an elementary school level (Table 4) [37].

Table 4. American population according to reading level (N=328 million).
Reading levelUS population (millions), n (%)
6th grade323 (98.5)
7th grade317 (96.6)
12th grade287 (87.5)
University 1st year190 (57.9)
University 2nd year127 (38.7)
University 3rd to 4th year95 (28.9)
Master’s and Professional degree33 (10.1)
Doctorate7 (2.1)

Studies have similarly found that the readability levels of web-based health information are above the reading levels of most Americans [4,5,7,35,36]. The American Medical Association, therefore, recommends a sixth-grade reading level for all patient-oriented educational materials [36]. However, text difficulty increases from medical jargon and an effort to maintain quality and accuracy of health information. The top 5 most readable websites were The University of Michigan Comprehensive Cancer Center, GI Oncology Program (average readability score 5.76), Pancreatic Cancer UK (average readability score 10.70), The American Gastroenterological Association (average readability score 11.94), Roswell Park Comprehensive Cancer Center (average readability score 12.43), and Virginia Mason (average readability score 12.69). We found only 1 academic website that was written at an adequate reading level [28]. Previous studies found that only 13% (3/24) [4], 5% (3/58) [5], and 0% (0/54) [7] of the websites evaluated were written at an appropriate reading level. Our findings support the paradox that increased quality and accuracy come with a tradeoff of challenging language, longer article length, and higher readability scores [36,38].

The second important finding of this study was similar quality and accuracy scores among different website affiliations. This homogeneity of low quality among different publishers is common in web-based materials describing other health care conditions [4,38,39]. Although no difference was discernible in this study, possibly due to smaller sample size and higher variability, some prior studies have shown important differences. Academic sites have previously been found to be of high accuracy but more difficult to read, while private sites tend to be easier to read with lower accuracy of information [40]. Media websites are both difficult to read and have the lowest accuracy scores [36]. Differences have also been linked to the internet domain used. Research shows that organization domains (ie, “.org”) are an indicator of more accurate information [41] while sites that list references and those without financial interests are also associated with higher quality [42,43].

Our findings demonstrate the importance of balancing high-quality information (higher DISCERN scores) with lower reading levels (readability scores closer to 6) in patient handouts, websites, and reading materials [36,44]. For example, the material produced by the University of Michigan provides many pictorial representations of anatomy, pathology, procedures, and surgery. Houts et al [45] found that visual aids increase comprehension of complex medical information, which increases the understanding of health information, particularly among less literate patients [46]. Another avenue is patient testimonials. Although this study excluded websites that were mostly in the audiovisual format, video narrative presentations of breast cancer treatment wherein patients relate their experiences to other patients increase engagement with the material, with study participants spending more time viewing the information compared to text [47].

The Health of the Net was developed in 1995 to address issues with limited web-based health information quality and lack of supporting evidence. It is overseen by the Health of the Net Foundation in collaboration with the World Health Organization and consists of 8 principles that websites should follow to achieve the Code of Conduct (Health of the Net code) certification. The certification and display of the Health of the Net code seal are intended to help consumers identify reliable websites [48]. Another possible method to improve web-based information is for clinicians writing this information to consider why patients seek web-based information in the first place. For patients with cancer, seeking web-based information typically occurs right after receiving a diagnosis and before starting treatment. Patients may feel they received insufficient information from their providers and may turn to the internet to “fill in the gaps.” Information-seeking behavior is also a coping mechanism by which patients convince themselves that all treatment options have been explored [49]. Treating physicians should be aware of these reasons and redirect patients to credible and appropriate sites for information.

There are some limitations in this study design that are important to consider. First, our results are subject to selection bias and confounding. An examination of the first 30 results from our Google search represents a very small fraction of the websites that contain information regarding pancreatic cysts (eg, a search of “pancreatic cysts” renders over 2 million results on Google). Further, internet search engines use complex algorithms based on geographical locations and previous searches performed. Despite this, most internet users read only the first few pages of the results and 2 separate search engines yielded common results; thus, our analysis is likely a valid representation of an initial search a patient may conduct [50]. Additionally, our search was limited to the English language and our results have limited external validity outside the United States. The small sample size precluded advanced statistical analysis comparing findings among different website affiliations. Other limitations include the DISCERN instrument, which requires an element of subjective analysis, and the fact that our raters were not blinded to website affiliation. Readability tests also do not measure understanding of the material nor do they account for reader motivation, prior knowledge and attitudes, and problems such as poor vision and illness as well as the role of active voice, personalization, and presentation of the information, including font, font size, and illustrations [51,52]. These limitations underline the importance of follow-up with health care providers to clear up any potential misunderstandings and secure additional imaging or treatment when needed. Our study also did not analyze “information completeness” as a variable. Critical information such as genetic implications, costs of surveillance, and treatment was not recorded in our study design.

In summary, our findings demonstrate a gap in the quality and readability of web-based health information regarding pancreatic cyst management. These websites require peer review to balance improved quality with writing closer to a sixth-grade reading level. Gastroenterology and leading cancer organizations should advocate for improving web-based information by calling for a complete description of cancer risk stratification and treatments, visible sources of information and references, and appropriate readability, regardless of the website affiliation. These improvements can help less literate patients understand the information, reduce stress and anxiety after a new diagnosis, and facilitate shared decision making with providers.

Authors' Contributions

SPO participated in the acquisition, analysis, and interpretation of data and in the drafting and editing of the manuscript. HZ participated in data acquisition and editing the manuscript. MRW participated in editing the manuscript. JEC participated in conceptualization, data analysis and interpretation, and editing the manuscript.

Conflicts of Interest

JEC received a travel grant from AbbVie Inc and received minor food and beverage from Boston Scientific Corporation and Cook Medical LLC.

  1. Moris M, Bridges MD, Pooley RA, Raimondo M, Woodward TA, Stauffer JA, et al. Association Between Advances in High-Resolution Cross-Section Imaging Technologies and Increase in Prevalence of Pancreatic Cysts From 2005 to 2014. Clin Gastroenterol Hepatol 2016 Apr;14(4):585-593.e3. [CrossRef] [Medline]
  2. Fox S. The social life of health information. Pew Research Center. 2014 Jan 15.   URL: https://www.pewresearch.org/fact-tank/2014/01/15/the-social-life-of-health-information/ [accessed 2020-04-13]
  3. McMullan RD, Berle D, Arnáez S, Starcevic V. The relationships between health anxiety, online health information seeking, and cyberchondria: Systematic review and meta-analysis. J Affect Disord 2019 Feb 15;245:270-278. [CrossRef] [Medline]
  4. Fowler GE, Baker DM, Lee MJ, Brown SR. A systematic review of online resources to support patient decision-making for full-thickness rectal prolapse surgery. Tech Coloproctol 2017 Nov;21(11):853-862 [FREE Full text] [CrossRef] [Medline]
  5. Ghai V, Pergialiotis V, Jan H, Duffy JMN, Doumouchtsis SK, CHORUS: an International Collaboration Harmonising Outcomes‚ Research‚Standards in Urogynaecology Women’s Health. Obstetric anal sphincter injury: a systematic review of information available on the internet. Int Urogynecol J 2019 May;30(5):713-723. [CrossRef] [Medline]
  6. Harris VC, Links AR, Hong P, Walsh J, Schoo DP, Tunkel DE, et al. Consulting Dr. Google: Quality of Online Resources About Tympanostomy Tube Placement. Laryngoscope 2018 Feb;128(2):496-501 [FREE Full text] [CrossRef] [Medline]
  7. Hirsch M, Aggarwal S, Barker C, Davis CJ, Duffy JMN. Googling endometriosis: a systematic review of information available on the Internet. Am J Obstet Gynecol 2017 May;216(5):451-458.e1. [CrossRef] [Medline]
  8. Tan SS, Goonawardene N. Internet Health Information Seeking and the Patient-Physician Relationship: A Systematic Review. J Med Internet Res 2017 Jan 19;19(1):e9 [FREE Full text] [CrossRef] [Medline]
  9. Baker L, Wagner TH, Singer S, Bundorf MK. Use of the Internet and e-mail for health care information: results from a national survey. JAMA 2003 May 14;289(18):2400-2406. [CrossRef] [Medline]
  10. Company reports 2019. eMarketer.   URL: https://www.emarketer.com/ [accessed 2020-04-23]
  11. Search engine market share worldwide Mar 2019 - Mar 2020. Statscounter - Global.   URL: https://gs.statcounter.com/search-engine-market-share [accessed 2020-04-23]
  12. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999 Feb;53(2):105-111 [FREE Full text] [CrossRef] [Medline]
  13. Readable.   URL: https://readable.com [accessed 2019-12-15]
  14. Pancreatic cysts. Mayo Clinic.   URL: https://www.mayoclinic.org/diseases-conditions/pancreatic-cysts/symptoms-causes/syc-20375993 [accessed 2020-04-23]
  15. Pancreatic cysts. Memorial Sloan Kettering Cancer Center.   URL: https://www.mskcc.org/cancer-care/types/pancreatic-cysts [accessed 2020-04-23]
  16. Pancreatic cysts. The National Pancreas Foundation.   URL: https://pancreasfoundation.org/patient-information/ailments-pancreas/pancreatic-cysts/ [accessed 2020-04-23]
  17. Identifying and treating cysts on the pancreas. Healthline.   URL: https://www.healthline.com/health/cyst-on-pancreas [accessed 2020-04-23]
  18. Pancreatic cysts. Digestive Specialists Inc.   URL: https://www.digestivespecialists.com/conditions/pancreatic-cysts [accessed 2020-04-23]
  19. Do pancreatic cysts become cancerous? Roswell Park Comprehensive Cancer Center.   URL: https://www.roswellpark.org/cancertalk/201711/do-pancreatic-cysts-become-cancerous [accessed 2020-04-23]
  20. Marks JW. Pancreatic cysts. MedicineNet.   URL: https://www.medicinenet.com/pancreatic_cysts/article.htm#what_is_the_pancreas [accessed 2020-04-23]
  21. Pancreatic cysts and pseudocysts. Cleveland Clinic.   URL: https://my.clevelandclinic.org/health/diseases/14627-pancreatic-cysts-and-pseudocysts [accessed 2020-04-23]
  22. What you should know about pancreatic cysts. Lets Win! Pancreatic Cancer. 2018 Jun 08.   URL: https://letswinpc.org/managing-pancreatic-cancer/2018/06/08/what-you-should-know-pancreatic-cysts/ [accessed 2020-04-23]
  23. Nassour I, Choti MA. Types of Pancreatic Cysts. JAMA 2016 Sep 20;316(11):1226 [FREE Full text] [CrossRef] [Medline]
  24. Pancreatic cysts. Columbia Surgery.   URL: https://columbiasurgery.org/conditions-and-treatments/pancreatic-cysts [accessed 2020-04-23]
  25. Pancreatic cysts. University of California San Francisco.   URL: https://surgicaloncology.surgery.ucsf.edu/conditions--procedures/pancreatic-cysts.aspx [accessed 2020-04-23]
  26. Symptoms of pancreatic cysts and masses. Stanford Health Care.   URL: https:/​/stanfordhealthcare.​org/​medical-conditions/​digestion-and-metabolic-health/​pancreatic-cysts/​symptoms.​html [accessed 2020-04-23]
  27. American Gastroenterological Association. AGA Section. Managing Pancreatic Cysts: A Patient Guide. Gastroenterology 2015 Aug;149(2):498-499 [FREE Full text] [CrossRef] [Medline]
  28. A patient's guide to pancreatic cysts. University of Michigan Comprehensive Cancer Center, GI Oncology Program. 2012.   URL: https://www.rogelcancercenter.org/files/pancreatic-cancer/pancreatic-cyst-handbook.pdf [accessed 2020-04-23]
  29. Bennington-Castro J. What is a pancreatic cyst. EveryDay Health. 2018 Feb 01.   URL: https://www.everydayhealth.com/pancreatitis/guide/pancreatic-cyst/ [accessed 2020-04-23]
  30. Pancreatic cysts. Pancreatic Cancer UK.   URL: https:/​/www.​pancreaticcancer.org.uk/​information-and-support/​facts-about-pancreatic-cancer/​types-of-pancreatic-cancer/​pancreatic-cysts/​ [accessed 2020-04-23]
  31. Earlier, more frequent removal of some pancreatic cysts may decrease cancer risk for some patients. Johns Hopkins Medicine. 2019 May 14.   URL: https:/​/www.​hopkinsmedicine.org/​news/​newsroom/​news-releases/​earlier-more-frequent-removal-of-some-pancreatic-cysts-may-decrease-cancer-risk-for-some-patients [accessed 2020-04-23]
  32. Dryden J. Biomarker predicts which pancreatic cysts may become cancerous. MedicalXpress. 2019 Jun 05.   URL: https://medicalxpress.com/news/2019-06-biomarker-pancreatic-cysts-cancerous.html [accessed 2020-04-23]
  33. Pancreatic cysts. MedStar Georgetown University Hospital.   URL: https://www.medstargeorgetown.org/our-services/gastroenterology/conditions/pancreatic-cysts/ [accessed 2020-04-23]
  34. Pancreatic cysts FAQs. Virginia Mason.   URL: https://www.virginiamason.org/pancreatic-cysts-faqs [accessed 2020-04-23]
  35. Lovett J, Gordon C, Patton S, Chen CX. Online information on dysmenorrhoea: An evaluation of readability, credibility, quality and usability. J Clin Nurs 2019 Oct;28(19-20):3590-3598 [FREE Full text] [CrossRef] [Medline]
  36. Storino A, Castillo-Angeles M, Watkins AA, Vargas C, Mancias JD, Bullock A, et al. Assessing the Accuracy and Readability of Online Health Information for Patients With Pancreatic Cancer. JAMA Surg 2016 Sep 01;151(9):831-837. [CrossRef] [Medline]
  37. Bustamante J. College Graduation Statistics. Educationdata.org. 2019 Jun 08.   URL: https://educationdata.org/number-of-college-graduates/ [accessed 2020-04-30]
  38. Bailey MA, Coughlin PA, Sohrabi S, Griffin KJ, Rashid ST, Troxler MA, et al. Quality and readability of online patient information for abdominal aortic aneurysms. J Vasc Surg 2012 Jul;56(1):21-26 [FREE Full text] [CrossRef] [Medline]
  39. Sullivan TB, Anderson JT, Ahn UM, Ahn NU. Can Internet information on vertebroplasty be a reliable means of patient self-education? Clin Orthop Relat Res 2014 May;472(5):1597-1604 [FREE Full text] [CrossRef] [Medline]
  40. Purdy AC, Idriss A, Ahern S, Lin E, Elfenbein DM. Dr Google: The readability and accuracy of patient education websites for Graves' disease treatment. Surgery 2017 Nov;162(5):1148-1154. [CrossRef] [Medline]
  41. Fallis D, Frické M. Indicators of accuracy of consumer health information on the Internet: a study of indicators relating to information for managing fever in children in the home. J Am Med Inform Assoc 2002;9(1):73-79 [FREE Full text] [CrossRef] [Medline]
  42. Martin-Facklam M, Kostrzewa M, Schubert F, Gasse C, Haefeli WE. Quality markers of drug information on the Internet: an evaluation of sites about St. John's wort. Am J Med 2002 Dec 15;113(9):740-745. [CrossRef] [Medline]
  43. Martin-Facklam M, Kostrzewa M, Martin P, Haefeli WE. Quality of drug information on the World Wide Web and strategies to improve pages with poor information quality. An intervention study on pages about sildenafil. Br J Clin Pharmacol 2004 Jan;57(1):80-85 [FREE Full text] [CrossRef] [Medline]
  44. Walsh TM, Volsko TA. Readability assessment of internet-based consumer health information. Respir Care 2008 Oct;53(10):1310-1315. [Medline]
  45. Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient Educ Couns 2006 May;61(2):173-190. [CrossRef] [Medline]
  46. Garcia-Retamero R, Cokely E. Communicating Health Risks With Visual Aids. Curr Dir Psychol Sci 2013 Sep 25;22(5):392-399. [CrossRef]
  47. Shaffer VA, Owens J, Zikmund-Fisher BJ. The effect of patient narratives on information search in a web-based breast cancer decision aid: an eye-tracking study. J Med Internet Res 2013 Dec 17;15(12):e273 [FREE Full text] [CrossRef] [Medline]
  48. HONcode: How it started. Health On the Net. 2017 May 02.   URL: https://www.hon.ch/HONcode/Patients/background.html [accessed 2020-04-13]
  49. Eysenbach G. The impact of the Internet on cancer outcomes. CA Cancer J Clin 2003;53(6):356-371 [FREE Full text] [CrossRef] [Medline]
  50. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 09;324(7337):573-577 [FREE Full text] [CrossRef] [Medline]
  51. Davis TC, Arnold C, Berkel HJ, Nandy I, Jackson RH, Glass J. Knowledge and attitude on screening mammography among low-literate, low-income women. Cancer 1996 Nov 01;78(9):1912-1920. [CrossRef] [Medline]
  52. Mumford ME. A descriptive study of the readability of patient information leaflets designed by nurses. J Adv Nurs 1997 Nov;26(5):985-991. [CrossRef] [Medline]


SMOG: Simple Measure of Gobbledygook


Edited by D Vollmer Dahlke; submitted 08.11.20; peer-reviewed by P Palacios, R Staerkle; comments to author 29.12.20; revised version received 21.01.21; accepted 25.01.21; published 16.03.21

Copyright

©Sven P Oman, Himesh Zaver, Mark Waddle, Juan E Corral. Originally published in JMIR Cancer (http://cancer.jmir.org), 16.03.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on http://cancer.jmir.org/, as well as this copyright and license information must be included.