This paper is in the following e-collection/theme issue:

Breast Cancer (200) Consumer & Patient Education and Shared-Decision Making (636) Information Quality in Digital Media (317) Patient Education for Cancer (138) Emotional, Social, Psychological Support for Cancer (188) Quality/Credibility of eHealth Information and Trust Issues (486) Information Seeking, Information Needs (421) eHealth Literacy / Digital Literacy (352)

Published on in Vol 7, No 4 (2021): Oct-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/25783, first published .
Assessing the Quality of Online Health Information About Breast Cancer from Chinese Language Websites: Quality Assessment Survey

Assessing the Quality of Online Health Information About Breast Cancer from Chinese Language Websites: Quality Assessment Survey

Assessing the Quality of Online Health Information About Breast Cancer from Chinese Language Websites: Quality Assessment Survey

Authors of this article:

Weiwei Sun1 Author Orcid Image ;   Aijing Luo2, 3 Author Orcid Image ;   Zhiwei Bian1 Author Orcid Image ;   Bin Zhao1 Author Orcid Image ;   Peng Liu1 Author Orcid Image ;   Kai Wang1 Author Orcid Image ;   Yuwen Liu1 Author Orcid Image ;   Wenzhao Xie3, 4 Author Orcid Image ;   Fuzhi Wang1 Author Orcid Image
Article Authors Cited by (5) Tweetations (2) Metrics

    Original Paper

    1School of Health Management, Bengbu Medical College, Bengbu, China

    2The Second Xiangya Hospital, Central South University, Changsha, China

    3Key Laboratory of Medical Information Research, College of Hunan Province, Central South University, Changsha, China

    4The Third Xiangya Hospital, Central South University, Cahngsha, China

    *these authors contributed equally

    Corresponding Author:

    Fuzhi Wang, PhD

    School of Health Management

    Bengbu Medical College

    2600#, Donghai Rd, Anhui

    Bengbu, 233000

    China

    Phone: 86 18855202156

    Email: wfz.bbmc@foxmail.com


    Background: In China, the internet has become one of the most important ways to obtain information about breast cancer. However, quantitative evaluations of the quality of Chinese health websites and the breast cancer treatment information they publish are lacking.

    Objective: This study aimed to evaluate the quality of Chinese breast cancer websites and the value, suitability, and accuracy of the breast cancer treatment information they publish.

    Methods: Chinese breast cancer health websites were searched and manually screened according to their Alexa and Baidu search engine rankings. For each website included in the survey, which was conducted on April 8, 2019, the three most recently published papers on the website that met the inclusion criteria were included for evaluation. Three raters assessed all materials using the LIDA, DISCERN, and Suitability Assessment of Materials (SAM) tools and the Michigan Checklist. Data analysis was completed with the Statistical Package for Social Sciences (SPSS) version 20.0 and Microsoft Excel 2010.

    Results: This survey included 20 Chinese breast cancer websites and 60 papers on breast cancer treatment. The LIDA tool was used to evaluate the quality of the 20 websites. The LIDA’s scores of the websites (mean=54.85, SD 3.498; total possible score=81) were low. In terms of the layout, color scheme, search facility, browsing facility, integration of nontextual media, submission of comments, declaration of objectives, content production method, and robust method, more than half of the websites scored 0 (never) or 1 (sometimes). For the online breast cancer treatment papers, the scores were generally low. Regarding suitability, 32 (53.33%) papers were evaluated as presenting unsuitable material. Regarding accuracy, the problems were that the papers were largely not original (44/60, 73%) and lacked references (46/60, 77%).

    Conclusions: The quality of Chinese breast cancer websites is poor. The color schemes, text settings, user comment submission functions, and language designs should be improved. The quality of Chinese online breast cancer treatment information is poor; the information has little value to users, and pictorial information is scarcely used. The online breast cancer treatment information is accurate but lacks originality and references. Website developers, governments, and medical professionals should play a full role in the design of health websites, the regulation of online health information, and the use of online health information.

    JMIR Cancer 2021;7(4):e25783

    doi:10.2196/25783

    Keywords

    online health information; breast cancer; Chinese language websites; quantitative evaluations 


    Breast cancer is a malignant tumor of the breast epithelium. Since 1980, the incidence of breast cancer has been increasing worldwide. The age-standardized mortality rate based on the world standard population was 182.6 per million in 2018, and breast cancer is the leading cause of death due to cancer in women (15.0%) [1]. In the past decade, the incidence of breast cancer in China has also been rapidly growing, with a prevalence of 1%-2%, higher than that in other countries [2]. According to the latest national cancer statistics released by the National Cancer Centre of China in January 2019, breast cancer is the fourth-leading cause of death due to cancer among women in China and is one of the malignant tumors threatening the health of Chinese women [3].

    For cancer-related information, the majority of the public is more likely to search the internet first. Because cancer is a major disease that is associated with strong privacy and sensitivity, breast cancer patients tend to first seek relevant health information from the internet to deepen their understanding of the disease and assist them in making decisions on health behavior [4]. The process of searching for health information about breast cancer is influenced by subjective and objective factors. The subjective factors include the users' information literacy level. Some research results have shown that the educational level of internet users [5], their attitude toward online health information [6], and their ability to acquire [7] and evaluate [8] online health information have positive effects on their rational utilization of this type of information. The objective factors mainly include the quality of online health information.

    In Europe and North America, there is much research on online cancer health information [9,10]. The quality assessment of online health information about cancer has attracted extensive attention from scholars worldwide. Garfinkle [11] evaluated the readability, quality, and accuracy of online health information for patients with low anterior resection syndrome following surgery for rectal cancer and found that online health information is lacking and too complex for patients to understand. Another study [12] assessed the availability and quality of information about female oncofertility on the websites of (inter)national oncology, fertility, and oncofertility organizations and suggested that the availability and quality of online health information be improved and that high-quality resources be recommended by physicians. The quality of online health information is a complex concept involving more than 20 dimensions, as perceived by consumers. The most widely reported criteria used by consumers were trustworthiness, expertise, and objectivity, and the most widely reported indicators were website owner/sponsor, consensus among multiple sources, the characteristics of writing and language, advertisements, content authorship, and interface design [13].

    In China, according to the 43rd China Internet Network Development Statistics report released by the China Internet Network Information Centre, as of December 2018, China had 829 million internet users [14]. Online health information services have become an important way for people to obtain health information [15]. The quality of health information service platforms is affected by the quality of the websites and the information they publish [16]. However, due to the imbalance between the rapid development of informatization and the regulation of online information, the quality of online health information service websites and their information has been uneven [17]. According to the results of a recent study [18], Baidu is the most popular online information source for breast cancer; however, more than half (55.1%) of those surveyed were dissatisfied with the online information. To date, China's domestic research on online health information has focused on discussing online health information evaluation indexes [19] and evaluation tools [20] and on theoretical research on online health information service platforms [21]. However, there is still a lack of in-depth research on the quantitative assessment of health information websites and the information they publish.

    Currently, there are many tools for online information quality evaluation. Some evaluation tools are highly targeted. For example, the Suitability Assessment of Materials (SAM) tool can be used to evaluate the applicability of information [22]. The Simplified Measure of Gobbledygook (SMOG) tool can be used to evaluate the readability of information [23], while the Health on the Net Foundation Code of Conduct (HONcode) has proposed a special code of ethics for the release of online health information for health websites [24]. Other tools focus on evaluating the quality of online health information from multiple dimensions; for example, the Michigan Checklist includes an evaluation of online health information quality and website design [25]. In addition to assessing the information quality of online treatment schemes, the DISCERN tool uses several items to evaluate the reliability of websites and has been used to evaluate the readability, suitability, and quality of online health information [22] In this study, we carefully examined relevant online health information quality evaluation tools and assessed the effectiveness of these tools and the independence between subdimensions, finding that some subdimensions of these evaluation tools are repeated. To evaluate the quality of online breast cancer health information as comprehensively as possible and to avoid duplication of the subdimensions of the evaluation tools, we chose to conduct our study based on the two dimensions of website quality (usability and reliability) and three dimensions of information quality (value, suitability, and accuracy).

    The purpose of this study was to evaluate the quality of Chinese health breast cancer websites and to evaluate the quality of online breast cancer treatment information in terms of value, suitability, and accuracy. This study aims to provide support for breast cancer patients and caregivers to make effective use of online health information services and to make reasonable health decisions by analyzing the quality of and the problems with online health information in China.


    Sample

    The evaluation of the quality of online health information about breast cancer was divided into two parts: First, the quality of Chinese websites that publish breast cancer health information was evaluated; second, the quality of Chinese online papers on breast cancer treatment was evaluated. The quality of websites mainly depends on their functionality. The quality of papers mainly depends on the health-related content.

    The initial screening of Chinese breast cancer websites was completed in two steps. The first step consisted of selecting the top 100 websites as research samples based on the results of Chinese medical and health websites provided by Webmaster’s House and the Alexa ranking. The second step consisted of using the Baidu search engine to select the results of the first 20 pages from the list of search results, with “breast cancer” or “breast tumor” used as the search keyword. In all, 38 breast cancer–related health websites were manually screened. ChinaZTM is the most well-known basic web service provider in China, providing users with Alexa ranking queries, website traffic queries, and other services on Chinese websites. Alexa has the largest number of Uniform Resource Locators (URLs) and detailed ranking information [26]. Alexa China provides free official data queries of Chinese website rankings, which can reflect the traffic and popularity of a website to some extent [27]. Baidu is the most visited Chinese search engine in the world [28,29]. Based on the search results of these two platforms and after eliminating 26 duplicate websites, 112 websites were included in the research sample pool of this study. The inclusion criteria for the health websites were as follows: (1) the websites were Chinese websites, (2) the information released by the websites was obviously relevant to breast cancer health, (3) the websites were not intended to sell merchandise, and (4) the websites were not official hospital websites. We excluded hospital websites because official websites provide basic information about the hospital, such as an introduction to the hospital and departments. Moreover, there is little detailed health information about breast cancer on hospital websites. Based on the inclusion criteria, 20 breast cancer health websites were finally included in the survey.

    The inclusion criteria for Chinese online papers on breast cancer were as follows: (1) the papers were written in Chinese, (2) the papers were related to breast cancer treatment information, (3) the papers were not for advertising, and (4) the papers included text and pictures. According to the date of publication of the papers, three recently published papers on each website that met the inclusion criteria were selected as samples to evaluate the quality of online treatment information about breast cancer.

    The samples were was collected on April 8, 2019. The sample collection process is shown in Multimedia Appendix 1.

    Tools

    Quality Assessment Tool for Websites

    LIDA was used to evaluate the usability and reliability of health websites on breast cancer. This tool was developed by Minervation, a British consulting company in the health care field, in 2007 and was designed for professionals to evaluate all aspects of health websites, focusing on the degree of recognition of health websites by professionals [30]. The evaluation of usability included four dimensions: clarity, consistency, functionality, and engageability. The evaluation of reliability included three dimensions: currency, conflicts of interest, and content production. In this study, a total of 27 items were used to evaluate the websites. Each question was scored on a scale of 0 to 3, where 0 indicated never, 1 indicated sometimes, 2 indicated mostly, and 3 indicated always.

    Quality Assessment Tool for Papers

    The quality of the papers was assessed based on three key parameters: value, suitability, and accuracy.

    Value Assessment Tool

    The value of online papers on breast cancer was assessed using a 7-item scale selected from the DISCERN tool. DISCERN is a tool for judging the quality of written consumer health information about treatment choices; it was developed by the British Library in 1999 [31]. DISCERN consists of a total of 16 questions, and it was the first tool in the world for evaluating the information quality of health websites. It includes three dimensions: the evaluation of websites, the evaluation of the value of therapeutic papers, and overall evaluation. DISCERN is a validated tool that has adequate internal consistency (α=.78) and satisfactory interrater reliability [32]. To assess the value of papers on breast cancer treatment, we selected only the second dimension of the DISCERN scale. It includes 7 questions, each rated on a 5-point Likert scale ranging from 1=no (ie, the criterion is not fulfilled by the publication) to 5=yes (ie, the criterion is fulfilled by the publication).

    Suitability Assessment Tool

    The SAM tool was used to evaluate the suitability of online health information about breast cancer. SAM, developed and designed by Doak, is an objective assessment tool for evaluating the availability and reliability of health materials [33]. SAM includes 6 dimensions: content (4 items), literacy demand (5 items), graphics (5 items), layout and typography (3 items), learning stimulation and motivation (3 items), and cultural appropriateness (2 items). Each item is scored on a scale of 0 to 2 points, where 0 indicates not suitable, 1 indicates adequate, and 2 indicates superior. In this study, considering that there was no front cover with online health information, we removed one item (cover graphic) that did not apply. The higher the final score of a paper is, the better its suitability.

    Accuracy Assessment Tool

    Six items in the Michigan Checklist were selected to evaluate the accuracy of the papers. The Michigan Checklist was created by the University of Michigan in 1999, and it focuses on evaluating health websites and their content. The scale included two aspects: content and usability. Because this study mainly evaluated the accuracy of papers published on breast cancer websites, the content of the scale (items 18-23) was selected to evaluate the accuracy of the papers:

    • #18. Are sources cited or credited?
    • #19. Is a bibliography or resource list available?
    • #20. Can you identify errors or significant omissions in information presented?
    • #21. Are opinions or misleading/biased information presented as fact?
    • #22. Does information presented as factual appear to be accurate to the best of your knowledge?
    • #23. Is there an identifiable conflict of interest?

    Rating Process

    The evaluation was performed by three assessors. Assessor 1 (author SWW) holds a master’s degree in medical informatics and has 7 years of experience in medical information analysis and research. Assessor 2 (author WFZ) holds a master's degree in computer science and a doctorate in social medicine and has 8 years of experience in computer software development. Assessor 3 (author BZW) holds a doctor of medicine degree and a clinical physician qualification certificate. Two websites (six papers) were used for experimental evaluation. Before the test, the three researchers (assessors) carefully read the scales and usage instructions of the four assessment tools (the LIDA, DISCERN, and SAM tools and the Michigan Checklist) to understand the purpose and significance of the evaluation items. The evaluation was divided into two steps. First, assessors 1 and 2 used LIDA to evaluate the quality of the websites. Then, assessors 1 and 3 used the other three scales to evaluate the quality of the selected papers. To ensure the consistency of the evaluation results, the subdimension was adopted as the evaluation unit; that is, the N-th+1 dimension was evaluated after the evaluation of the N-th dimension of all samples was completed. The evaluation process adopted a parallel mode. Two assessors independently evaluated a given sample simultaneously. In the case of diverging evaluation results, the final results were determined through real-time negotiation.

    Ethical Approval

    Ethical approval for the study was obtained from the Medical Ethics Committee of the BengBu Medical College (BBMC; reference no. 2017054).

    Statistical Analysis

    Analysis was conducted using the Statistical Package for Social Sciences (SPSS) version 20.0 (SPSS Inc., Chicago, IL, USA) and Microsoft Excel 2010 (Microsoft Inc., Washington DC, USA). All values are expressed as the mean ± SD.


    Characteristics of the Breast Cancer Websites

    The characteristics of the breast cancer websites are shown in Table 1. All 20 websites had internet content provider (ICP) registration numbers. Of the 20 websites, 17 (85%) were corporate websites and 3 (15%) were personal websites. There were 3 (15%) websites with Baidu weights of 7-9, 10 (50%) websites with Baidu weights of 4-6, and 7 (35%) websites with Baidu weights of less than 3. Regarding the number of years since website registration, there were 4 (20%) websites that had been registered for more than 15 years, 8 (40%) websites that had been registered for 5-10 years, and only 1 (5%) website that had been registered for less than 5 years. In terms of regional distribution, 17 (85%) websites were registered in eastern China, 1 (5%) was registered in central China, and 2 (10%) were registered in western China.

    Table 1. Characteristics of the breast cancer websites (N=20).
    CharacteristicGroupn (%)
    NatureEnterprise17 (85)
    Personal3 (15)
    Global rankingLess than 10,0004 (20)
    10,000-30,0005 (25)
    30,000-60,0002 (10)
    More than 60,0006 (30)
    —a3 (15)
    Traffic ranking10,0004 (20)
    10,000-30,0005 (25)
    30,000-60,0003 (15)
    More 60,0005 (25)
    —3 (15)
    Week of Alexa ranking50002 (10)
    5000-10,0002 (10)
    10,000-30,0006 (30)
    More than 30,0004 (20)
    —6 (30)
    ICPb certifiedYes20 (100)
    No0 (0)
    Baidu weightc7-93 (15)
    4-610 (50)
    Less than 37 (35)
    Years since registrationMore than 154 (20)
    10-158 (40)
    5-107 (35)
    Less than 51 (5)
    RegionEastern part17 (85)
    Central part1 (5)
    Western part2 (10)

    a—: not available.

    bICP: internet content provider. ICPs are telecom operators providing comprehensive internet information services and value-added services to a vast number of users. The required certificate is the ICP certificate. Profit-making websites must handle ICP certificates; otherwise, they are illegal businesses.

    cBaidu weight: Baidu weights are evaluated data that are used to estimate search engine traffic by the webmaster tool through an analysis of the ranking of a website’s keywords. The evaluated data are divided into 0-9 for a total of 10 grades. Baidu weights are related to the number of keywords and traffic. The more keywords there are, the higher the weight of accumulation. The higher the keyword flow is, the higher the cumulative weight will be.

    Quality of Breast Cancer Websites

    We used LIDA to evaluate the quality (usability and reliability) of the 20 breast cancer websites. The evaluation results are shown in Table 2. The evaluation results showed that the overall score of the quality evaluation of the breast cancer websites was 54.85±3.498 (81 points). With regard to the layout, color scheme, search facility, browsing facility, integration of nontextual media, submission of comments, declaration of objectives, content production method, and robust method, the scores of the websites were low. The results showed that the quality of the websites needs to be improved.

    Table 2. Descriptive statistics of LIDA items.
    ItemScore=0Score=1Score=2Score=3Mean (SD)
    2.1 Clarity, n (%)

    		2.1.1 User scope0 (0)1 (5)19 (95)0 (0)1.950 (0.224)

    		2.1.2 Knowledge level0 (0)3 (15)16 (80)1 (5)1.900 (0.447)

    		2.1.3 Layout0 (0)13 (65)6 (30)1 (5)1.400 (0.598)

    		2.1.4 Navigation0 (0)3 (15)10 (50)7 (35)2.200 (0.696)

    		2.1.5 Location in the website0 (0)0 (0)9 (45)11 (55)2.550 (0.510)

    		2.1.6 Color scheme0 (0)10 (50)6 (30)4 (20)1.700 (0.801)
    2.2 Consistency, n (%)

    		2.2.1 Page layout0 (0)0 (0)13 (65)7 (35)2.350 (0.489)

    		2.2.2 Navigation links0 (0)0 (0)20 (100)0 (0)2.000 (0.000)

    		2.2.3 Website structure0 (0)0 (0)20 (100)0 (0)2.000 (0.000)
    2.3 Functionality, n (%)

    		2.3.1 Search facility0 (0)7 (35)13 (65)0 (0)1.650 (0.489)

    		2.3.2 Browsing facility0 (0)5 (25)11 (55)4 (20)1.950 (0.686)

    		2.3.3 Cognitive overhead0 (0)0 (0)20 (100)0 (0)2.000 (0.000)

    		2.3.4 Navigation tools0 (0)0 (0)20 (100)0 (0)2.000 (0.000)

    		2.3.5 Third-party plug-ins0 (0)0 (0)20 (100)0 (0)3.000 (0.000)
    2.4 Engageability, n (%)

    		2.4.1 Effective judgement0 (0)0 (0)20 (100)0 (0)2.000 (0.000)

    		2.4.2 Interactivity0 (0)3 (15)17 (85)0 (0)1.850 (0.366)

    		2.4.3 Personalized experience0 (0)2 (10)14 (70)4 (20)2.100 (0.553)

    		2.4.4 Integration of nontextual media0 (0)12 (60)8 (40)0 (0)1.400 (0.503)
    3.1 Currency, n (%)

    		3.1.1 Recent events0 (0)3 (15)17 (85)0 (0)1.850 (0.366)

    		3.1.2 Submit comments0 (0)14 (70)6 (30)0 (0)1.600 (0.940)

    		3.1.3 Updated1 (5)1 (5)1 (5)17 (85)2.700 (0.801)
    3.2 Conflicts of interest, n (%)

    		3.2.1 Who runs the website?0 (0)0 (0)0 (0)20 (100)3.000 (0.000)

    		3.2.2 Pay for the website0 (0)2 (10)18 (90)0 (0)1.900 (0.308)

    		3.2.3 Declaration of objectives0 (0)5 (25)4 (20)11 (55)2.300 (0.865)
    3.3 Content production, n (%)

    		3.3.1 Content production method0 (0)1 (5)19 (95)0 (0)1.950 (0.224)

    		 3.3.2 Robust method0 (0)9 (45)11 (55)0 (0)1.550 (0.510)

    		3.3.3 Original sources0 (0)0 (0)20 (100)0 (0)2.000 (0.000)

    Quality of Breast Cancer Papers

    Value

    The evaluation of the value of the papers was mainly based on seven questions. The questions were used to evaluate the treatment (or treatments) described in the publication. The overall rating of the value of the papers on breast cancer was low. The highest scores were for items on whether the description indicated that there may be more than one possible treatment choice (item 14) and on whether the paper supports shared decision making (item 15). The items with low scores included information about the risks of each treatment (item 11), what would happen without treatment (item 12), and how the treatment choices affect the overall quality of life (item 13). The specific results are shown in Table 3.

    Table 3. Descriptive statistics of the DISCERN items.
    ItemScore=1, n (%)Score=2, n (%)Score=3, n (%)Score=4, n (%)Score=5, n (%)Mean (SD)
    #9. Does it describe how each treatment works?16 (27)20 (34)14 (23)8 (13)2 (3)2.333 (1.115)
    #10. Does it describe the benefits of each treatment?7 (11)16 (27)28 (47)6 (10)3 (5)2.700 (0.979)
    #11. Does it describe the risks of each treatment?24 (40)18 (30)12(20)4 (7)2 (3)2.033 (1.089)
    #12. Does it describe what would happen if no treatment were used?44 (73)11 (18)3 (5)1 (2)1 (2)1.400 (0.807)
    #13. Does it describe how the treatment choices affect overall quality of life?25 (42)16 (27)12(20)4 (6)3 (5)2.067 (1.163)
    #14. Is it clear that there may be more than one possible treatment choice?2 (3)15 (25)9 (15)9 (15)25 (42)3.667 (1.336)
    #15. Does it provide support for shared decision making?3 (5)14 (23)14 (23)13 (22)16 (27)3.417 (1.253)
    Suitability

    The suitability evaluation results of the papers are shown in Table 4. In this study, after evaluation, the highest SAM score was 42 points (100%). Of 60 papers, only 1 (1.67%) met the criteria for superior suitability, as established by SAM, 27 (45%) papers met the criteria for adequate suitability, and 32 (53.33%) papers were evaluated as not suitable material. The graphics (0.85/8 points), literacy demand (4.18/10 points), and layout and typography (2.53/6 points) scores were low, and the graphics score was the lowest. Three other aspects were also evaluated: content (3.82/6 points), cultural appropriateness (1.85/4 points), and learning stimulation (2.63/6 points).

    Table 4. Descriptive statistics of SAMa items.
    FactorScore=0bScore=1cScore=2dMean (SD)
    Content, n (%)

    		1) It is important that readers understand the purpose of the materials. If they do not, they may miss the main point.4 (6)28 (47)28 (47)1.400 (0.616)

    		2) Adult learners usually want to solve their problems rather than learn facts. The content of most interest and use is likely to be behavior information to help solve their problems.1 (2)57 (95)2 (3)1.017 (0.225)

    		3) Scope should be limited to the purpose/objectives of the material and to what can reasonably be learned in the time typically allocated to reading the information.11 (18)39 (65)10 (17)0.983 (0.596)

    		4) A summary offers readers a chance to see the key points in other words or examples. They are important; readers often miss the key points when they first read them.36 (60)23 (38)1 (2)0.417 (0.530)
    Literacy demand, n (%)

    		1) The text reading level is an important factor in whether your target group understands your document.26 (43)32 (54)2 (3)0.600 (0.558)

    		2) A conversational style and active voice lead to easy-to understand text. Simple sentences are used extensively.3 (22)46 (76)1 (2)0.800 (0.443)

    		3) It is best to use common, explicit words and avoid words that express general terms.0 (0)59 (98)1 (2)1.017 (0.129)

    		4) We learn new facts/behaviors more quickly when told the context first.16 (27)43 (71)1 (2)0.750 (0.474)

    		5) Headers or topic captions tell briefly what is coming up next. These “road signs” make the text look less formidable, and prepare the reader’s thought process to expect the next topic.7 (12)45 (75)8 (13)1.017 (0.504)
    Graphics, n (%)

    		1) Simple line drawings can promote realism without including distracting details. Visuals are accepted and remembered better when they portray what is familiar and easily recognized.44 (73)14 (24)2 (3)0.300 (0.530)

    		2) Non-essential details, such as room background, elaborate borders, and unneeded color, can distract the reader, whose eyes may be “captured” by these details. The illustrations should visually represent the key points.44 (73)12 (20)4 (7)0.333 (0.601)

    		3) Many readers do not understand the purpose of lists, charts, and graphs. Explanations and directions are essential.55 (92)2 (3)3 (5)0.133 (0.468)

    		4) Captions can quickly tell the reader what the graphic is all about and where to focus within the graphic. A graphic without a caption is usually an inferior instruction and a missed learning opportunity.55 (92)5 (8)0 (0)0.083 (0.279)
    Layout and typography, n (%)

    		1) Layout has a substantial influence on the suitability of materials.23 (38)34 (57)3 (5)0.667 (0.572)

    		2) Type size and fonts can make text easy or difficult for readers at all skill levels.0 (0)60 (100)0 (0)1.000 (0.000)

    		3) Few people can remember more than seven independent items. For adults with low literacy skills, the limit may be three- to five-item lists. Longer lists need to be broken into smaller chunks.17 (28)34 (57)9 (15)0.867 (0.650)
    Learning stimulation and motivation, n (%)

    		1) When a reader responds to an instruction, chemical changes take place in the brain that enhance retention in long-term memory. Readers should be asked to solve problems, to make choices, to demonstrate, etc.36 (60)19 (32)5 (8)0.483 (0.651)

    		2) People often learn more readily by observation, by doing something for themselves rather than by reading or being told, and when specific, familiar instances are used rather than the abstract or general.8 (13)36 (60)16 (27)1.133 (0.632)

    		3) People are more motivated to learn when they believe the tasks/behaviors are doable by them.2 (3)55 (92)3 (5)1.017 (0.291)
    Cultural appropriateness, n (%)

    		1) A valid measure of cultural appropriateness of material is how well its logic, language, and experience (inherent in the instruction) match the logic, language, and experience of the intended audience.12 (20)41 (68)7 (12)0.917 (0.561)

    		2) To be accepted, an instruction must present cultural images and examples in realistic and positive ways.6 (10)52 (87)2 (3)0.933 (0.362)

    aSAM: Suitability Assessment of Materials.

    bScore 0: not suitable.

    cScore 1: adequate.

    dScore 2: superior.

    Accuracy

    For originality and for listing references, the scores were generally low. Of the 60 papers, 44 (73%) were unoriginal and 46 (77%) did not have a bibliography or resource list available. The top three items were “whether the paper contains errors or omissions”, “whether the paper is misleading or biased,” and “whether the paper is accurate.” More than 97% (n=58) of the papers were correct in their content descriptions, with no errors, omissions, or misleading hints. The evaluation results are shown in Table 5.

    Table 5. Descriptive statistics of the Michigan Checklist items.
    ItemScore=–2, n (%)Score=+2, n (%)Score=–3, n (%)Score=+3, n (%)
    #18. Are sources cited or credited?—a—44 (73)16 (27)
    #19. Is a bibliography or resource list available?46 (77)14 (23)——
    #20. Can you identify errors or significant omissions in information presented?——2 (3)58 (97)
    #21. Are opinions or misleading/biased information presented as fact?

    		1 (2)59 (98)
    #22. Does information presented as factual appear to be accurate to the best of your knowledge?1 (2)59 (98)——
    #23. Is there an identifiable conflict of interest?14 (23)46 (77)——

    aN/A: not applicable.


    Principal Findings

    Generally, the quality of Chinese breast cancer websites is poor. The quality of the online papers on breast cancer is also poor. The quality of online information service platforms, which are an important medium for the dissemination of health information in the new media environment [34], affects the public's health decisions [35].

    Similar to the results of previous research, this study found that the format (updated and who runs the website) of Chinese breast cancer websites is good [36], but the color scheme, text setting, function of user comment submission, and language design should be improved. For example, using colors to mark the title can make the paper clearer, but this setting is not effective for people with color cognitive impairment. Another similar issue is font size; rather than fancy colors, older users want to be able to read information with a larger font size and higher contrast than younger users. For user groups such as the elderly, special services and personalized layout options can be provided, which requires further thinking by health website owners. Another prominent problem is that the user comment submission function is poor. Pang’s study [37] showed that health websites should provide functions for story sharing and memorials for women. These functions provide an outlet for women to share their feelings of grief and loss [37]. Therefore, website designers should focus on personalization and provide a comment section to allow users to submit comments and share experiences on specific content. In addition, China is a multiethnic country, and many ethnic groups have their own languages. Given that health websites are for users nationwide, almost all the websites in this study fail to support multiple languages, which greatly limits the effective dissemination of health information on the websites.

    The quality of online breast cancer treatment information is poor. Online treatment information is of little value to users making breast cancer treatment decisions. Although the evaluation of the value of treatment options presents a “modest” result, the assessment of the benefits and risks of treatment is low, and online papers on breast cancer treatment tend to give compromised and biased advice. Although patients with breast cancer have a clear and urgent need for treatment information [38], the results of this study suggest that doctors are still the most valuable source of information for patients who want to know more about breast cancer treatment. Therefore, breast cancer website developers should provide easy-to-understand online health information that meets the needs of breast cancer patients and is useful for treatment.

    Regarding the suitability of papers on breast cancer treatment, most of the papers have easy-to-understand titles that clearly describe the purpose of the papers. The layout and cultural appropriateness are also good. However, regarding the use of pictures, some papers use pictures with little relevance and that lack explanatory descriptions. Tables are rarely used, and table captions are lacking. Especially with regard to some professional medical knowledge, the lack of descriptions often makes it difficult for users to understand the desired health information on a deeper level. Another important problem we found was that many websites do not give proper explanations of medical terms, which increases the users’ difficulty in reading and increases the level of literacy required to understand the text. Certainly, a few health websites in China have realized this problem, such as 39 Health NetworkTM [39], which provides hyperlinks to detailed explanations of medical terms to help users better understand the health information disseminated. However, most websites fail to do this. Therefore, to better enable users to understand the online health information they seek, health websites should cooperate with professional doctors, nurses, and health care providers, making full use of the professional advantages of medical personnel and providing effective guidance to internet users consuming the online health information.

    Although the results of the manual evaluation of the accuracy of the papers by tumor surgeons indicated that the papers are not obviously wrong, biased, or misleading, most papers quoted others (44/60, 73%) and did not provide references or a resource list (46/60, 77%). We randomly selected eight papers and tried to search them using the Baidu search engine. The results showed that these eight papers exist on a large number of websites at the same time. For papers, indifference to copyright is an urgent problem in the dissemination of online health information in China that must be solved. Website operators should strengthen the copyright awareness of online information and regulate their own information publishing behavior. The problem of protecting the copyright of online health information may also be an important research topic in the future.

    In addition, there are extensive recessive advertisements (the headline or image often contains attractive health-related information, but when you click the link, it turns out to be a page designed to sell products) on all the large health websites, and they are usually embedded in a page as a picture or video. We assessed the accuracy of health knowledge disseminated in several papers that contained recessive advertisements and found no significant errors. However, the existence of such recessive advertisements still leads users to have negative subjective feelings, reduces trust in the content of the papers, and may mislead users with regard to their health behaviors. Once misled by such information, users may choose less mature treatment methods, even leading to the delay of standard treatment [40]. In this regard, we are reminded of the Wei Zexi incident, which was a tragic case of a young man who died because he trusted false medical information on the internet and chose inappropriate treatment for a disease [41]. Although the Technical Manual for the Generation and Dissemination of Health Science Information (Media Edition) and Recommendations for Public Recognition and Utilization of Online Health Information (2017 Edition) were published in 2017, monitoring the reliability and accuracy of online health information remains an important task in China [42]. The Chinese government, however, needs to strengthen the monitoring of the quality of online health information to prevent such information from misleading the public with regard to diagnosis and treatment behavior.

    Limitations

    This study had some limitations. First, we chose influential websites as the research objects of this paper according to their traffic rankings and excluded some websites with low traffic, which caused selection bias. Second, although there are some assessment tools for the readability of written materials (eg, SMOG), to the best of our knowledge, there is no assessment tool for the readability of Chinese written materials. Therefore, an evaluation of the readability of written materials in this study was lacking. Third, video and animation are more important than textual information for users to understand health information. However, due to the lack of relevant evaluation tools, we could not evaluate the quality of these types of multimedia information. Finally, all evaluations were performed by researchers, and their perceptions may differ from those of users. Despite these limitations, the results of this study are still highly valuable for improving the quality of Chinese online health information about breast cancer.

    Conclusion

    In this study, we found that the quality of Chinese breast cancer websites is poor and that the quality of online health information is not ideal. Most websites can provide users with a convenient and easy-to-use breast cancer information retrieval platform, but the breast cancer–related health information they publish is of little value for users making decisions about breast cancer treatment. At the same time, there are also some problems, such as difficulty in tracing the source of information, a lack of copyright awareness, and a lack of advertising supervision. Therefore, developers should design health websites that meet the needs of breast cancer users [43]. Breast cancer users should choose trustworthy health websites that provide accurate information. The government should strengthen the standardized management of health websites to ensure that the health information published on the websites is accurate, up to date, and effective. In addition, the government should strengthen cooperation between websites and medical professional organizations, such as by establishing professional medical customer services and official WeChat accounts, to ensure that users can obtain effective guidance and suggestions from medical professionals when using online health information.

    Acknowledgments

    This study was supported by funding from the National Planning Office of Philosophy and Social Sciences of the People’s Republic of China (grant no. 17AZD037), the “512 Talents Project” of Bengbu Medical College (BBMC; BY51201209), the Innovation Team of Health Information Management and Application Research, BBMC (BYKC201913), and the Natural Science Foundation of Changsha City (grant no. kq2014270). We thank American Journal Experts (AJE) for language editing.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Searching and screening flow for breast cancer websites and papers.

    PNG File , 83 KB
    1. Wang N, Liu S, Yang L, Zhang X, Yuan Y, Li H, et al. Interpretation on the report of Global Cancer Statistics 2018. J Multidiscip Cancer Manage (Electronic Version) 2019 Jan 30;5(1):87-97 [FREE Full text] [CrossRef]
    2. He M, Zhu B, Zhong Y, Wang L, Yang L, Liao X, et al. Analysis of the incidence and mortality trend of breast cancer in Chinese women from 2005 to 2013. Chin J Dis Control Prev 2019 Jan;23(01):10-14 [FREE Full text] [CrossRef]
    3. National Cancer Center. The Latest Cancer Report in 2019 in China. 2019 Jan 30.   URL: http://www.360doc.com/content/19/0130/00/52645714_812065860.shtml [accessed 2019-05-26]
    4. Littlechild SA, Barr L. Using the Internet for information about breast cancer: a questionnaire-based study. Patient Educ Couns 2013 Sep;92(3):413-417 [FREE Full text] [CrossRef] [Medline]
    5. Fritz UAA, Pfaff H, Roth L, Swora M, Groß SE. [Influence of sociodemographic factors on type of and stage at diagnosis in breast cancer. Gesundheitswesen 2020 Sep 16;82(8-09):684-690. [CrossRef] [Medline]
    6. Wang F, Luo A, Luo D, Hu D, Xie W, Chen Z, et al. The association between health information attitudes and skills in patients with chronic disease in China. Glob Health Promot 2016 Dec 09;23(4):59-69. [CrossRef] [Medline]
    7. Kim SC, Shah DV, Namkoong K, McTavish FM, Gustafson DH. Predictors of online health information seeking among women with breast cancer: the role of social support perception and emotional well-being. J Comput Mediat Commun 2013 Jan 25;18(2):98-118 [FREE Full text] [CrossRef] [Medline]
    8. Synnot AJ, Hill SJ, Garner KA, Summers MP, Filippini G, Osborne RH, et al. Online health information seeking: how people with multiple sclerosis find, assess and integrate treatment information to manage their health. Health Expect 2016 Jun;19(3):727-737 [FREE Full text] [CrossRef] [Medline]
    9. Foroughi F, Lam AK, Lim MS, Saremi N, Ahmadvand A. "Googling" for cancer: an infodemiological assessment of online search interests in Australia, Canada, New Zealand, the United Kingdom, and the United States. JMIR Cancer 2016 May 04;2(1):e5 [FREE Full text] [CrossRef] [Medline]
    10. Mazzocut M, Truccolo I, Antonini M, Rinaldi F, Omero P, Ferrarin E, et al. Web conversations about complementary and alternative medicines and cancer: content and sentiment analysis. J Med Internet Res 2016 Jun 16;18(6):e120 [FREE Full text] [CrossRef] [Medline]
    11. Garfinkle R, Wong-Chong N, Petrucci A, Sylla P, Wexner SD, Bhatnagar S, et al. Assessing the readability, quality and accuracy of online health information for patients with low anterior resection syndrome following surgery for rectal cancer. Colorectal Dis 2019 May 04;21(5):523-531. [CrossRef] [Medline]
    12. de Man AM, Rashedi A, Nelen W, Anazodo A, Rademaker A, de Roo S, et al. Female fertility in the cancer setting: availability and quality of online health information. Hum Fertil (Camb) 2020 Sep 19;23(3):170-178. [CrossRef] [Medline]
    13. Sun Y, Zhang Y, Gwizdka J, Trace CB. Consumer evaluation of the quality of online health information: systematic literature review of relevant criteria and indicators. J Med Internet Res 2019 May 02;21(5):e12522 [FREE Full text] [CrossRef] [Medline]
    14. China Internet Network Information Center. The 43rd Statistical Report on Internet Development in China. 2019 Feb 28.   URL: https://www.cnnic.net.cn/hlwfzyj/hlwxzbg/hlwtjbg/201902/t20190228_70645.htm [accessed 2019-05-26]
    15. Liu Y. Construction of an M-Health based information support program for women with breast cancer during diagnosis and treatment process. Second Military Medical University 2017 Jun 1:1-152 [FREE Full text]
    16. Hu Y, Yu J, Ai W. Research on quality evaluation of health information service platform based on user's perspective. Chin J Health Educ 2018 Nov 6:912-915,928. [CrossRef]
    17. Peng Y, Hu Y. Foreign medical and health information service platform for the public and its enlightenments. Chin J Med Libr Inf Sci 2017 May 15;26(05):13-17 [FREE Full text] [CrossRef]
    18. Li Y, Ye S, Zhou Y, Mao F, Guo H, Lin Y, et al. Web-based medical information searching by Chinese patients with breast cancer and its influence on survival: observational study. J Med Internet Res 2020 Apr 17;22(4):e16768 [FREE Full text] [CrossRef] [Medline]
    19. Tang X, Du J, Li J, Dai T. Review on rating instruments for evaluating the quality of health web sites and the framework development in China. Chin J Health Educ 2015 Mar 20;31(03):297-301 [FREE Full text] [CrossRef]
    20. Sun L, Cao J. Applicationinspiration of network health information quality assessment system in oversea. Med Soc 2011 Jul 10;24(07):15-17. [CrossRef]
    21. Qian M, Xu Z, Wang S. Information service quality of online health platform based on user participation. J Chin Soc Sci Tech Inf 2019 Feb 24;38(02):132-142. [CrossRef]
    22. Robins S, Barr HJ, Idelson R, Lambert S, Zelkowitz P. Online health information regarding male infertility: an evaluation of readability, suitability, and quality. Interact J Med Res 2016 Oct 21;5(4):e25 [FREE Full text] [CrossRef] [Medline]
    23. Vivekanantham A, Protheroe J, Muller S, Hider S. Evaluating on-line health information for patients with polymyalgia rheumatica: a descriptive study. BMC Musculoskelet Disord 2017 Jan 26;18(1):43-43 [FREE Full text] [CrossRef] [Medline]
    24. Boyer C, Selby M, Scherrer JR, Appel RD. The Health On the Net Code of Conduct for medical and health websites. Comput Biol Med 1998 Sep;28(5):603-610. [CrossRef] [Medline]
    25. Neumark Y, Flum L, Lopez-Quintero C, Shtarkshall R. Quality of online health information about oral contraceptives from Hebrew-language websites. Isr J Health Policy Res 2012 Sep 24;1(1):38 [FREE Full text] [CrossRef] [Medline]
    26. Cao Z, Li Q, Mo Z. Research on energy saving network information source determination scheme based on Alexa ranking system. Inf Res 2010 Sep 15:87-89.
    27. Alexa. Baidu Encyclopedia.   URL: https://baike.baidu.com/item/Alexa/75129 [accessed 2019-05-20]
    28. Baidu. Baidu Encyclopedia.   URL: https://baike.baidu.com/item/%E7%99%BE%E5%BA%A6/6699 [accessed 2019-05-20]
    29. Li H. Patent analysis of search engine technology of Baidu Company. China Invention Patent 2019 Apr 16;16(04):99-106.
    30. Minervation. The LIDA Instrument: Minervation Validation Instrument for Health Care Web Sites.   URL: http://www.minervation.com/wp-content/uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf [accessed 2019-03-10]
    31. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999 Feb;53(2):105-111 [FREE Full text] [CrossRef] [Medline]
    32. Sowter J, Astin F, Dye L, Marshall P, Knapp P. Assessment of the quality and content of website health information about herbal remedies for menopausal symptoms. Maturitas 2016 Jun;88:16-22. [CrossRef] [Medline]
    33. Doak C, Doak L, Root J. Teaching Patients with Low Literacy Skills. Lippincott, MD: Williams & Wilkins; 2016.
    34. Deng S, Zhao H. Quality evaluation of foreign network health information: a review of indicators, tools and results. Inf Docum Services 2017 Jan 25:67-74.
    35. Ayers SL, Kronenfeld JJ. Chronic illness and health-seeking information on the Internet. Health (London) 2007 Jul;11(3):327-347. [CrossRef] [Medline]
    36. Zhang H, Ma J, Jiang C, Zhou Y, Di J. Evaluation research on health website information quality. J Med Inform 2013 Jul 20;34(7):2-6 [FREE Full text]
    37. Pang P, Temple-Smith M, Bellhouse C, Trieu VH, Kiropoulos L, Williams H, et al. Online health seeking behaviours: what information is sought by women experiencing miscarriage? Stud Health Technol Inform 2018;252:118-125. [CrossRef] [Medline]
    38. Shan Y, Xie Y, Wang H, Huang Y, Hu R. Investigation on health information needs of newly diagnosed breast cancer patients. Nurs J of Chin PLA 2010 Sep 23:1387-1388 [FREE Full text]
    39. Diseases encyclopaedia. 39Health. 2019.   URL: http://jbk.39.net/ [accessed 2019-05-26]
    40. Li Y, Zhou Y, Mao F, Guan J, Lin Y, Wang X, et al. The influence on survival of delay in the treatment initiation of screening detected non-symptomatic breast cancer. Sci Rep 2019 Jul 15;9(1):10158 [FREE Full text] [CrossRef] [Medline]
    41. Ang J. Research on the evolution and guidance of online public opinion based on the lifecycle of incident theory: case studies of the "Wei Zexi incident" and "Lei Yang incident". J DALI Univ 2018;3(5):56-60 [FREE Full text] [CrossRef]
    42. Wang FZ, Luo D, Sun WW, Yan TT, Hu DH, Pan W, et al. Health information literacy and barriers of online health information seeking among digital immigrants in rural China: a preliminary survey. SAGE Open 2019 Jun 12;9(2):215824401985694-215824401985698. [CrossRef]
    43. Pang PC, Chang S, Verspoor K, Pearce J. Designing health websites based on users' web-based information-seeking behaviors: a mixed-method observational study. J Med Internet Res 2016 Jun 06;18(6):e145 [FREE Full text] [CrossRef] [Medline]

    ‎
    BBMC: BengBu Medical College
    HONcode: Health on the Net Foundation Code of Conduct
    ICP: internet content provider
    SAM: Suitability Assessment of Materials
    SMOG: Simplified Measure of Gobbledygook
    SPSS: Statistical Package for Social Sciences
    URL: Uniform Resource Locator

    Edited by R Kukafka, G Eysenbach; submitted 18.11.20; peer-reviewed by C Cheng, M Jordan-Marsh, PCI Pang; comments to author 27.01.21; revised version received 22.03.21; accepted 24.10.21; published 18.11.21

    Copyright

    ©Weiwei Sun, Aijing Luo, Zhiwei Bian, Bin Zhao, Peng Liu, Kai Wang, Yuwen Liu, Wenzhao Xie, Fuzhi Wang. Originally published in JMIR Cancer (https://cancer.jmir.org), 18.11.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on https://cancer.jmir.org/, as well as this copyright and license information must be included.