To minimize the bias introduced by personalized recommendation algorithms, we used 3 tactics: creating a new Douyin account specifically for evaluation, disabling Douyin’s personalized recommendations to eliminate differential content recommendations caused by user habits, and banning access to mobile location services. All videos were viewed without any actions such as downloading, liking, commenting, collecting, or sharing. Evaluation tasks were carried out by 2 qualified radiologists (Chuangying Zhu and HY) from the division of radiology in a tertiary teaching hospital.

The keywords “钼靶” (“mammography” in Chinese) and “钼靶检查” (“mammography screening” in Chinese) were searched in the Douyin app on October 22, 2023, with no limits placed on the release time. Douyin offers 3 ways to filter videos: overall ranking, most recent, and most likes. We used the overall ranking mode to retrieve the top 100 videos because most consumers use this default sorting option. We chose the threshold number of 100 for 2 reasons. First, Douyin’s search function takes topic relevance into account; the most relevant mammography videos tend to appear at the top of the results list, and it is difficult to observe any pertinent videos when the results exceed 100. Second, most general health users apply the “least effort” principle when searching for information on the web; they tend to concentrate on the top search results. In this study, we included videos directly related to mammography. The exclusion criteria were videos not in Chinese, videos not related to mammography, duplicate videos, videos shorter than 10 seconds in length, and videos that were unavailable.

A Microsoft Excel spreadsheet was created by a researcher for data collection. Video information analyzed in this study was the identity of the uploader; the duration in seconds; the number of “likes” as indicated by the heart icon; the number of comments, saves, and shares the video received; and the number of days since the video was uploaded.

We divided the videos into 2 main groups according to whether the uploaders were professional or nonprofessional. Professional videos consisted of videos uploaded directly by board-certified physicians, health channels, and hospital channels. Most health channels and hospital videos were narrated by doctors. Nonprofessional videos included those uploaded by patients and other individuals.

The quality and reliability of the video were evaluated based on the following criteria: the accuracy and comprehensiveness of the content, the clarity and fluency of information delivery, and the overall usefulness of the video to its intended audience.

No validated tools for assessing mammography video content are available in the literature. According to the American Cancer Society recommendations for the early detection of breast cancer [28] and the China Anti-Cancer Association Breast Cancer Diagnosis and Treatment Guide and Standard (2024 Edition) [6], 2 qualified radiologists (Chunmin Zhu and RH) from the division of radiology in a tertiary teaching hospital, with more than 10 years of experience in the radiological profession, developed a completeness checklist to assess the quality of mammography video content (Table 1). The 7 categories cover most aspects of mammography: conception, examination process, applicable objects, precautions, combined other examinations, advantages, and report. A video was awarded 1 point in each domain if it mentioned the content listed in Table 1, resulting in a final score ranging from 0 to 11. A score of 0 indicated that there was no accurate content in any of the 7 earlier-mentioned areas of mammography, whereas a score of 11 indicated that a video contained accurate information in all areas. Videos were then further categorized as useful or unhelpful according to the final score. Videos with a score of 0 were considered unhelpful if they only dealt with personal experiences or testimonies without providing any scientific content, whereas useful videos received a score of ≥1. We used the modified DISCERN (mDISCERN) tool and the Global Quality Score (GQS), previously used in many studies of Douyin, as instruments to assess the quality of information in each video.

The DISCERN criteria are a validated scoring system developed by an Oxford University research team to assess the information quality and reliability of content related to consumer health information on treatment options [29]. The mDISCERN tool was modified by Singh et al [30] and is based on a 5-point Likert scale that examines goals, reliability of information sources, bias, areas of uncertainty, and additional sources. According to the mDISCERN score, the reliability of video content is considered good for a DISCERN score of >3 points, moderate for a DISCERN score of 3 points, and poor for a DISCERN score of <3 points.

The GQS, which was developed by Bernard et al [31], is a 5-point Likert scale used to assess the quality of a video based on the flow of information, completeness of the information presented on a particular topic, and usefulness of information to patients. A GQS of 1 is considered very poor, 2 is considered poor, 3 is considered fair, 4 is considered good, and 5 is considered excellent. The detailed information for mDISCERN and GQS is available on the web as in Multimedia Appendices 1 and 2.

Before starting to score the videos, radiologists first reviewed the official DISCERN and GQS instructions and referred to a simplified Chinese version [32], the latter is more adapted to the Chinese language and culture. To ensure consistency, prescoring discussions were mandatory. After reaching a consensus on the first 20 videos, the evaluators independently reviewed the subsequent entries. The original scores of the 2 radiologists (C Zhou and HY) were independently recorded. The scores of mDISCERN and GQS given by the 2 researchers (C Zhou and HY) were averaged to obtain an overall score, which was then used in the analysis. Any disagreements about the completeness checklist were resolved by consensus.

SPSS (version 27.0; IBM Corp) was used for data entry and analysis. Data are summarized as frequency (n) and percentage (%) for categorical variables and median (IQR) for ordinal variables. The normality of the data was analyzed using the Shapiro-Wilk test. Because the data were not normally distributed, the Mann-Whitney U test was used to compare the continuous variables between the 2 groups. Cronbach α coefficients were used to calculate the agreement between the 2 researchers. Spearman correlation tests were used to assess relationships between parameters. The correlations were interpreted based on the magnitude of the Spearman correlation coefficient (r), with the following thresholds used as a guide to describe the strength of the relationships: r<0.1 is considered a negligible correlation, 0.1≤r<0.4 is a weak correlation, 0.4≤ r<0.7 is a moderate correlation, 0.7≤r< 0.9 is a strong correlation, and r≥0.9 is a very strong correlation. These thresholds were adapted from conventional guidelines for interpreting correlation coefficients, as discussed in the literature [33]. Differences were considered statistically significant at a P value of <.05.

No clinical data, human specimens, or laboratory animals were involved in this study. All information used in this study was obtained from publicly released Douyin videos, and none of the data involved personal privacy. In addition, the study did not involve any interaction with users; therefore, no ethics review was required. All data were deidentified, and no individual users, videos, or screenshots are identifiable in this manuscript or its supplementary materials.

In total, 200 videos were screened, and 136 were included in the study. The 64 excluded videos were 1 video in a non-Chinese language, 5 irrelevant videos, 48 duplicate videos, 7 short videos, and 3 unavailable videos (Figure 1).

The statistical analysis showed that the mammography videos ranged from 11 to 876 seconds. At the time of this study, the 136 short mammography videos had received 1,788,786 likes, 288,802 comments, 110,224 saves, and 598,393 shares. Each short video received 2 to 256,000 likes, 0 to 81,000 comments, 0 to 21,000 saves, and 0 to 145,000 shares. The most recent video was posted 21 days before the data collection, while the oldest had been on Douyin for more than 3 years. The median duration of the videos was 49.5 (IQR 32.5‐76.75) seconds; the median indicators of engagement comprised 414 (IQR 155.75‐1887.25) likes, 50.5 (IQR 20‐286.25) comments, 56 (IQR 19-201.75) saves, and 61.5 (IQR 12‐275.75) shares; and the median time since upload was 382.5 (IQR 116.25‐635.75) days. The characteristics of the included videos are shown in Table 2.

Most of the videos in our sample were contributed by professional users (112/136, 82.4%), while a relatively small proportion were contributed by nonprofessional users (24/136, 17.6%). Among professional users, most videos were created by board-certified physicians, followed by hospital channels and health channels (Table 3). Only 1 imaging physician was involved in the posting of 3 videos. The median video duration was significantly longer (P<.001) in the nonprofessional group and received significantly more comments (P=.004; Table 4).

As mentioned earlier, the selected videos were divided into useful and unhelpful groups based on scores of the completeness checklist. Of the 136 selected videos, the number of videos containing useful and unhelpful information was 105 (77.2%) and 31 (22.8%), respectively. Notably, despite uniformity in video days since upload between groups, uploads by unhelpful groups garnered more engagement metrics such as likes (median 6892, IQR 585‐104,000), comments (median 1305, IQR 130‐4103), saves (median 748, IQR 53-4381), and shares (median 1056, IQR 50‐6071) relative to useful group, and this differential attains statistical significance (P<.01 for all; Table 5). Because the number of nonprofessional uploaders in the useful group was small (7/105), we could not compare this group.

Useful videos were analyzed based on the information they contained. Among all the categories, the advantages of mammography were the most frequently covered topic (53/105, 50.5%), followed in descending order by applicable objects (50/105, 47.6%), conception (47/105, 44.8%), examination process (44/105, 41.9%), combined other examinations (42/105, 40%), report (26/105, 24.8%), and precautions (11/105, 10.5%; Multimedia Appendix 3). Most of these videos (97/105, 92.4%) scored <5 points, and only 1 video received a maximum score of 7.

The median (IQR) mDISCERN score and GQS of all videos were 2 (1‐2) and 2.5 (1.63‐3), respectively. The Cronbach α coefficients for reliability between the raters were 0.94 and 0.97 for the GQS and mDISCERN, respectively. The mDISCERN score and GQS of the videos in the useful and professional groups were significantly higher than those in the unhelpful and nonprofessional groups (all P<.001).

Spearman correlation analysis revealed certain correlations among the characteristics of the videos. The video duration was positively correlated with the number of comments (r=0.23; P=.008), saves (r=0.20; P=.02), and shares (r=0.19; P=.02). Across all videos, Spearman correlation analysis revealed positive and significant correlations among the number of likes, comments, saves, shares, and days since upload (P<.05 for each pair).

The GQS was negatively or positively correlated with the number of likes (r=−0.24; P=.004), comments (r=−0.29; P<.001), and saves (r=−0.20; P=.02) as well as with the mDISCERN score (r=0.65; P<.001). The mDISCERN score was found to be negatively correlated with the number of likes (r=−0.26; P=.002), comments (r=−0.36; P<.001), saves (r=−0.22; P=.009), and shares (r=−0.18; P=.03). The correlation coefficients (r) reported in this study are generally below 0.39, indicating weak associations. In cases where the correlation coefficients are below 0.1, we consider these to be negligible. We acknowledge that the statistical significance of these correlations may be influenced by the sample size; therefore, we place greater emphasis on the magnitude of the correlation coefficients to better reflect the strength of the relationships. More detailed analytical results are shown in Table 6.

This is the first study in the literature to evaluate Douyin content on mammography videos. According to the findings of 2 independent reviewers (C Zhou and HY), more than three-quarters of the videos were uploaded by professional individuals or institutions, and videos containing content primarily concerned with disease knowledge were of higher quality and more reliable. Nevertheless, the overall quality of the mammography videos was poor according to the completeness checklist, GQS, and mDISCERN score. Additionally, the fact that seekers gave higher ratings to the lower-quality videos than the higher-quality videos suggests that most health viewers are not able to identify poor-quality medical information in videos.

The rapid development of digital technology and the widespread application of mobile intelligent terminals have caused various new media to become important platforms for sharing and exchanging scientific knowledge. This has further expanded the channels through which the public can understand and obtain information, broadening the breadth and depth of knowledge. There was an unprecedented reliance on social media platforms to seek information during the COVID-19 pandemic [34]. Douyin is a representative national short video platform, and watching videos every day has become a part of many people’s lives.

Currently, uploaders who share health information on the Douyin app are required to obtain certification materials that verify their affiliation with tertiary A hospitals as doctors. In our study, approximately 80% of mammography-related Douyin contents were uploaded by professional users. Most of them were clinicians; only 1 was an imaging specialist. These findings show that clinicians in tertiary A hospitals with a high level of expertise are enthusiastic about participating in the popularization of mammography-related information. A previous study also showed that radiology-related content on the increasingly popular social media platform TikTok is mainly posted by nonphysician radiology personnel [35]. In addition, our results suggest that the videos cannot cover all aspects of mammography, which may be due to the limited short length of Douyin videos. Furthermore, the most prevalent content of the videos was the advantages of mammography in detecting calcifications; few videos fully addressed other types of content during the examination. This finding may indicate that most publishers believe that the unique advantage of mammography is to help detect breast cancer at an earlier stage. As radiologists, they may be more likely to focus on pain and positioning or precautions during the examination and have a more accurate understanding of diagnostic reports [36]. Pain and discomfort during mammography may influence participation in screening programs and be detrimental to cancer prevention efforts [37]. More senior radiologists should be encouraged to become involved in mammography popularization. Specialized training and publicity should be provided to meet the public’s need for knowledge about mammography.

The current results indicate that the reliability and educational quality of mammography-related videos on Douyin are unsatisfactory, with median mDISCERN and GQS evaluations across all videos stood at 2.5 (IQR 1.63-3), and 2 (IQR 1-2), respectively. This finding is in accord with previous studies that have examined low-quality videos on various health topics and found that this information may not be reliable on Douyin [38,39]. Studies on other video platforms, such as YouTube, also showed that the overall quality of videos providing disease information was poor [40,41]. Because the content of most videos lacks peer or institutional quality review, many may not be subject to quality control and may not be evidence-based; it is therefore not surprising that much of this content is inadequate [42]. Thus, patients should access certified organizations and sites such as those certified with the Health on the Net Foundation Code of Conduct certificate to obtain professional information and avoid being misled by social media. The Health on the Net Foundation Code of Conduct was created as a practical solution to help internet users recognize reliable health-related information on the internet while distinguishing it from potentially erroneous or hazardous content [43]. However, contrary to all these findings, in previous studies of the quality of Douyin videos on children with humeral supracondylar fractures, chronic obstructive pulmonary disease, and cosmetic surgery, the overall information quality and reliability of these short videos were satisfactory in China [44-46]. This might be explained by the assessment instrument’s lack of comparability between different disease categories and the bias introduced by the use of different scoring criteria among different researchers.

The results also showed that videos posted by professionals had significantly higher reliability and GQS than those posted by individuals. This finding indicates that ownership is an important element that can be used to assess the reliability of videos. Video content may be considered trustworthy when produced by professionals such as doctors, medical organizations, and health information websites [47]. Unfortunately, our regression analysis revealed that the number of likes, comments, and saves had a weak negative correlation with both the mDISCERN score and GQS. The results showed that lay users had difficulty distinguishing useful information from a large number of videos. A common misconception is that digital information accuracy is directly related to the number of hits or views [48]. There are thousands of health-related videos promoting misleading information that get millions of views, such as videos that disparage vaccinations [49,50]. These results also indicate that effective regulatory measures are needed to control scientifically accredited information. In the future, it would be beneficial to develop an algorithm that ranks videos preferentially uploaded by a trusted medical center or professional. If the public had less access to unhelpful videos, the damage could be less.

This study has some limitations. First, this was a cross-sectional study that examined a very small portion of a very large amount of data. The number of views, likes, and dislikes of health-related videos on the internet changes over time. The “snapshot” approach to data collection seems to be the main limitation of this study because the results may vary with the use of different search terms and according to the date and time of the search. Second, because of the limitations of the search criteria, it was not possible to include all video resources that fit the topic of this study. Although we included a relatively small percentage of videos, we considered it to be sufficiently representative, as videos beyond the top 100 have no significant impact on the analysis. Third, we only included videos uploaded on Douyin, which is a Chinese video-sharing platform; thus, the findings may not be generalizable to other social media platforms (eg, YouTube) or to other countries. Subsequent cross-linguistic research is required to fill this gap. Finally, the GQS and DISCERN are subjective assessment tools. Although 2 independent experts (C Zhou and HY) determined the ratings iteratively and used Cronbach α coefficients to quantify the agreement between the 2 raters, subjective differences still cannot be ignored. Looking ahead, future research should include broader cross-linguistic comparative studies, using more appropriate assessment instruments to validate our findings.

According to the findings of our study, a majority of the Douyin videos concerning mammography were uploaded by clinicians and exhibited poor quality and reliability. Patients should not use these videos as the only source of information about mammography because they may lead to misdirected or inappropriate interventions. Douyin is often used to obtain health-related information, and radiologists should be encouraged to provide useful and accurate videos and to instruct patients appropriately. From the standpoint of preventing and curing breast cancer, there is a need for stricter standards and procedures for video publishing to improve the quality of medical content.