This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on http://cancer.jmir.org/, as well as this copyright and license information must be included.
Human papillomavirus (HPV) vaccination rates fall far short of Healthy People 2020 objectives. A leading reason is that clinicians do not recommend the vaccine consistently and strongly to girls and boys in the age group recommended for vaccination. Although Web-based HPV vaccine educational interventions for clinicians have been created to promote vaccination recommendations, rigorous evaluations of these interventions have not been conducted. Such evaluations are important to maximize the efficacy of educational interventions in promoting clinician recommendations for HPV vaccination.
The objectives of our study were (1) to expand previous research by systematically identifying HPV vaccine Web-based educational interventions developed for clinicians and (2) to evaluate the quality of these Web-based educational interventions as defined by access, content, design, user evaluation, interactivity, and use of theory or models to create the interventions.
Current HPV vaccine Web-based educational interventions were identified from general search engines (ie, Google), continuing medical education search engines, health department websites, and professional organization websites. Web-based educational interventions were included if they were created for clinicians (defined as individuals qualified to deliver health care services, such as physicians, clinical nurses, and school nurses, to patients aged 9 to 26 years), delivered information about the HPV vaccine and how to increase vaccination rates, and provided continuing education credits. The interventions’ content and usability were analyzed using 6 key indicators: access, content, design, evaluation, interactivity, and use of theory or models.
A total of 21 interventions were identified, out of which 7 (33%) were webinars, 7 (33%) were videos or lectures, and 7 (33%) were other (eg, text articles, website modules). Of the 21 interventions, 17 (81%) identified the purpose of the intervention, 12 (57%) provided the date that the information had been updated (7 of these were updated within the last 6 months), 14 (67%) provided the participants with the opportunity to provide feedback on the intervention, and 5 (24%) provided an interactive component. None of the educational interventions explicitly stated that a theory or model was used to develop the intervention.
This analysis demonstrates that a substantial proportion of Web-based HPV vaccine educational interventions has not been developed using established health education and design principles. Interventions designed using these principles may increase strong and consistent HPV vaccination recommendations by clinicians.
The human papillomavirus (HPV) causes almost all cervical cancers, 50% of vulvar cancers, 65% of vaginal cancers, 90% of anal cancers, and 35% of penile cancers [
Missed opportunities for clinicians to recommend and administer the vaccine, as well as a lack of strong and consistent recommendations by clinicians who do recommend the vaccine, are primary reasons for low HPV vaccination rates in the United States [
To improve clinicians’ HPV vaccine recommendations, numerous Web-based HPV vaccine educational interventions for clinicians have been created. Web-based educational interventions have become a popular delivery method for health care professionals to obtain continuing education (CE) [
Web-based CE interventions created using strategic health communication design principles—communicating effectively with intended users and taking into account audience factors such as culture, race, ethnicity, language, access, functional needs (ie, disabilities), and expectations [
Evaluation of the leading Web-based HPV vaccine educational interventions is important in guiding efforts to promote clinician recommendations for the HPV vaccine [
Rigorous evaluation is a central component of developing successful health education interventions [
We systematically identified current HPV vaccine Web-based educational interventions by examining general search engines (eg, Google), continuing medical education search engines (eg, PedsUniversity, MedScape), health department websites, and professional organization websites. The following search terms and variations of search terms were used within each of the search engines and websites: “clinicians,” “healthcare providers,” “HPV webinars,” “HPV vaccination webinars,” “HPV online education,” and “HPV continuing medical education.” Web-based educational interventions were included if they were (1) created for clinicians (defined as individuals qualified to deliver health care services, such as physicians, clinical nurses, and school nurses, to patients aged between 9 and 26 years); (2) delivered information about the HPV vaccine and how to increase vaccination rates; and (3) provided continuing education credits. Interventions were excluded if they were created for patients, parents, or adolescents, or if they focused on vaccines other than the HPV vaccine. We conducted the search from April 2016 to August 2017 and identified a total of 178 interventions. Of these, 21 interventions met all 3 research criteria for this study.
The study team developed a quality evaluation framework based on criteria established in the literature for evaluating health-related websites [
Internal reliability of the subindicators was calculated using the Krippendorff’s alpha coefficient (K-alpha; for additional information please see De Swert, 2012) [
To examine access of the educational interventions [
Content was evaluated using 7 subindicators [
The design components of the interventions were evaluated by layout and graphics [
Evaluation was assessed using 3 subindicators [
The indicator for interactivity included 2 subindicators [
The theory and models indicator was assessed by examining whether there was an explicit statement that a theory or model was used to develop the intervention (score ranging from 0-1) [
Once interrater reliability was established for the evaluation tool with all indicator scores above .80, 2 independent coders (Mr Bishop and Ms McDonald) utilized the tool to evaluate the educational interventions identified. One of the coders, Mr Bishop is a health education doctoral student with expertise in sexuality education and evaluated the first 11 interventions. The other coder, Ms McDonald is a health education doctoral student with expertise in school health and evaluated the remaining 10 interventions. Frequency distributions were calculated for each of the subindicators to determine specific strength and weaknesses of the interventions. Because this study assessed access, content, and design aspects of interventions and did not include human subjects; this study is considered nonhuman subjects research and hence institutional review board approval was not required.
A total of 21 interventions were identified out of which, 7 (33%) were webinars; 7 (33%) documentary, TV series, or videos; and 7 (33%) other (eg, text article, modules). Medscape, a health information website, provided 10 (48%) interventions, Continuing Nursing Education University provided 2 (10%), CDC provided 3 (14%), professional organizations (eg, American Academy of Pediatrics and Texas Medical Association) provided 3 (14%), nonprofit organizations (eg, Indiana Immunization Coalition) provided 1 (5%), a federally-authorized regional organization (The Suwannee River Area Health Education Center) provided 1 (5%), and a university (Boston University School of Medicine Continuing Medical Education and Continuing Nursing Education) provided 1 (5%).
On the basis of the evaluation indicators, 13 (62%) interventions required registration, but all interventions were accessible without cost to the participant (K-alpha=1.0). Additionally, 17 (81%) educational interventions identified the purpose of the intervention (K-alpha=1.0), and 12 (57%) provided a date when the information had been updated: 7 (33%) were updated in the last 6 months (K-alpha=1.0). In assessing presentation of clear references, 18 (86%) interventions provided references (K-alpha=1.0), and 8 (38%) provided links to other sources (K-alpha=1.0). Most interventions (18/21, 85%) provided reliable references or sources (K-alpha=1.0). The references or sources included information from the CDC (n=16), published peer-reviewed literature (n=16), American Cancer Society (n=5), National Cancer Institute (n=4), Institutes of Medicine (n=4), WHO (n=2), and American Academy of Pediatrics (n=1). Of the 21 interventions, 14 (67%) had a statement of disclosures from the authors, sponsors, or developers (K-alpha=1.0).
For the design subindicators, 2 interventions were documentary or videos that did not include text, and therefore, were not included in the total sample for these subindicators. All interventions (n=19) included easy-to-read font size, font style, color, and line spacing (K-alpha=1.0 for these 3 subindicators). For the subindicator “Graphics were clearly labeled,” only 13 interventions included graphics; thus, the sample for this subindicator is 13 interventions. Out of the 13 interventions, there were 10 (77%) interventions with a minimum of 75% of graphics labeled, 2 (15%) with a minimum of 50% of graphics labeled, and 1 (8%) intervention with a minimum of 25% of graphics labeled. No intervention had 0% of graphics labeled (K-alpha=1.0).
Of the 21 interventions, 17 (81%) included an evaluation for participant outcomes: 17 (81%) assessed HPV and HPV vaccine knowledge, and none assessed attitudes toward HPV and the HPV vaccine. Furthermore, 14 (67%) interventions provided the participants the opportunity to evaluate or provide feedback (K-alpha=1.0). Five (24%) interventions included an interactive component (K-alpha=1.0). The most commonly used interactive component was a polling or knowledge check activity (n=4). No educational intervention explicitly stated a theory or model that was used to develop the intervention.
Web-based educational intervention quality evaluation results (n=21).
Indicator and subindicator | Scoring frequency | ||
Yes | No | ||
Registration required | 13 | 8 | |
Cost | 0 | 21 | |
Date information was updated | 12 | 9 | |
Identification of purpose | 17 | 4 | |
Presentation of clear references | 18 | 3 | |
Links to other sources | 8 | 13 | |
Reliable references and sourcesa | 18 | 3 | |
Statement indicating content was developed or reviewed by experts | 20 | 14 | |
Disclosure of authors, sponsors, or developers | 14 | 7 | |
Font style was easy to readb | 19 | 0 | |
Font size was easy to readb | 19 | 0 | |
Font color and page color contrast was easy to readb | 19 | 0 | |
Line spacing was easy to readb | 19 | 0 | |
Graphics were clearly labeledc | 13 | 0 | |
Evaluation for participant outcomesd | 17 | 4 | |
Participant provided opportunity to evaluate intervention | 14 | 7 | |
Location to direct participant questions | 6 | 15 | |
Included interactive componente | 5 | 16 | |
Theory or model was used to develop intervention | 0 | 21 |
aThe Centers for Disease Control and Prevention (n=16) and published peer reviewed literature (n=16) were the most common frequency cited sources, followed by American Cancer Society (n=5), National Institutes of Health (n=4), Institute of Medicine (n=4), World Health Organization (n=2), Food and Drug Administration (n=1), and the American Academy of Pediatrics (n=1).
bTwo interventions were a documentary or TV series that did not include any type of font or graphics for informational purposes. Therefore, for the Design subindicators font style, font size, font color, and line spacing, the sample size was n=19.
cFor the graphic subindicator, eight interventions did not include graphics for informational purpose. Therefore, the sample size was n=13. There were 10 interventions with a minimum of 75% of graphics labeled, 2 interventions with a minimum of 50% of graphics labeled, 1 intervention with a minimum of 25% of graphics labeled.
dSpecific levels of evaluation for participant outcomes include 17 interventions assessing HPV and HPV vaccine knowledge, and no intervention assessing attitudes toward HPV and the HPV vaccine.
eFive interventions provided participant interactivity. Four interventions included an interactive knowledge check, and 1 intervention included directions to email reminders and newsletters.
Quality summary scores for Web-based interventions.
Intervention titlea | Summary score (out of 25) |
HPV Vaccine Safety and Efficacy | 20 |
HPV Vaccines: Updates and Clinical Perspective | 20 |
Increasing Adolescent Immunization Coverage | 20 |
Don’t Wait Vaccinate! The Prevention of HPV Cancers (Part 2) | 19 |
HPV Vaccination is Cancer Prevention (2017 Update) | 19 |
Overcoming Gender and Socioeconomic Disparities in HPV Vaccination | 19 |
You are the Key to HPV Cancer Prevention | 18b |
Don’t Wait Vaccinate! The Prevention of HPV Cancers | 17 |
Immunization: You Call the Shots-Module Eight-HPV, 2016 | 17 |
Immunization: You Call the Shots-Module Eighteen—Vaccine Administration | 17c |
You are the Key to HPV Cancer Prevention | 17d |
ACIP Releases Pediatric Vaccine Schedule | 16c |
Adolescent Immunizations: Strongly Recommending the HPV Vaccine | 16 |
AAP Provides Guidance for Parents Who Refuse Vaccination | 15c |
ACIP Releases Adult Vaccine Recommendations | 15c |
CDC Updates Guideline Recommendations for HPV Vaccination | 15c |
Human Papillomavirus (HPV) Vaccine Safety | 15c |
The Story of HPV: Yesterday, Today, and Tomorrow | 14 |
HPV Vaccines: Updates and Clinical Perspective | 13 |
Putting HPV Vaccine Knowledge Into Practice | 7e |
HPV Documentary—Division of Continuing Medical Education | 2e |
aHPV: Human Papillomavirus; ACIP: Advisory Committee on Immunization Practices; AAP: American Academy of Pediatrics; CDC: Centers for Disease Control and Prevention.
bYou are the key to HPV Cancer Prevention intervention published 9/2/2015 and expires 9/7/2017.
cThese interventions did not include any type of graphics for informational purpose. Therefore, the total score is out of 24.
dYou are the key to HPV Cancer Prevention intervention published 4/21/2016 and expires 4/21/2018.
eThese interventions were documentaries and did not include any type of font or graphics for informational purposes. Therefore, the total score is out of 20.
This study provides a systematic, evidence-based assessment of the strengths and weaknesses of current HPV vaccine Web-based educational interventions. Strengths of the assessed Web-based educational interventions include: (1) being developed by experts in the field; (2) providing reliable references or sources; (3) providing clinicians with access to CEs for no cost; (4) following basic design principles with easy-to-read fonts, colors, and graphics; and (5) consistently providing evaluation opportunities for participant knowledge outcomes. Weaknesses of the educational interventions included lack of: (1) evaluation of outcomes including participants’ attitudes about HPV vaccination, intention to recommended vaccination, and recommendation of behaviors; (2) theory-based interventions; (3) opportunity for participants to provide feedback or evaluation of the intervention; (4) links to other sources or resources; and (5) interactivity throughout the intervention.
HPV vaccination rates are well below the Healthy People 2020 objective [
We found that none of the interventions examined included a statement that a theory was used to create the intervention. To improve outcome behaviors and increase clinician HPV vaccine recommendation behaviors, intervention developers should utilize science and evidence that supports effective medical education and behavior change [
None of the Web-based educational interventions included in this study evaluated HPV attitudes, intention to recommend vaccination, or actual recommendation behavior. This is concerning given that clinician attitudes are an important predictor of vaccine recommendations. Clinicians have reported concerns regarding HPV vaccine safety [
Although face-to-face educational interventions have shown to improve clinicians’ willingness to provide immunizations and routinely screen immunization records at visits [
While this study provides innovative insight into the quality of Web-based HPV vaccine educational interventions created for clinicians, there are limitations that should be considered. First, only Web-based educational interventions were evaluated, and these results cannot be generalized to other types of interventions such as face-to-face lectures, grand rounds, or seminars. There would be substantial benefit to conducting evaluations of face-to-face lectures and seminar materials to assess all venues of continuing education for clinicians regarding the HPV vaccine. Second, this quality evaluation did not assess participants’ experience of the intervention and therefore, cannot identify every area for improvement. Data were not collected from participants themselves regarding usability: this study identified only 7 indicators of usability. Third, this study was a quality evaluation and did not evaluate participant outcomes (eg, knowledge, attitudes, recommendation behaviors) after completing the intervention. Fourth, the evaluation was conducted using only the materials that were accessible at the time of the study, and there is the potential that a component (such as, a follow-up emailed evaluation after the intervention to participants) was not included in this evaluation. Despite these limitations, these findings provide valuable information for those who develop Web-based continuing education interventions regarding HPV vaccines, by providing a quantitative approach to identifying the design and usability strengths and weaknesses of HPV vaccine Web-based educational interventions.
The data resulting from this study have the potential to help shift current research practice paradigms. The findings suggest that those who develop Web-based educational interventions to promote HPV vaccine recommendations utilize design science principles, a powerful approach and process that includes participatory action research to iteratively develop and evaluate health education interventions [
The quality evaluation of these interventions demonstrated that Web-based interventions were based on reliable sources, developed by experts, and were created with critical design aspects (eg, font style, size, and color were easy to read, graphics were clearly labeled). However, there were limited outcome evaluations for users measuring attitudes, intentions, or behaviors, and lack of user interactivity. Results from this study suggest best practices for designing, refining, and implementing Web-based interventions to promote HPV vaccination within the clinician population.
Intervention characteristics.
American Academy of Pediatrics
Advisory Committee on Immunization Practices
Centers for Disease Control and Prevention
continuing education
human papillomavirus
No funding was received for this study.
BLR is the primary author, JMB conducted intervention evaluation and assisted in writing the manuscript, SLM conducted intervention evaluation and assisted in writing the manuscript, JK provided expert guidance for the evaluation and revised the manuscript, and GLK provided overall guidance for the evaluation and provided extensive revisions of the manuscript.
None declared.