JC JMIR Cancer JMIR Cancer 2369-1999 JMIR Publications Toronto, Canada v4i1e3 29453187 10.2196/cancer.9114 Original Paper Original Paper Quality of Web-Based Educational Interventions for Clinicians on Human Papillomavirus Vaccine: Content and Usability Assessment Eysenbach Gunther Chien Tsair-Wei Dela Cruz May Rose Rosen Brittany L CHES, PhD 1
School of Human Services University of Cincinnati PO Box 210068 Cincinnati, OH, 45221 United States 1 513 556 3872 brittany.rosen@uc.edu
http://orcid.org/0000-0002-3151-9927 Bishop James M MEd, CHES 1 http://orcid.org/0000-0002-0731-9407 McDonald Skye L MS, CHES 1 http://orcid.org/0000-0002-8956-7470 Kahn Jessica A MD, MPH 2 3 http://orcid.org/0000-0002-6313-4976 Kreps Gary L PhD 4 http://orcid.org/0000-0001-5980-129X 1 School of Human Services University of Cincinnati Cincinnati, OH United States 2 Department of Pediatrics Cincinnati Children's Hospital Medical Center Cincinnati, OH United States 3 College of Medicine University of Cincinnati Cincinnati, OH United States 4 Center for Health and Risk Communication Department of Communication George Mason University Fairfax, VA United States Corresponding Author: Brittany L Rosen brittany.rosen@uc.edu Jan-Jun2018 16 02 2018 4 1 e3 4 10 2017 29 11 2017 13 12 2017 13 12 2017 ©Brittany L Rosen, James M Bishop, Skye L McDonald, Jessica A Kahn, Gary L Kreps. Originally published in JMIR Cancer (http://cancer.jmir.org), 16.02.2018. 2018

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on http://cancer.jmir.org/, as well as this copyright and license information must be included.

 Background

Human papillomavirus (HPV) vaccination rates fall far short of Healthy People 2020 objectives. A leading reason is that clinicians do not recommend the vaccine consistently and strongly to girls and boys in the age group recommended for vaccination. Although Web-based HPV vaccine educational interventions for clinicians have been created to promote vaccination recommendations, rigorous evaluations of these interventions have not been conducted. Such evaluations are important to maximize the efficacy of educational interventions in promoting clinician recommendations for HPV vaccination.

 Objective

The objectives of our study were (1) to expand previous research by systematically identifying HPV vaccine Web-based educational interventions developed for clinicians and (2) to evaluate the quality of these Web-based educational interventions as defined by access, content, design, user evaluation, interactivity, and use of theory or models to create the interventions.

 Methods

Current HPV vaccine Web-based educational interventions were identified from general search engines (ie, Google), continuing medical education search engines, health department websites, and professional organization websites. Web-based educational interventions were included if they were created for clinicians (defined as individuals qualified to deliver health care services, such as physicians, clinical nurses, and school nurses, to patients aged 9 to 26 years), delivered information about the HPV vaccine and how to increase vaccination rates, and provided continuing education credits. The interventions’ content and usability were analyzed using 6 key indicators: access, content, design, evaluation, interactivity, and use of theory or models.

 Results

A total of 21 interventions were identified, out of which 7 (33%) were webinars, 7 (33%) were videos or lectures, and 7 (33%) were other (eg, text articles, website modules). Of the 21 interventions, 17 (81%) identified the purpose of the intervention, 12 (57%) provided the date that the information had been updated (7 of these were updated within the last 6 months), 14 (67%) provided the participants with the opportunity to provide feedback on the intervention, and 5 (24%) provided an interactive component. None of the educational interventions explicitly stated that a theory or model was used to develop the intervention.

 Conclusions

This analysis demonstrates that a substantial proportion of Web-based HPV vaccine educational interventions has not been developed using established health education and design principles. Interventions designed using these principles may increase strong and consistent HPV vaccination recommendations by clinicians.

 papillomavirus vaccines internet program evaluation health personnel Introduction

The human papillomavirus (HPV) causes almost all cervical cancers, 50% of vulvar cancers, 65% of vaginal cancers, 90% of anal cancers, and 35% of penile cancers [1]. Recent studies have shown that the incidence of oral cancers caused by HPV is increasing [2-4]. The 9-valent HPV vaccine (9vHPV) is nearly 100% effective in preventing precancerous lesions caused by 7 genotypes [5], accounting for 81% of cervical cancer cases [6]. Despite ample evidence that licensed HPV vaccines are safe to use and effective in preventing certain anogenital cancers, only 42% of adolescent females and 28% of adolescent males have completed the HPV vaccine series [7]. These HPV vaccination rates fall short of Healthy People 2020’s objective of 80% coverage for girls and boys aged between 13 and 15 years [8].

Missed opportunities for clinicians to recommend and administer the vaccine, as well as a lack of strong and consistent recommendations by clinicians who do recommend the vaccine, are primary reasons for low HPV vaccination rates in the United States [9]. Factors contributing to missed clinical recommendation opportunities and insufficiently strong and consistent recommendations include providers’ limited knowledge of HPV and the vaccine, discomfort discussing a topic related to sexual behavior, concerns about vaccine safety, parental resistance, preference for vaccinating older adolescents, and lack of time or incentive to educate parents about the vaccine [10-19]. While improving communication between health care providers and parents is considered a critical component to increasing vaccination rates [20-26], health care providers report they do not feel well-prepared to provide strong vaccination recommendations [15,27,28]. Additionally, 75% of clinicians expressed they would benefit from continuing education about recommending the HPV vaccine [29].

To improve clinicians’ HPV vaccine recommendations, numerous Web-based HPV vaccine educational interventions for clinicians have been created. Web-based educational interventions have become a popular delivery method for health care professionals to obtain continuing education (CE) [30], as Web-based interventions provide an opportunity to quickly update and address health education topics at low cost [31]. With rapid proliferation of Web-based health education programs, there is a potential risk of neglecting fundamental health communication and education design principles in these programs that are important to ensure their efficacy [32-34].

Web-based CE interventions created using strategic health communication design principles—communicating effectively with intended users and taking into account audience factors such as culture, race, ethnicity, language, access, functional needs (ie, disabilities), and expectations [35,36]—are likely to increase clinicians’ knowledge, perceptions, attitudes, and practice behaviors [37-42]. Numerous Web-based HPV vaccine educational interventions have been introduced to accommodate clinicians’ educational needs [43]. However, an extensive and comprehensive review (of CINAHL, MEDLINE, ERIC, and Ebsco Academic Complete databases, using variations and Boolean connectors with the following terms: “online intervention,” “online program,” “HPV vaccine,” “clinicians,” “physicians,” “nurses and evaluation”) demonstrated that there has been no rigorous evaluation of the design, content, and usability levels of these programs. Without such evaluation data, it is unknown whether such interventions are achieving their intended outcomes, and which aspects of the interventions should be refined [44,45].

Evaluation of the leading Web-based HPV vaccine educational interventions is important in guiding efforts to promote clinician recommendations for the HPV vaccine [44,46]. Evaluation of Web-based interventions, using established health communication and education design principles can (1) identify strengths and weaknesses of educational interventions from the users’ perspective; (2) determine whether interventions are worth the time, resources, and expense for continued implementation; and (3) provide the evidence for designing optimally effective interventions [47]. Additionally, evaluation data can provide insights into any negative unintended consequences of the interventions, such as boomerang and iatrogenic effects [48,49] and ensures that interventions address audiences’ unique needs, culture, and expectations [35]. Evaluation research is vital not only to improve health outcomes but also to ensure that interventions are cost-effective [50]. Estimated health care cost in the United States was $2.7 trillion (18% of the gross domestic product) in 2011 [51], leading policy makers to prioritize identifying effective, evidence-based methods to prevent disease and manage rising health costs [50].

Rigorous evaluation is a central component of developing successful health education interventions [45,52] and essential for understanding clinicians’ educational needs and assessing outcomes [50]. However, current Web-based HPV vaccine interventions developed for clinicians have not been evaluated using health education and design principles. Therefore, the objectives of the study were to: (1) systematically identify HPV vaccine Web-based educational interventions developed for clinicians, and (2) evaluate the quality of Web-based educational interventions as defined by access, content, design, evaluation, interactivity, and use of theory or models to create the interventions.

 Methods Sample

We systematically identified current HPV vaccine Web-based educational interventions by examining general search engines (eg, Google), continuing medical education search engines (eg, PedsUniversity, MedScape), health department websites, and professional organization websites. The following search terms and variations of search terms were used within each of the search engines and websites: “clinicians,” “healthcare providers,” “HPV webinars,” “HPV vaccination webinars,” “HPV online education,” and “HPV continuing medical education.” Web-based educational interventions were included if they were (1) created for clinicians (defined as individuals qualified to deliver health care services, such as physicians, clinical nurses, and school nurses, to patients aged between 9 and 26 years); (2) delivered information about the HPV vaccine and how to increase vaccination rates; and (3) provided continuing education credits. Interventions were excluded if they were created for patients, parents, or adolescents, or if they focused on vaccines other than the HPV vaccine. We conducted the search from April 2016 to August 2017 and identified a total of 178 interventions. Of these, 21 interventions met all 3 research criteria for this study.

 Instrument

The study team developed a quality evaluation framework based on criteria established in the literature for evaluating health-related websites [53-56]. The quality evaluation framework assessed Web-based interventions using 6 key indicators: access, content, design, evaluation, interactivity, and theory or models [53-56]. Each key indicator was scored using various subindicators: higher scores for the indicators designated higher quality interventions.

Internal reliability of the subindicators was calculated using the Krippendorff’s alpha coefficient (K-alpha; for additional information please see De Swert, 2012) [57]. This coefficient was selected because it provides information on the reliability of variables, not coders, and its robust calculations are not impacted by sample size, multiple coders, or missing data [58]. After calculating Krippendorff’s alpha coefficient for 2 rounds of samples with 2 independent coders (Dr Rosen, a PhD trainer faculty member with expertise in HPV vaccination uptake, and a doctoral graduate research assistant in health education), the evaluation tool was considered to be internally reliable given that all indicator scores were above .80, which is considered the norm for acceptable reliability [57].

Access

To examine access of the educational interventions [53,54], 2 subindicators were used to measure different components of access. These subindicators included whether registration was required to access the intervention (score ranging from 0-1) and the cost of the intervention (score ranging from 0-1).

 Content

Content was evaluated using 7 subindicators [53,55]: identification of purpose (score ranging from 0-1), date on which the information was updated (score ranging from 0-1), presentation of clear references (score ranging from 0-1), and links to other sources (score ranging from 0-1). Additionally, reliable sources (score ranging from 0-1) were assessed and whether the intervention included reliable sources, the type of source (eg, Centers for Disease Control and Prevention, National Institutes of Health, published peer-reviewed literature) was included in the scoring metric. The final 2 subindicators included a statement indicating that content was developed or reviewed by experts (score ranging from 0-1) and a statement of disclosure of authors, sponsors, or developers (score ranging from 0-1).

 Design

The design components of the interventions were evaluated by layout and graphics [53]. The layout of the intervention was assessed by examining font and line spacing. Specifically, font was assessed by whether the style was easy to read (score ranging from 0-1), size was easy to read (score ranging from 0-1), text color and page color contrast were easy to read (score ranging from 0-1), and line spacing was easy to read (score ranging from 0-1). Graphics were assessed to determine if they were clearly labeled, and scores ranged from 0 to 3 with 0 indicating 0% of graphics were labeled, 1 indicating a minimum of 25% of the graphics were labeled, 2 indicating a minimum of 50% of the graphics were labeled, and 3 indicating a minimum of 75% of the graphics were labeled.

 Evaluation

Evaluation was assessed using 3 subindicators [53-55]: whether participant outcomes were evaluated (eg, knowledge and attitudes; score ranging from 0-1), the level of that evaluation (score ranging from 0-2; 0 indicating no evaluation, 1 indicating an evaluation of HPV or HPV vaccine knowledge, and 2 indicating an evaluation of HPV or HPV vaccine attitudes) and whether the participant was provided an opportunity to evaluate the intervention (score ranging from 0-1).

 Interactivity

The indicator for interactivity included 2 subindicators [54,55]. The first subindicator assessed whether there was a location for participants to direct questions during the educational intervention (score ranging from 0-1). The second subindicator assessed whether the intervention included any interactive components (score ranging from 0-1). If the intervention included any interactive component, the interactive component was recorded in the scoring metric. The interactive components included discussion boards, “ask the expert” bulletin boards, sign up for email reminders, sign up for newsletters, and other interactive components.

 Theory and Models

The theory and models indicator was assessed by examining whether there was an explicit statement that a theory or model was used to develop the intervention (score ranging from 0-1) [54-56]. If a theory or model was used to develop the intervention, the theory or model was recorded in the scoring metric.

 Procedure

Once interrater reliability was established for the evaluation tool with all indicator scores above .80, 2 independent coders (Mr Bishop and Ms McDonald) utilized the tool to evaluate the educational interventions identified. One of the coders, Mr Bishop is a health education doctoral student with expertise in sexuality education and evaluated the first 11 interventions. The other coder, Ms McDonald is a health education doctoral student with expertise in school health and evaluated the remaining 10 interventions. Frequency distributions were calculated for each of the subindicators to determine specific strength and weaknesses of the interventions. Because this study assessed access, content, and design aspects of interventions and did not include human subjects; this study is considered nonhuman subjects research and hence institutional review board approval was not required.

 Results Intervention Characteristics

A total of 21 interventions were identified out of which, 7 (33%) were webinars; 7 (33%) documentary, TV series, or videos; and 7 (33%) other (eg, text article, modules). Medscape, a health information website, provided 10 (48%) interventions, Continuing Nursing Education University provided 2 (10%), CDC provided 3 (14%), professional organizations (eg, American Academy of Pediatrics and Texas Medical Association) provided 3 (14%), nonprofit organizations (eg, Indiana Immunization Coalition) provided 1 (5%), a federally-authorized regional organization (The Suwannee River Area Health Education Center) provided 1 (5%), and a university (Boston University School of Medicine Continuing Medical Education and Continuing Nursing Education) provided 1 (5%). Multimedia Appendix 1 includes the characteristics of the interventions.

 Quality Evaluation

On the basis of the evaluation indicators, 13 (62%) interventions required registration, but all interventions were accessible without cost to the participant (K-alpha=1.0). Additionally, 17 (81%) educational interventions identified the purpose of the intervention (K-alpha=1.0), and 12 (57%) provided a date when the information had been updated: 7 (33%) were updated in the last 6 months (K-alpha=1.0). In assessing presentation of clear references, 18 (86%) interventions provided references (K-alpha=1.0), and 8 (38%) provided links to other sources (K-alpha=1.0). Most interventions (18/21, 85%) provided reliable references or sources (K-alpha=1.0). The references or sources included information from the CDC (n=16), published peer-reviewed literature (n=16), American Cancer Society (n=5), National Cancer Institute (n=4), Institutes of Medicine (n=4), WHO (n=2), and American Academy of Pediatrics (n=1). Of the 21 interventions, 14 (67%) had a statement of disclosures from the authors, sponsors, or developers (K-alpha=1.0).

For the design subindicators, 2 interventions were documentary or videos that did not include text, and therefore, were not included in the total sample for these subindicators. All interventions (n=19) included easy-to-read font size, font style, color, and line spacing (K-alpha=1.0 for these 3 subindicators). For the subindicator “Graphics were clearly labeled,” only 13 interventions included graphics; thus, the sample for this subindicator is 13 interventions. Out of the 13 interventions, there were 10 (77%) interventions with a minimum of 75% of graphics labeled, 2 (15%) with a minimum of 50% of graphics labeled, and 1 (8%) intervention with a minimum of 25% of graphics labeled. No intervention had 0% of graphics labeled (K-alpha=1.0).

Of the 21 interventions, 17 (81%) included an evaluation for participant outcomes: 17 (81%) assessed HPV and HPV vaccine knowledge, and none assessed attitudes toward HPV and the HPV vaccine. Furthermore, 14 (67%) interventions provided the participants the opportunity to evaluate or provide feedback (K-alpha=1.0). Five (24%) interventions included an interactive component (K-alpha=1.0). The most commonly used interactive component was a polling or knowledge check activity (n=4). No educational intervention explicitly stated a theory or model that was used to develop the intervention. Table 1 provides additional results from the evaluation, and Table 2 provides the quality summary score for each Web-based intervention.

Web-based educational intervention quality evaluation results (n=21).

Indicator and subindicator Scoring frequency
Yes No
Access
Registration required 13 8
Cost 0 21
Content
Date information was updated 12 9
Identification of purpose 17 4
Presentation of clear references 18 3
Links to other sources 8 13
Reliable references and sourcesa 18 3
Statement indicating content was developed or reviewed by experts 20 14
Disclosure of authors, sponsors, or developers 14 7
Design
Font style was easy to readb 19 0
Font size was easy to readb 19 0
Font color and page color contrast was easy to readb 19 0
Line spacing was easy to readb 19 0
Graphics were clearly labeledc 13 0
Evaluation
Evaluation for participant outcomesd 17 4
Participant provided opportunity to evaluate intervention 14 7
Interactivity
Location to direct participant questions 6 15
Included interactive componente 5 16
Theory or model(s)
Theory or model was used to develop intervention 0 21

aThe Centers for Disease Control and Prevention (n=16) and published peer reviewed literature (n=16) were the most common frequency cited sources, followed by American Cancer Society (n=5), National Institutes of Health (n=4), Institute of Medicine (n=4), World Health Organization (n=2), Food and Drug Administration (n=1), and the American Academy of Pediatrics (n=1).

bTwo interventions were a documentary or TV series that did not include any type of font or graphics for informational purposes. Therefore, for the Design subindicators font style, font size, font color, and line spacing, the sample size was n=19.

cFor the graphic subindicator, eight interventions did not include graphics for informational purpose. Therefore, the sample size was n=13. There were 10 interventions with a minimum of 75% of graphics labeled, 2 interventions with a minimum of 50% of graphics labeled, 1 intervention with a minimum of 25% of graphics labeled.

dSpecific levels of evaluation for participant outcomes include 17 interventions assessing HPV and HPV vaccine knowledge, and no intervention assessing attitudes toward HPV and the HPV vaccine.

eFive interventions provided participant interactivity. Four interventions included an interactive knowledge check, and 1 intervention included directions to email reminders and newsletters.

Quality summary scores for Web-based interventions.

Intervention titlea Summary score (out of 25)
HPV Vaccine Safety and Efficacy 20
HPV Vaccines: Updates and Clinical Perspective 20
Increasing Adolescent Immunization Coverage 20
Don’t Wait Vaccinate! The Prevention of HPV Cancers (Part 2) 19
HPV Vaccination is Cancer Prevention (2017 Update) 19
Overcoming Gender and Socioeconomic Disparities in HPV Vaccination 19
You are the Key to HPV Cancer Prevention 18b
Don’t Wait Vaccinate! The Prevention of HPV Cancers 17
Immunization: You Call the Shots-Module Eight-HPV, 2016 17
Immunization: You Call the Shots-Module Eighteen—Vaccine Administration 17c
You are the Key to HPV Cancer Prevention 17d
ACIP Releases Pediatric Vaccine Schedule 16c
Adolescent Immunizations: Strongly Recommending the HPV Vaccine 16
AAP Provides Guidance for Parents Who Refuse Vaccination 15c
ACIP Releases Adult Vaccine Recommendations 15c
CDC Updates Guideline Recommendations for HPV Vaccination 15c
Human Papillomavirus (HPV) Vaccine Safety 15c
The Story of HPV: Yesterday, Today, and Tomorrow 14
HPV Vaccines: Updates and Clinical Perspective 13
Putting HPV Vaccine Knowledge Into Practice 7e
HPV Documentary—Division of Continuing Medical Education 2e

aHPV: Human Papillomavirus; ACIP: Advisory Committee on Immunization Practices; AAP: American Academy of Pediatrics; CDC: Centers for Disease Control and Prevention.

bYou are the key to HPV Cancer Prevention intervention published 9/2/2015 and expires 9/7/2017.

cThese interventions did not include any type of graphics for informational purpose. Therefore, the total score is out of 24.

dYou are the key to HPV Cancer Prevention intervention published 4/21/2016 and expires 4/21/2018.

eThese interventions were documentaries and did not include any type of font or graphics for informational purposes. Therefore, the total score is out of 20.

 Discussion Principal Findings

This study provides a systematic, evidence-based assessment of the strengths and weaknesses of current HPV vaccine Web-based educational interventions. Strengths of the assessed Web-based educational interventions include: (1) being developed by experts in the field; (2) providing reliable references or sources; (3) providing clinicians with access to CEs for no cost; (4) following basic design principles with easy-to-read fonts, colors, and graphics; and (5) consistently providing evaluation opportunities for participant knowledge outcomes. Weaknesses of the educational interventions included lack of: (1) evaluation of outcomes including participants’ attitudes about HPV vaccination, intention to recommended vaccination, and recommendation of behaviors; (2) theory-based interventions; (3) opportunity for participants to provide feedback or evaluation of the intervention; (4) links to other sources or resources; and (5) interactivity throughout the intervention.

HPV vaccination rates are well below the Healthy People 2020 objective [8], and clinicians report that they would benefit from CE regarding the HPV vaccine [29]. Because clinicians’ HPV vaccine recommendation is one of the most important predictors of HPV vaccination uptake [59-62], ensuring that clinicians are equipped with current and accurate information is critical [63]. Clinicians, however, are continually challenged in providing parents and patients with evidence-based HPV vaccine information because of changing vaccine guidelines and the volume of information and sources available [64]. Thus, clinicians’ report obtaining a large portion of HPV vaccine information from professional organizations [65] possibly because of lack of time needed to identify multiple sources of accurate information [66]. In this study, only 3 interventions were provided by 2 professional organizations, including the American Academy of Pediatrics and the Texas Medical Association. Medscape, a health information website, provided almost half of the interventions. Given that professional organizations are cited by clinicians as an important and trusted source of HPV vaccine information, professional organizations need to increase efforts to collaborate with health information websites and other organizations and institutions to provide evidence- and theory-based interventions. A recent study demonstrated that organizations working on cancer research identified the ability to leverage resources, lower costs, increase organization reputation, and the development of new tools and methodology as benefits to interorganizational collaboration [67]. Therefore, interorganizational collaboration to provide clinicians with HPV vaccine Web-based interventions has the potential to improve outcomes related to HPV vaccination rates and cancer risk reduction.

We found that none of the interventions examined included a statement that a theory was used to create the intervention. To improve outcome behaviors and increase clinician HPV vaccine recommendation behaviors, intervention developers should utilize science and evidence that supports effective medical education and behavior change [54]. Theories can be used for quality assessment and improvement by identifying factors contributing to behavior change and which factors are ineffective. Overall, interventions based in theory provide an advantage in changing behavior by providing a logical and systematic approach to increasing clinicians’ recommendation of the HPV vaccine [54].

None of the Web-based educational interventions included in this study evaluated HPV attitudes, intention to recommend vaccination, or actual recommendation behavior. This is concerning given that clinician attitudes are an important predictor of vaccine recommendations. Clinicians have reported concerns regarding HPV vaccine safety [68-70], a lack of self-confidence in providing strong vaccine recommendations [15,27,28], and belief that it is not important for adolescents to receive the HPV vaccine at the recommended age of 11 to 12 years [71]. Therefore, interventions should be designed with the goal of changing clinician attitudes and vaccine recommendations, and evaluation of these outcomes is a key component of successful interventions [45,52]. Evaluation of outcomes is also important for the translation of health communication research into efforts to promote clinician recommendations of the HPV vaccine [44,46]. Finally, evaluation is essential for understanding clinicians’ educational needs and assessing program outcomes addressing important health issues [50].

Although face-to-face educational interventions have shown to improve clinicians’ willingness to provide immunizations and routinely screen immunization records at visits [72], evaluations specifically assessing HPV-related Web-based educational interventions are limited [73]. Only 2 published studies provided evaluation results on webinars designed to increase adolescent vaccination rates. Results suggest webinars have the potential to increase clinician recommendation behaviors and adolescent Tdap, meningococcal, and HPV vaccination rates similar to in-person educational interventions [74,75]. Web-based educational interventions create a unique platform to provide clinicians with the knowledge and skills needed to promote the HPV vaccine among adolescents. One important component of Web-based educational interventions is interactivity [54,55]. Interactive components encourage users to be actively involved in the intervention and have been linked to short-term behavioral improvements [76]. Furthermore, Kreps and Neuhauser pinpoint interactivity as a communication attribute with the ability to exponentially improve health promotion [76]. Even though interactivity can have a significant impact on participants, only 5 interventions from this study included an interactive component. Because Web-based educational interventions continue to gain popularity because of convenience and economic benefits [31], the lack of interactivity in the majority of HPV-related Web-based educational interventions is alarming given the research supporting the importance of interactive components. More research should be conducted to determine the impact of interactive components in HPV-related Web-based educational interventions on clinicians’ HPV vaccination recommendation behaviors.

 Limitations

While this study provides innovative insight into the quality of Web-based HPV vaccine educational interventions created for clinicians, there are limitations that should be considered. First, only Web-based educational interventions were evaluated, and these results cannot be generalized to other types of interventions such as face-to-face lectures, grand rounds, or seminars. There would be substantial benefit to conducting evaluations of face-to-face lectures and seminar materials to assess all venues of continuing education for clinicians regarding the HPV vaccine. Second, this quality evaluation did not assess participants’ experience of the intervention and therefore, cannot identify every area for improvement. Data were not collected from participants themselves regarding usability: this study identified only 7 indicators of usability. Third, this study was a quality evaluation and did not evaluate participant outcomes (eg, knowledge, attitudes, recommendation behaviors) after completing the intervention. Fourth, the evaluation was conducted using only the materials that were accessible at the time of the study, and there is the potential that a component (such as, a follow-up emailed evaluation after the intervention to participants) was not included in this evaluation. Despite these limitations, these findings provide valuable information for those who develop Web-based continuing education interventions regarding HPV vaccines, by providing a quantitative approach to identifying the design and usability strengths and weaknesses of HPV vaccine Web-based educational interventions.

 Future Work

The data resulting from this study have the potential to help shift current research practice paradigms. The findings suggest that those who develop Web-based educational interventions to promote HPV vaccine recommendations utilize design science principles, a powerful approach and process that includes participatory action research to iteratively develop and evaluate health education interventions [77]. Additional qualitative, multi-approach evaluation research is needed to further assess the content (eg, the specific messages provided to clinicians about the HPV vaccine and recommendation behaviors) and usability of these interventions from the participants’ perspective. Further evaluation research is needed to ensure that interventions are being developed using all design principles and are effective at increasing strong and consistent HPV vaccine recommendations from clinicians.

 Conclusions

The quality evaluation of these interventions demonstrated that Web-based interventions were based on reliable sources, developed by experts, and were created with critical design aspects (eg, font style, size, and color were easy to read, graphics were clearly labeled). However, there were limited outcome evaluations for users measuring attitudes, intentions, or behaviors, and lack of user interactivity. Results from this study suggest best practices for designing, refining, and implementing Web-based interventions to promote HPV vaccination within the clinician population.

 Multimedia Appendix 1

Intervention characteristics.

 Abbreviations AAP

American Academy of Pediatrics

 ACIP

Advisory Committee on Immunization Practices

 CDC

Centers for Disease Control and Prevention

 CE

continuing education

 HPV

human papillomavirus

No funding was received for this study.

BLR is the primary author, JMB conducted intervention evaluation and assisted in writing the manuscript, SLM conducted intervention evaluation and assisted in writing the manuscript, JK provided expert guidance for the evaluation and revised the manuscript, and GLK provided overall guidance for the evaluation and provided extensive revisions of the manuscript.

None declared.

 CDC 2017 03 06 HPV-Associated Cancer Statistics https://www.cdc.gov/cancer/hpv/statistics/index.htm Brown LM Check DP Devesa SS Oropharyngeal cancer incidence trends: diminishing racial disparities Cancer Causes Control 2011 05 22 5 753 63 10.1007/s10552-011-9748-1 21380619 Hocking JS Stein A Conway EL Regan D Grulich A Law M Brotherton JM Head and neck cancer in Australia between 1982 and 2005 show increasing incidence of potentially HPV-associated oropharyngeal cancers Br J Cancer 2011 03 1 104 5 886 91 10.1038/sj.bjc.6606091 21285981 6606091 PMC3048203 Reddy VM Cundall-Curry D Bridger MW Trends in the incidence rates of tonsil and base of tongue cancer in England, 1985-2006 Ann R Coll Surg Engl 2010 11 92 8 655 9 10.1308/003588410X12699663904871 20615309 577 PMC3229372 Petrosky E Bocchini Jr JA Hariri S Chesson H Curtis CR Saraiya M Unger ER Markowitz LE Centers for Disease Control and Prevention (CDC) Use of 9-valent human papillomavirus (HPV) vaccine: updated HPV vaccination recommendations of the advisory committee on immunization practices MMWR Morb Mortal Wkly Rep 2015 03 27 64 11 300 4 25811679 mm6411a3 Saraiya M Population-based HPV genotype attribution in HPV-associated cancers 2017 Proceedings of the Anal Intraepithelial Neoplasia Society Conference March 13-15, 2015 Atlanta, Georgia Reagan-Steiner S Yankey D Jeyarajah J Elam-Evans LD Curtis CR MacNeil J Markowitz LE Singleton JA National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years - United States, 2015 MMWR Morb Mortal Wkly Rep 2016 08 26 65 33 850 8 10.15585/mmwr.mm6533a4 27561081 Healthy People 2017-09-21 Immunization and infectious diseases https://www.healthypeople.gov/2020/topics-objectives/topic/immunization-and-infectious-diseases Centers for Disease Control and Prevention (CDC) Human papillomavirus vaccination coverage among adolescent girls, 2007-2012, and postlicensure vaccine safety monitoring, 2006-2013 - United States MMWR Morb Mortal Wkly Rep 2013 07 26 62 29 591 5 23884346 mm6229a4 Quinn GP Murphy D Malo TL Christie J Vadaparampil ST A national survey about human papillomavirus vaccination: what we didn't ask, but physicians wanted us to know J Pediatr Adolesc Gynecol 2012 08 25 4 254 8 10.1016/j.jpag.2012.02.007 22516792 S1083-3188(12)00048-4 PMC3408795 Vadaparampil ST Murphy D Rodriguez M Malo TL Quinn GP Qualitative responses to a national physician survey on HPV vaccination Vaccine 2013 04 26 31 18 2267 72 10.1016/j.vaccine.2013.02.063 23499608 S0264-410X(13)00252-1 PMC3828557 Daley MF Crane LA Markowitz LE Black SR Beaty BL Barrow J Babbel C Gottlieb SL Liddon N Stokley S Dickinson LM Kempe A Human papillomavirus vaccination practices: a survey of US physicians 18 months after licensure Pediatrics 2010 09 126 3 425 33 10.1542/peds.2009-3500 20679306 peds.2009-3500 Perkins RB Clark JA What affects human papillomavirus vaccination rates? A qualitative analysis of providers' perceptions Womens Health Issues 2012 22 4 e379 86 10.1016/j.whi.2012.04.001 22609253 S1049-3867(12)00023-0 McCave EL Influential factors in HPV vaccination uptake among providers in four states J Community Health 2010 12 35 6 645 52 10.1007/s10900-010-9255-4 20336354 Perkins RB Clark JA Providers' attitudes toward human papillomavirus vaccination in young men: challenges for implementation of 2011 recommendations Am J Mens Health 2012 07 6 4 320 3 10.1177/1557988312438911 22398992 1557988312438911 Yarnall KS Pollak KI Østbye T Krause KM Michener JL Primary care: is there enough time for prevention? Am J Public Health 2003 04 93 4 635 41 12660210 PMC1447803 Solberg LI Nordin JD Bryant TL Kristensen AH Maloney SK Clinical preventive services for adolescents Am J Prev Med 2009 11 37 5 445 54 10.1016/j.amepre.2009.06.017 19840701 S0749-3797(09)00490-5 Yarnall KS Østbye T Krause KM Pollak KI Gradison M Michener JL Family physicians as team leaders: “time” to share the care Prev Chronic Dis 2009 04 6 2 A59 19289002 A59 PMC2687865 Holman DM Benard V Roland KB Watson M Liddon N Stokley S Barriers to human papillomavirus vaccination among US adolescents: a systematic review of the literature JAMA Pediatr 2014 01 168 1 76 82 10.1001/jamapediatrics.2013.2752 24276343 1779687 PMC4538997 Kimmel SR Adolescent vaccination: physician challenges and solutions J Adolesc Health 2010 02 46 2 S16 S23 10.1016/j.jadohealth.2009.11.219 Constantine NA Jerman P Acceptance of human papillomavirus vaccination among Californian parents of daughters: a representative statewide analysis J Adolesc Health 2007 02 40 2 108 15 10.1016/j.jadohealth.2006.10.007 17259050 S1054-139X(06)00413-7 Guerry SL De Rosa CJ Markowitz LE Walker S Liddon N Kerndt PR Gottlieb SL Human papillomavirus vaccine initiation among adolescent girls in high-risk communities Vaccine 2011 03 9 29 12 2235 41 10.1016/j.vaccine.2011.01.052 21288799 S0264-410X(11)00091-0 Perkins RB Pierre-Joseph N Marquez C Iloka S Clark JA Why do low-income minority parents choose human papillomavirus vaccination for their daughters? J Pediatr 2010 10 157 4 617 22 10.1016/j.jpeds.2010.04.013 20472250 S0022-3476(10)00313-6 PMC3459226 Litton AG Desmond RA Gilliland J Huh WK Franklin FA Factors associated with intention to vaccinate a daughter against HPV: a statewide survey in Alabama J Pediatr Adolesc Gynecol 2011 06 24 3 166 71 10.1016/j.jpag.2011.01.004 21397534 S1083-3188(11)00005-2 PMC3100399 Dorell C Yankey D Strasser S Parent-reported reasons for nonreceipt of recommended adolescent vaccinations, national immunization survey: teen, 2009 Clin Pediatr (Phila) 2011 12 50 12 1116 24 10.1177/0009922811415104 21856964 0009922811415104 Rosenthal SL Weiss TW Zimet GD Ma L Good MB Vichnin MD Predictors of HPV vaccine uptake among women aged 19-26: importance of a physician's recommendation Vaccine 2011 01 29 29 5 890 5 10.1016/j.vaccine.2009.12.063 20056186 S0264-410X(09)01980-X Saraiya M Rosser JI Cooper CP Cancers that U.S. physicians believe the HPV vaccine prevents: findings from a physician survey, 2009 J Womens Health (Larchmt) 2012 02 21 2 111 7 10.1089/jwh.2011.3313 22216920 PMC5555392 Gowda C Schaffer S Dombkowski K Dempsey A Understanding attitudes toward adolescent vaccination and the decision-making dynamic among adolescents, parents and providers BMC Public Health 2012 12 509 22768870 Bruno DM Wilson TE Gany F Aragones A Identifying human papillomavirus vaccination practices among primary care providers of minority, low-income and immigrant patient populations Vaccine 2014 32 33 4149 54 24886959 Cook DA Levinson AJ Garside S Dupras DM Erwin PJ Montori VM Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis Acad Med 2010 05 85 5 909 22 10.1097/ACM.0b013e3181d6c319 20520049 00001888-201005000-00042 Olivieri JJ Knoll MB Arn PH Education format and resource preferences among registrants of a pediatric-focused CME website Med Teach 2009 31 8 e333 7 19811195 Shaw T Barnet S Mcgregor D Avery J Using the knowledge, process, practice (KPP) model for driving the design and development of online postgraduate medical education Med Teach 2015 37 1 53 8 24934171 Goldberg H McKhann G Students test scores are improved in a virtual learning environment Adv Physiol Educ 2000 23 1 59 66 10902528 Harris JM Novalis-Marine C Harris RB Women physicians are early adopters of on-line continuing medical education J Contin Educ Health Prof 2003 23 4 221 8 14730792 Neuhauser L Paul K Fischhoff B Brewer NT Downs JS Readability, comprehension and usability Communicating Risks and Benefits: An Evidence Based User's Guide 2011 Silver Spring, MD US Department of Health and Human Services 129 48 Kreps GL Strategic use of communication to market cancer prevention and control to vulnerable populations Health Mark Q 2008 25 1-2 204 16 10.1080/07359680802126327 18935885 Fordis M King JE Ballantyne CM Jones PH Schneider KH Spann SJ Greenberg SB Greisinger AJ Comparison of the instructional efficacy of Internet-based CME with live interactive CME workshops: a randomized controlled trial J Am Med Assoc 2005 09 7 294 9 1043 51 10.1001/jama.294.9.1043 16145024 294/9/1043 Allison JJ Kiefe CI Wall T Casebeer L Ray MN Spettell CM Hook 3rd EW Oh MK Person SD Weissman NW Multicomponent Internet continuing medical education to promote chlamydia screening Am J Prev Med 2005 04 28 3 285 90 10.1016/j.amepre.2004.12.013 15766617 S0749-3797(04)00366-6 Casebeer LL Strasser SM Spettell CM Wall TC Weissman N Ray MN Allison JJ Designing tailored Web-based instruction to improve practicing physicians' preventive practices J Med Internet Res 2003 5 3 e20 10.2196/jmir.5.3.e20 14517111 PMC1550567 Carney PA Dietrich AJ Freeman Jr DH Mott LA A standardized-patient assessment of a continuing medical education program to improve physicians' cancer-control clinical skills Acad Med 1995 01 70 1 52 8 7826445 Harris Jr JM Kutob RM Surprenant ZJ Maiuro RD Delate TA Can Internet-based education improve physician confidence in dealing with domestic violence? Fam Med 2002 04 34 4 287 92 12017143 Marinopoulos SS Dorman T Ratanawongsa N Wilson LM Ashar BH Magaziner JL Miller RG Thomas PA Prokopowicz GP Qayyum R Bass EB AHRQ 2007 2018-02-07 Effectiveness of Continuing Medical Education https://archive.ahrq.gov/downloads/pub/evidence/pdf/cme/cme.pdf Rosen BL Kreps G Shepard A Content and usability evaluation of HPV vaccine online educational interventions for clinicians 2017 Proceeding of the American Academy of Health Behavior 2017 Annual Scientific Meeting March 19-22, 2017 Tucson, Arizona Kreps GL Translating health communication research into practice: the importance of implementing and sustaining evidence-based health communication interventions Atlantic J Commun 2012 01 25 20 1 5 15 10.1080/15456870.2012.637024 Kreps GL Evaluating new health information technologies: expanding the frontiers of health care delivery and health promotion Stud Health Technol Inform 2002 80 205 12 12026130 Kreps GL Thompson TL Parrott R Nussbaum JF Translating health communication research into practice: the influence of health communication scholarship on health policy, pratice, and outcomes The handbook of health communication. 2nd edition 2011 New York, NY Routledge 595 608 Kreps GL Epilogue: lessons learned about evaluating health communication programs J Health Commun 2014 12 19 12 1510 4 10.1080/10810730.2014.954085 25491585 Cho H Salmon CT Unintended effects of health communication campaigns J Commun 2007 06 57 2 293 317 10.1111/j.1460-2466.2007.00344.x Ringold DJ Boomerang effects in response to public health interventions: some unintended consequences in the alcoholic beverage market J Consumer Policy 2002 25 1 27 63 10.1023/A:1014588126336 Neuhauser L Kreps GL Integrating design science theory and methods to improve the development and evaluation of health communication programs J Health Commun 2014 19 12 1460 71 10.1080/10810730.2014.954081 25491581 Centers for Medicare and Medicaid Services National health care expenditures data. Report 2012 Baltimore, MD CMS. Office of the Actuary, National Health Statistics Group Kreps GL Evaluating health communication programs to enhance health care and health promotion J Health Commun 2014 19 12 1449 59 10.1080/10810730.2014.954080 25491580 Kim P Eng TR Deering MJ Maxfield A Review of published criteria for evaluating health-related websites West J Med 1999 170 6 329 32 18751150 PMC1305683 Evers KE Prochaska JM Prochaska JO Driskell M Cummins CO Velicer WF Strengths and weaknesses of health behavior change programs on the internet J Health Psychol 2003 01 8 1 63 70 10.1177/1359105303008001435 22113901 8/1/63 Cummins CO Prochaska JO Driskell MM Evers KE Wright JA Prochaska JM Velicer WF Development of review criteria to evaluate health behavior change websites J Health Psychol 2003 01 8 1 55 62 10.1177/1359105303008001434 22113900 8/1/55 Yen P Bakken S Review of health information technology usability study methodologies J Am Med Inform Assoc 2012 06 19 3 413 22 10.1136/amiajnl-2010-000020 21828224 amiajnl-2010-000020 PMC3341772 De Swert K 2017-09-21 Calculating inter-coder reliability in media content analysis using Krippendorff’s Alpha http://www.polcomm.org/wp-content/uploads/ICR01022012.pdf Hayes AF Krippendorff K Answering the call for a standard reliability measure for coding data Commun Methods Meas 2007 04 1 1 77 89 10.1080/19312450709336664 Caskey R Lindau ST Alexander GC Knowledge and early adoption of the HPV vaccine among girls and young women: results of a national survey J Adolesc Health 2009 11 45 5 453 62 10.1016/j.jadohealth.2009.04.021 19837351 S1054-139X(09)00184-0 Conroy K Rosenthal SL Zimet GD Jin Y Bernstein DI Glynn S Kahn JA Human papillomavirus vaccine uptake, predictors of vaccination, and self-reported barriers to vaccination J Womens Health (Larchmt) 2009 10 18 10 1679 86 10.1089/jwh.2008.1329 19785564 Gamble HL Klosky JL Parra GR Randolph ME Factors influencing familial decision-making regarding human papillomavirus vaccination J Pediatr Psychol 2010 08 35 7 704 15 10.1093/jpepsy/jsp108 19966315 jsp108 PMC2915623 Do H Seng P Talbot J Acorda E Coronado GD Taylor VM HPV vaccine knowledge and beliefs among Cambodian American parents and community leaders Asian Pac J Cancer Prev 2009 10 3 339 44 19640169 PMC2858686 Pickering LK Baker CJ Freed GL Gall SA Grogg SE Poland GA Rodewald LE Schaffner W Stinchfield P Tan L Zimmerman RK Orenstein WA Infectious Diseases Society of America Immunization programs for infants, children, adolescents, and adults: clinical practice guidelines by the Infectious Diseases Society of America Clin Infect Dis 2009 09 15 49 6 817 40 10.1086/605430 19659433 Hillenbrand KM What is going on with vaccines: keeping up with the childhood immunization schedule J Public Health Manag Pract 2007 13 6 544 52 10.1097/01.PHH.0000296128.54767.d7 17984706 00124784-200711000-00003 Bynum SA Malo TL Lee J Guiliano AR Vadaparampil ST HPV vaccine information-seeking behaviors among US physicians: government, media, or colleagues? Vaccine 2011 07 18 29 32 5090 3 10.1016/j.vaccine.2011.04.134 21619906 S0264-410X(11)00741-9 PMC3138888 Verhoeven F Steehouder MF Hendrix RM van Germert-Pijnen JE Factors affecting health care workers' adoption of a website with infection control guidelines Int J Med Inform 2009 10 78 10 663 78 10.1016/j.ijmedinf.2009.06.001 19577956 S1386-5056(09)00089-6 Harris JK Provan KG Johnson KJ Leischow SJ Drawbacks and benefits associated with inter-organizational collaboration along the discovery-development-delivery continuum: a cancer research network case study Implement Sci 2012 07 25 7 69 10.1186/1748-5908-7-69 22831463 1748-5908-7-69 PMC3443066 Rosen BL Ashwood D Richardson GB School nurses' professional practice in the HPV vaccine decision making process J Sch Nurs 2016 04 32 2 138 48 10.1177/1059840515583312 25962388 1059840515583312 Alexander AB Best C Stupiansky N Zimet GD A model of health care provider decision making about HPV vaccination in adolescent males Vaccine 2015 08 7 33 33 4081 6 10.1016/j.vaccine.2015.06.085 26143612 S0264-410X(15)00900-7 Young JL Bernheim RG Korte JE Stoler MH Guterbock TM Rice LW Human papillomavirus vaccination recommendation may be linked to reimbursement: a survey of Virginia family practitioners and gynecologists J Pediatr Adolesc Gynecol 2011 12 24 6 380 5 10.1016/j.jpag.2011.06.016 21906978 S1083-3188(11)00293-2 Henrikson NB Tuzzio L Gilkey MB McRee AL “You're never really off time”: Healthcare providers' interpretations of optimal timing for HPV vaccination Prev Med Rep 2016 4 94 7 10.1016/j.pmedr.2016.05.002 27413667 S2211-3355(16)30032-8 PMC4929064 Boom JA Nelson CS Laufman LE Kohrt AE Kozinetz CA Improvement in provider immunization knowledge and behaviors following a peer education intervention Clin Pediatr (Phila) 2007 10 46 8 706 17 10.1177/0009922807301484 17522285 0009922807301484 Berenson AB Rahman M Hirth JM Rupp RE Sarpong KO A brief educational intervention increases providers' human papillomavirus vaccine knowledge Hum Vaccin Immunother 2015 11 6 1331 6 10.1080/21645515.2015.1022691 25945895 PMC4514429 Moss JL Reiter PL Dayton A Brewer NT Increasing adolescent immunization by webinar: a brief provider intervention at federally qualified health centers Vaccine 2012 07 13 30 33 4960 3 10.1016/j.vaccine.2012.05.042 22652406 S0264-410X(12)00760-8 Gilkey MB Dayton AM Moss JL Sparks AC Grimshaw AH Bowling JM Brewer NT Increasing provision of adolescent vaccines in primary care: a randomized controlled trial Pediatrics 2014 08 134 2 e346 53 10.1542/peds.2013-4257 25002671 peds.2013-4257 PMC4531274 Kreps GL Neuhauser L New directions in eHealth communication: opportunities and challenges Patient Educ Couns 2010 03 78 3 329 36 10.1016/j.pec.2010.01.013 20202779 S0738-3991(10)00022-4 Neuhauser L Kreps GL Morrison K Athanasoulis M Kirienko N Van Brunt D Using design science and artificial intelligence to improve health communication: chronologyMD case example Patient Educ Couns 2013 08 92 2 211 7 10.1016/j.pec.2013.04.006 23726219 S0738-3991(13)00164-X