This paper is in the following e-collection/theme issue:

Doctor-Patient Communication on Cancer, Prevention, and Screening (41) e-Learning and Medical Education (645) Quality/Credibility of eHealth Information and Trust Issues (485) Information Quality in Digital Media (316) Continuing Medical Education (CME) for Doctors (120) HPV Vaccination (51)

Published on in Vol 4, No 1 (2018): Jan-Jun

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/9114, first published .
Quality of Web-Based Educational Interventions for Clinicians on Human Papillomavirus Vaccine: Content and Usability Assessment

Quality of Web-Based Educational Interventions for Clinicians on Human Papillomavirus Vaccine: Content and Usability Assessment

Quality of Web-Based Educational Interventions for Clinicians on Human Papillomavirus Vaccine: Content and Usability Assessment

Authors of this article:

Brittany L Rosen1 Author Orcid Image ;   James M Bishop1 Author Orcid Image ;   Skye L McDonald1 Author Orcid Image ;   Jessica A Kahn2, 3 Author Orcid Image ;   Gary L Kreps4 Author Orcid Image
Article Authors Cited by (8) Tweetations (2) Metrics

    Original Paper

    1School of Human Services, University of Cincinnati, Cincinnati, OH, United States

    2Department of Pediatrics, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States

    3College of Medicine, University of Cincinnati, Cincinnati, OH, United States

    4Center for Health and Risk Communication, Department of Communication, George Mason University, Fairfax, VA, United States

    Corresponding Author:

    Brittany L Rosen, CHES, PhD

    School of Human Services

    University of Cincinnati

    PO Box 210068

    Cincinnati, OH, 45221

    United States

    Phone: 1 513 556 3872

    Email: brittany.rosen@uc.edu


    Background: Human papillomavirus (HPV) vaccination rates fall far short of Healthy People 2020 objectives. A leading reason is that clinicians do not recommend the vaccine consistently and strongly to girls and boys in the age group recommended for vaccination. Although Web-based HPV vaccine educational interventions for clinicians have been created to promote vaccination recommendations, rigorous evaluations of these interventions have not been conducted. Such evaluations are important to maximize the efficacy of educational interventions in promoting clinician recommendations for HPV vaccination.

    Objective: The objectives of our study were (1) to expand previous research by systematically identifying HPV vaccine Web-based educational interventions developed for clinicians and (2) to evaluate the quality of these Web-based educational interventions as defined by access, content, design, user evaluation, interactivity, and use of theory or models to create the interventions.

    Methods: Current HPV vaccine Web-based educational interventions were identified from general search engines (ie, Google), continuing medical education search engines, health department websites, and professional organization websites. Web-based educational interventions were included if they were created for clinicians (defined as individuals qualified to deliver health care services, such as physicians, clinical nurses, and school nurses, to patients aged 9 to 26 years), delivered information about the HPV vaccine and how to increase vaccination rates, and provided continuing education credits. The interventions’ content and usability were analyzed using 6 key indicators: access, content, design, evaluation, interactivity, and use of theory or models.

    Results: A total of 21 interventions were identified, out of which 7 (33%) were webinars, 7 (33%) were videos or lectures, and 7 (33%) were other (eg, text articles, website modules). Of the 21 interventions, 17 (81%) identified the purpose of the intervention, 12 (57%) provided the date that the information had been updated (7 of these were updated within the last 6 months), 14 (67%) provided the participants with the opportunity to provide feedback on the intervention, and 5 (24%) provided an interactive component. None of the educational interventions explicitly stated that a theory or model was used to develop the intervention.

    Conclusions: This analysis demonstrates that a substantial proportion of Web-based HPV vaccine educational interventions has not been developed using established health education and design principles. Interventions designed using these principles may increase strong and consistent HPV vaccination recommendations by clinicians.

    JMIR Cancer 2018;4(1):e3

    doi:10.2196/cancer.9114

    Keywords

    papillomavirus vaccines; internet; program evaluation; health personnel 


    The human papillomavirus (HPV) causes almost all cervical cancers, 50% of vulvar cancers, 65% of vaginal cancers, 90% of anal cancers, and 35% of penile cancers [1]. Recent studies have shown that the incidence of oral cancers caused by HPV is increasing [2-4]. The 9-valent HPV vaccine (9vHPV) is nearly 100% effective in preventing precancerous lesions caused by 7 genotypes [5], accounting for 81% of cervical cancer cases [6]. Despite ample evidence that licensed HPV vaccines are safe to use and effective in preventing certain anogenital cancers, only 42% of adolescent females and 28% of adolescent males have completed the HPV vaccine series [7]. These HPV vaccination rates fall short of Healthy People 2020’s objective of 80% coverage for girls and boys aged between 13 and 15 years [8].

    Missed opportunities for clinicians to recommend and administer the vaccine, as well as a lack of strong and consistent recommendations by clinicians who do recommend the vaccine, are primary reasons for low HPV vaccination rates in the United States [9]. Factors contributing to missed clinical recommendation opportunities and insufficiently strong and consistent recommendations include providers’ limited knowledge of HPV and the vaccine, discomfort discussing a topic related to sexual behavior, concerns about vaccine safety, parental resistance, preference for vaccinating older adolescents, and lack of time or incentive to educate parents about the vaccine [10-19]. While improving communication between health care providers and parents is considered a critical component to increasing vaccination rates [20-26], health care providers report they do not feel well-prepared to provide strong vaccination recommendations [15,27,28]. Additionally, 75% of clinicians expressed they would benefit from continuing education about recommending the HPV vaccine [29].

    To improve clinicians’ HPV vaccine recommendations, numerous Web-based HPV vaccine educational interventions for clinicians have been created. Web-based educational interventions have become a popular delivery method for health care professionals to obtain continuing education (CE) [30], as Web-based interventions provide an opportunity to quickly update and address health education topics at low cost [31]. With rapid proliferation of Web-based health education programs, there is a potential risk of neglecting fundamental health communication and education design principles in these programs that are important to ensure their efficacy [32-34].

    Web-based CE interventions created using strategic health communication design principles—communicating effectively with intended users and taking into account audience factors such as culture, race, ethnicity, language, access, functional needs (ie, disabilities), and expectations [35,36]—are likely to increase clinicians’ knowledge, perceptions, attitudes, and practice behaviors [37-42]. Numerous Web-based HPV vaccine educational interventions have been introduced to accommodate clinicians’ educational needs [43]. However, an extensive and comprehensive review (of CINAHL, MEDLINE, ERIC, and Ebsco Academic Complete databases, using variations and Boolean connectors with the following terms: “online intervention,” “online program,” “HPV vaccine,” “clinicians,” “physicians,” “nurses and evaluation”) demonstrated that there has been no rigorous evaluation of the design, content, and usability levels of these programs. Without such evaluation data, it is unknown whether such interventions are achieving their intended outcomes, and which aspects of the interventions should be refined [44,45].

    Evaluation of the leading Web-based HPV vaccine educational interventions is important in guiding efforts to promote clinician recommendations for the HPV vaccine [44,46]. Evaluation of Web-based interventions, using established health communication and education design principles can (1) identify strengths and weaknesses of educational interventions from the users’ perspective; (2) determine whether interventions are worth the time, resources, and expense for continued implementation; and (3) provide the evidence for designing optimally effective interventions [47]. Additionally, evaluation data can provide insights into any negative unintended consequences of the interventions, such as boomerang and iatrogenic effects [48,49] and ensures that interventions address audiences’ unique needs, culture, and expectations [35]. Evaluation research is vital not only to improve health outcomes but also to ensure that interventions are cost-effective [50]. Estimated health care cost in the United States was $2.7 trillion (18% of the gross domestic product) in 2011 [51], leading policy makers to prioritize identifying effective, evidence-based methods to prevent disease and manage rising health costs [50].

    Rigorous evaluation is a central component of developing successful health education interventions [45,52] and essential for understanding clinicians’ educational needs and assessing outcomes [50]. However, current Web-based HPV vaccine interventions developed for clinicians have not been evaluated using health education and design principles. Therefore, the objectives of the study were to: (1) systematically identify HPV vaccine Web-based educational interventions developed for clinicians, and (2) evaluate the quality of Web-based educational interventions as defined by access, content, design, evaluation, interactivity, and use of theory or models to create the interventions.


    Sample

    We systematically identified current HPV vaccine Web-based educational interventions by examining general search engines (eg, Google), continuing medical education search engines (eg, PedsUniversity, MedScape), health department websites, and professional organization websites. The following search terms and variations of search terms were used within each of the search engines and websites: “clinicians,” “healthcare providers,” “HPV webinars,” “HPV vaccination webinars,” “HPV online education,” and “HPV continuing medical education.” Web-based educational interventions were included if they were (1) created for clinicians (defined as individuals qualified to deliver health care services, such as physicians, clinical nurses, and school nurses, to patients aged between 9 and 26 years); (2) delivered information about the HPV vaccine and how to increase vaccination rates; and (3) provided continuing education credits. Interventions were excluded if they were created for patients, parents, or adolescents, or if they focused on vaccines other than the HPV vaccine. We conducted the search from April 2016 to August 2017 and identified a total of 178 interventions. Of these, 21 interventions met all 3 research criteria for this study.

    Instrument

    The study team developed a quality evaluation framework based on criteria established in the literature for evaluating health-related websites [53-56]. The quality evaluation framework assessed Web-based interventions using 6 key indicators: access, content, design, evaluation, interactivity, and theory or models [53-56]. Each key indicator was scored using various subindicators: higher scores for the indicators designated higher quality interventions.

    Internal reliability of the subindicators was calculated using the Krippendorff’s alpha coefficient (K-alpha; for additional information please see De Swert, 2012) [57]. This coefficient was selected because it provides information on the reliability of variables, not coders, and its robust calculations are not impacted by sample size, multiple coders, or missing data [58]. After calculating Krippendorff’s alpha coefficient for 2 rounds of samples with 2 independent coders (Dr Rosen, a PhD trainer faculty member with expertise in HPV vaccination uptake, and a doctoral graduate research assistant in health education), the evaluation tool was considered to be internally reliable given that all indicator scores were above .80, which is considered the norm for acceptable reliability [57].

    Access

    To examine access of the educational interventions [53,54], 2 subindicators were used to measure different components of access. These subindicators included whether registration was required to access the intervention (score ranging from 0-1) and the cost of the intervention (score ranging from 0-1).

    Content

    Content was evaluated using 7 subindicators [53,55]: identification of purpose (score ranging from 0-1), date on which the information was updated (score ranging from 0-1), presentation of clear references (score ranging from 0-1), and links to other sources (score ranging from 0-1). Additionally, reliable sources (score ranging from 0-1) were assessed and whether the intervention included reliable sources, the type of source (eg, Centers for Disease Control and Prevention, National Institutes of Health, published peer-reviewed literature) was included in the scoring metric. The final 2 subindicators included a statement indicating that content was developed or reviewed by experts (score ranging from 0-1) and a statement of disclosure of authors, sponsors, or developers (score ranging from 0-1).

    Design

    The design components of the interventions were evaluated by layout and graphics [53]. The layout of the intervention was assessed by examining font and line spacing. Specifically, font was assessed by whether the style was easy to read (score ranging from 0-1), size was easy to read (score ranging from 0-1), text color and page color contrast were easy to read (score ranging from 0-1), and line spacing was easy to read (score ranging from 0-1). Graphics were assessed to determine if they were clearly labeled, and scores ranged from 0 to 3 with 0 indicating 0% of graphics were labeled, 1 indicating a minimum of 25% of the graphics were labeled, 2 indicating a minimum of 50% of the graphics were labeled, and 3 indicating a minimum of 75% of the graphics were labeled.

    Evaluation

    Evaluation was assessed using 3 subindicators [53-55]: whether participant outcomes were evaluated (eg, knowledge and attitudes; score ranging from 0-1), the level of that evaluation (score ranging from 0-2; 0 indicating no evaluation, 1 indicating an evaluation of HPV or HPV vaccine knowledge, and 2 indicating an evaluation of HPV or HPV vaccine attitudes) and whether the participant was provided an opportunity to evaluate the intervention (score ranging from 0-1).

    Interactivity

    The indicator for interactivity included 2 subindicators [54,55]. The first subindicator assessed whether there was a location for participants to direct questions during the educational intervention (score ranging from 0-1). The second subindicator assessed whether the intervention included any interactive components (score ranging from 0-1). If the intervention included any interactive component, the interactive component was recorded in the scoring metric. The interactive components included discussion boards, “ask the expert” bulletin boards, sign up for email reminders, sign up for newsletters, and other interactive components.

    Theory and Models

    The theory and models indicator was assessed by examining whether there was an explicit statement that a theory or model was used to develop the intervention (score ranging from 0-1) [54-56]. If a theory or model was used to develop the intervention, the theory or model was recorded in the scoring metric.

    Procedure

    Once interrater reliability was established for the evaluation tool with all indicator scores above .80, 2 independent coders (Mr Bishop and Ms McDonald) utilized the tool to evaluate the educational interventions identified. One of the coders, Mr Bishop is a health education doctoral student with expertise in sexuality education and evaluated the first 11 interventions. The other coder, Ms McDonald is a health education doctoral student with expertise in school health and evaluated the remaining 10 interventions. Frequency distributions were calculated for each of the subindicators to determine specific strength and weaknesses of the interventions. Because this study assessed access, content, and design aspects of interventions and did not include human subjects; this study is considered nonhuman subjects research and hence institutional review board approval was not required.


    Intervention Characteristics

    A total of 21 interventions were identified out of which, 7 (33%) were webinars; 7 (33%) documentary, TV series, or videos; and 7 (33%) other (eg, text article, modules). Medscape, a health information website, provided 10 (48%) interventions, Continuing Nursing Education University provided 2 (10%), CDC provided 3 (14%), professional organizations (eg, American Academy of Pediatrics and Texas Medical Association) provided 3 (14%), nonprofit organizations (eg, Indiana Immunization Coalition) provided 1 (5%), a federally-authorized regional organization (The Suwannee River Area Health Education Center) provided 1 (5%), and a university (Boston University School of Medicine Continuing Medical Education and Continuing Nursing Education) provided 1 (5%). Multimedia Appendix 1 includes the characteristics of the interventions.

    Quality Evaluation

    On the basis of the evaluation indicators, 13 (62%) interventions required registration, but all interventions were accessible without cost to the participant (K-alpha=1.0). Additionally, 17 (81%) educational interventions identified the purpose of the intervention (K-alpha=1.0), and 12 (57%) provided a date when the information had been updated: 7 (33%) were updated in the last 6 months (K-alpha=1.0). In assessing presentation of clear references, 18 (86%) interventions provided references (K-alpha=1.0), and 8 (38%) provided links to other sources (K-alpha=1.0). Most interventions (18/21, 85%) provided reliable references or sources (K-alpha=1.0). The references or sources included information from the CDC (n=16), published peer-reviewed literature (n=16), American Cancer Society (n=5), National Cancer Institute (n=4), Institutes of Medicine (n=4), WHO (n=2), and American Academy of Pediatrics (n=1). Of the 21 interventions, 14 (67%) had a statement of disclosures from the authors, sponsors, or developers (K-alpha=1.0).

    For the design subindicators, 2 interventions were documentary or videos that did not include text, and therefore, were not included in the total sample for these subindicators. All interventions (n=19) included easy-to-read font size, font style, color, and line spacing (K-alpha=1.0 for these 3 subindicators). For the subindicator “Graphics were clearly labeled,” only 13 interventions included graphics; thus, the sample for this subindicator is 13 interventions. Out of the 13 interventions, there were 10 (77%) interventions with a minimum of 75% of graphics labeled, 2 (15%) with a minimum of 50% of graphics labeled, and 1 (8%) intervention with a minimum of 25% of graphics labeled. No intervention had 0% of graphics labeled (K-alpha=1.0).

    Of the 21 interventions, 17 (81%) included an evaluation for participant outcomes: 17 (81%) assessed HPV and HPV vaccine knowledge, and none assessed attitudes toward HPV and the HPV vaccine. Furthermore, 14 (67%) interventions provided the participants the opportunity to evaluate or provide feedback (K-alpha=1.0). Five (24%) interventions included an interactive component (K-alpha=1.0). The most commonly used interactive component was a polling or knowledge check activity (n=4). No educational intervention explicitly stated a theory or model that was used to develop the intervention. Table 1 provides additional results from the evaluation, and Table 2 provides the quality summary score for each Web-based intervention.

    Table 1. Web-based educational intervention quality evaluation results (n=21).
    Indicator and subindicatorScoring frequency


    		YesNo
    Access


    		Registration required138

    		Cost021
    Content


    		Date information was updated129

    		Identification of purpose174

    		Presentation of clear references183

    		Links to other sources813

    		Reliable references and sourcesa183

    		Statement indicating content was developed or reviewed by experts2014

    		Disclosure of authors, sponsors, or developers147
    Design


    		Font style was easy to readb190

    		Font size was easy to readb190

    		Font color and page color contrast was easy to readb190

    		Line spacing was easy to readb190

    		Graphics were clearly labeledc130
    Evaluation


    		Evaluation for participant outcomesd174

    		Participant provided opportunity to evaluate intervention147
    Interactivity


    		Location to direct participant questions615

    		Included interactive componente516
    Theory or model(s)


    		Theory or model was used to develop intervention021

    aThe Centers for Disease Control and Prevention (n=16) and published peer reviewed literature (n=16) were the most common frequency cited sources, followed by American Cancer Society (n=5), National Institutes of Health (n=4), Institute of Medicine (n=4), World Health Organization (n=2), Food and Drug Administration (n=1), and the American Academy of Pediatrics (n=1).

    bTwo interventions were a documentary or TV series that did not include any type of font or graphics for informational purposes. Therefore, for the Design subindicators font style, font size, font color, and line spacing, the sample size was n=19.

    cFor the graphic subindicator, eight interventions did not include graphics for informational purpose. Therefore, the sample size was n=13. There were 10 interventions with a minimum of 75% of graphics labeled, 2 interventions with a minimum of 50% of graphics labeled, 1 intervention with a minimum of 25% of graphics labeled.

    dSpecific levels of evaluation for participant outcomes include 17 interventions assessing HPV and HPV vaccine knowledge, and no intervention assessing attitudes toward HPV and the HPV vaccine.

    eFive interventions provided participant interactivity. Four interventions included an interactive knowledge check, and 1 intervention included directions to email reminders and newsletters.

    Table 2. Quality summary scores for Web-based interventions.
    Intervention titleaSummary score (out of 25)
    HPV Vaccine Safety and Efficacy20
    HPV Vaccines: Updates and Clinical Perspective20
    Increasing Adolescent Immunization Coverage20
    Don’t Wait Vaccinate! The Prevention of HPV Cancers (Part 2)19
    HPV Vaccination is Cancer Prevention (2017 Update)19
    Overcoming Gender and Socioeconomic Disparities in HPV Vaccination19
    You are the Key to HPV Cancer Prevention18b
    Don’t Wait Vaccinate! The Prevention of HPV Cancers17
    Immunization: You Call the Shots-Module Eight-HPV, 201617
    Immunization: You Call the Shots-Module Eighteen—Vaccine Administration17c
    You are the Key to HPV Cancer Prevention17d
    ACIP Releases Pediatric Vaccine Schedule16c
    Adolescent Immunizations: Strongly Recommending the HPV Vaccine16
    AAP Provides Guidance for Parents Who Refuse Vaccination15c
    ACIP Releases Adult Vaccine Recommendations15c
    CDC Updates Guideline Recommendations for HPV Vaccination15c
    Human Papillomavirus (HPV) Vaccine Safety15c
    The Story of HPV: Yesterday, Today, and Tomorrow14
    HPV Vaccines: Updates and Clinical Perspective13
    Putting HPV Vaccine Knowledge Into Practice7e
    HPV Documentary—Division of Continuing Medical Education2e

    aHPV: Human Papillomavirus; ACIP: Advisory Committee on Immunization Practices; AAP: American Academy of Pediatrics; CDC: Centers for Disease Control and Prevention.

    bYou are the key to HPV Cancer Prevention intervention published 9/2/2015 and expires 9/7/2017.

    cThese interventions did not include any type of graphics for informational purpose. Therefore, the total score is out of 24.

    dYou are the key to HPV Cancer Prevention intervention published 4/21/2016 and expires 4/21/2018.

    eThese interventions were documentaries and did not include any type of font or graphics for informational purposes. Therefore, the total score is out of 20.


    Principal Findings

    This study provides a systematic, evidence-based assessment of the strengths and weaknesses of current HPV vaccine Web-based educational interventions. Strengths of the assessed Web-based educational interventions include: (1) being developed by experts in the field; (2) providing reliable references or sources; (3) providing clinicians with access to CEs for no cost; (4) following basic design principles with easy-to-read fonts, colors, and graphics; and (5) consistently providing evaluation opportunities for participant knowledge outcomes. Weaknesses of the educational interventions included lack of: (1) evaluation of outcomes including participants’ attitudes about HPV vaccination, intention to recommended vaccination, and recommendation of behaviors; (2) theory-based interventions; (3) opportunity for participants to provide feedback or evaluation of the intervention; (4) links to other sources or resources; and (5) interactivity throughout the intervention.

    HPV vaccination rates are well below the Healthy People 2020 objective [8], and clinicians report that they would benefit from CE regarding the HPV vaccine [29]. Because clinicians’ HPV vaccine recommendation is one of the most important predictors of HPV vaccination uptake [59-62], ensuring that clinicians are equipped with current and accurate information is critical [63]. Clinicians, however, are continually challenged in providing parents and patients with evidence-based HPV vaccine information because of changing vaccine guidelines and the volume of information and sources available [64]. Thus, clinicians’ report obtaining a large portion of HPV vaccine information from professional organizations [65] possibly because of lack of time needed to identify multiple sources of accurate information [66]. In this study, only 3 interventions were provided by 2 professional organizations, including the American Academy of Pediatrics and the Texas Medical Association. Medscape, a health information website, provided almost half of the interventions. Given that professional organizations are cited by clinicians as an important and trusted source of HPV vaccine information, professional organizations need to increase efforts to collaborate with health information websites and other organizations and institutions to provide evidence- and theory-based interventions. A recent study demonstrated that organizations working on cancer research identified the ability to leverage resources, lower costs, increase organization reputation, and the development of new tools and methodology as benefits to interorganizational collaboration [67]. Therefore, interorganizational collaboration to provide clinicians with HPV vaccine Web-based interventions has the potential to improve outcomes related to HPV vaccination rates and cancer risk reduction.

    We found that none of the interventions examined included a statement that a theory was used to create the intervention. To improve outcome behaviors and increase clinician HPV vaccine recommendation behaviors, intervention developers should utilize science and evidence that supports effective medical education and behavior change [54]. Theories can be used for quality assessment and improvement by identifying factors contributing to behavior change and which factors are ineffective. Overall, interventions based in theory provide an advantage in changing behavior by providing a logical and systematic approach to increasing clinicians’ recommendation of the HPV vaccine [54].

    None of the Web-based educational interventions included in this study evaluated HPV attitudes, intention to recommend vaccination, or actual recommendation behavior. This is concerning given that clinician attitudes are an important predictor of vaccine recommendations. Clinicians have reported concerns regarding HPV vaccine safety [68-70], a lack of self-confidence in providing strong vaccine recommendations [15,27,28], and belief that it is not important for adolescents to receive the HPV vaccine at the recommended age of 11 to 12 years [71]. Therefore, interventions should be designed with the goal of changing clinician attitudes and vaccine recommendations, and evaluation of these outcomes is a key component of successful interventions [45,52]. Evaluation of outcomes is also important for the translation of health communication research into efforts to promote clinician recommendations of the HPV vaccine [44,46]. Finally, evaluation is essential for understanding clinicians’ educational needs and assessing program outcomes addressing important health issues [50].

    Although face-to-face educational interventions have shown to improve clinicians’ willingness to provide immunizations and routinely screen immunization records at visits [72], evaluations specifically assessing HPV-related Web-based educational interventions are limited [73]. Only 2 published studies provided evaluation results on webinars designed to increase adolescent vaccination rates. Results suggest webinars have the potential to increase clinician recommendation behaviors and adolescent Tdap, meningococcal, and HPV vaccination rates similar to in-person educational interventions [74,75]. Web-based educational interventions create a unique platform to provide clinicians with the knowledge and skills needed to promote the HPV vaccine among adolescents. One important component of Web-based educational interventions is interactivity [54,55]. Interactive components encourage users to be actively involved in the intervention and have been linked to short-term behavioral improvements [76]. Furthermore, Kreps and Neuhauser pinpoint interactivity as a communication attribute with the ability to exponentially improve health promotion [76]. Even though interactivity can have a significant impact on participants, only 5 interventions from this study included an interactive component. Because Web-based educational interventions continue to gain popularity because of convenience and economic benefits [31], the lack of interactivity in the majority of HPV-related Web-based educational interventions is alarming given the research supporting the importance of interactive components. More research should be conducted to determine the impact of interactive components in HPV-related Web-based educational interventions on clinicians’ HPV vaccination recommendation behaviors.

    Limitations

    While this study provides innovative insight into the quality of Web-based HPV vaccine educational interventions created for clinicians, there are limitations that should be considered. First, only Web-based educational interventions were evaluated, and these results cannot be generalized to other types of interventions such as face-to-face lectures, grand rounds, or seminars. There would be substantial benefit to conducting evaluations of face-to-face lectures and seminar materials to assess all venues of continuing education for clinicians regarding the HPV vaccine. Second, this quality evaluation did not assess participants’ experience of the intervention and therefore, cannot identify every area for improvement. Data were not collected from participants themselves regarding usability: this study identified only 7 indicators of usability. Third, this study was a quality evaluation and did not evaluate participant outcomes (eg, knowledge, attitudes, recommendation behaviors) after completing the intervention. Fourth, the evaluation was conducted using only the materials that were accessible at the time of the study, and there is the potential that a component (such as, a follow-up emailed evaluation after the intervention to participants) was not included in this evaluation. Despite these limitations, these findings provide valuable information for those who develop Web-based continuing education interventions regarding HPV vaccines, by providing a quantitative approach to identifying the design and usability strengths and weaknesses of HPV vaccine Web-based educational interventions.

    Future Work

    The data resulting from this study have the potential to help shift current research practice paradigms. The findings suggest that those who develop Web-based educational interventions to promote HPV vaccine recommendations utilize design science principles, a powerful approach and process that includes participatory action research to iteratively develop and evaluate health education interventions [77]. Additional qualitative, multi-approach evaluation research is needed to further assess the content (eg, the specific messages provided to clinicians about the HPV vaccine and recommendation behaviors) and usability of these interventions from the participants’ perspective. Further evaluation research is needed to ensure that interventions are being developed using all design principles and are effective at increasing strong and consistent HPV vaccine recommendations from clinicians.

    Conclusions

    The quality evaluation of these interventions demonstrated that Web-based interventions were based on reliable sources, developed by experts, and were created with critical design aspects (eg, font style, size, and color were easy to read, graphics were clearly labeled). However, there were limited outcome evaluations for users measuring attitudes, intentions, or behaviors, and lack of user interactivity. Results from this study suggest best practices for designing, refining, and implementing Web-based interventions to promote HPV vaccination within the clinician population.

    Acknowledgments

    No funding was received for this study.

    Authors' Contributions

    BLR is the primary author, JMB conducted intervention evaluation and assisted in writing the manuscript, SLM conducted intervention evaluation and assisted in writing the manuscript, JK provided expert guidance for the evaluation and revised the manuscript, and GLK provided overall guidance for the evaluation and provided extensive revisions of the manuscript.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Intervention characteristics.

    PDF File (Adobe PDF File), 290KB
    1. CDC. 2017 Mar 06. HPV-Associated Cancer Statistics   URL: https://www.cdc.gov/cancer/hpv/statistics/index.htm [WebCite Cache]
    2. Brown LM, Check DP, Devesa SS. Oropharyngeal cancer incidence trends: diminishing racial disparities. Cancer Causes Control 2011 May;22(5):753-763. [CrossRef] [Medline]
    3. Hocking JS, Stein A, Conway EL, Regan D, Grulich A, Law M, et al. Head and neck cancer in Australia between 1982 and 2005 show increasing incidence of potentially HPV-associated oropharyngeal cancers. Br J Cancer 2011 Mar 1;104(5):886-891 [FREE Full text] [CrossRef] [Medline]
    4. Reddy VM, Cundall-Curry D, Bridger MW. Trends in the incidence rates of tonsil and base of tongue cancer in England, 1985-2006. Ann R Coll Surg Engl 2010 Nov;92(8):655-659 [FREE Full text] [CrossRef] [Medline]
    5. Petrosky E, Bocchini Jr JA, Hariri S, Chesson H, Curtis CR, Saraiya M, Centers for Disease Control and Prevention (CDC). Use of 9-valent human papillomavirus (HPV) vaccine: updated HPV vaccination recommendations of the advisory committee on immunization practices. MMWR Morb Mortal Wkly Rep 2015 Mar 27;64(11):300-304 [FREE Full text] [Medline]
    6. Saraiya M. Population-based HPV genotype attribution in HPV-associated cancers. 2017 Presented at: Proceedings of the Anal Intraepithelial Neoplasia Society Conference; March 13-15, 2015; Atlanta, Georgia.
    7. Reagan-Steiner S, Yankey D, Jeyarajah J, Elam-Evans LD, Curtis CR, MacNeil J, et al. National, regional, state, and selected local area vaccination coverage among adolescents aged 13-17 years - United States, 2015. MMWR Morb Mortal Wkly Rep 2016 Aug 26;65(33):850-858 [FREE Full text] [CrossRef] [Medline]
    8. Healthy People. Immunization and infectious diseases   URL: https://www.healthypeople.gov/2020/topics-objectives/topic/immunization-and-infectious-diseases [accessed 2017-09-21] [WebCite Cache]
    9. Centers for Disease Control and Prevention (CDC). Human papillomavirus vaccination coverage among adolescent girls, 2007-2012, and postlicensure vaccine safety monitoring, 2006-2013 - United States. MMWR Morb Mortal Wkly Rep 2013 Jul 26;62(29):591-595 [FREE Full text] [Medline]
    10. Quinn GP, Murphy D, Malo TL, Christie J, Vadaparampil ST. A national survey about human papillomavirus vaccination: what we didn't ask, but physicians wanted us to know. J Pediatr Adolesc Gynecol 2012 Aug;25(4):254-258 [FREE Full text] [CrossRef] [Medline]
    11. Vadaparampil ST, Murphy D, Rodriguez M, Malo TL, Quinn GP. Qualitative responses to a national physician survey on HPV vaccination. Vaccine 2013 Apr 26;31(18):2267-2272 [FREE Full text] [CrossRef] [Medline]
    12. Daley MF, Crane LA, Markowitz LE, Black SR, Beaty BL, Barrow J, et al. Human papillomavirus vaccination practices: a survey of US physicians 18 months after licensure. Pediatrics 2010 Sep;126(3):425-433. [CrossRef] [Medline]
    13. Perkins RB, Clark JA. What affects human papillomavirus vaccination rates? A qualitative analysis of providers' perceptions. Womens Health Issues 2012;22(4):e379-e386. [CrossRef] [Medline]
    14. McCave EL. Influential factors in HPV vaccination uptake among providers in four states. J Community Health 2010 Dec;35(6):645-652. [CrossRef] [Medline]
    15. Perkins RB, Clark JA. Providers' attitudes toward human papillomavirus vaccination in young men: challenges for implementation of 2011 recommendations. Am J Mens Health 2012 Jul;6(4):320-323. [CrossRef] [Medline]
    16. Yarnall KS, Pollak KI, Østbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health 2003 Apr;93(4):635-641. [Medline]
    17. Solberg LI, Nordin JD, Bryant TL, Kristensen AH, Maloney SK. Clinical preventive services for adolescents. Am J Prev Med 2009 Nov;37(5):445-454. [CrossRef] [Medline]
    18. Yarnall KS, Østbye T, Krause KM, Pollak KI, Gradison M, Michener JL. Family physicians as team leaders: “time” to share the care. Prev Chronic Dis 2009 Apr;6(2):A59 [FREE Full text] [Medline]
    19. Holman DM, Benard V, Roland KB, Watson M, Liddon N, Stokley S. Barriers to human papillomavirus vaccination among US adolescents: a systematic review of the literature. JAMA Pediatr 2014 Jan;168(1):76-82 [FREE Full text] [CrossRef] [Medline]
    20. Kimmel SR. Adolescent vaccination: physician challenges and solutions. J Adolesc Health 2010 Feb;46(2):S16-S23. [CrossRef]
    21. Constantine NA, Jerman P. Acceptance of human papillomavirus vaccination among Californian parents of daughters: a representative statewide analysis. J Adolesc Health 2007 Feb;40(2):108-115. [CrossRef] [Medline]
    22. Guerry SL, De Rosa CJ, Markowitz LE, Walker S, Liddon N, Kerndt PR, et al. Human papillomavirus vaccine initiation among adolescent girls in high-risk communities. Vaccine 2011 Mar 9;29(12):2235-2241. [CrossRef] [Medline]
    23. Perkins RB, Pierre-Joseph N, Marquez C, Iloka S, Clark JA. Why do low-income minority parents choose human papillomavirus vaccination for their daughters? J Pediatr 2010 Oct;157(4):617-622 [FREE Full text] [CrossRef] [Medline]
    24. Litton AG, Desmond RA, Gilliland J, Huh WK, Franklin FA. Factors associated with intention to vaccinate a daughter against HPV: a statewide survey in Alabama. J Pediatr Adolesc Gynecol 2011 Jun;24(3):166-171 [FREE Full text] [CrossRef] [Medline]
    25. Dorell C, Yankey D, Strasser S. Parent-reported reasons for nonreceipt of recommended adolescent vaccinations, national immunization survey: teen, 2009. Clin Pediatr (Phila) 2011 Dec;50(12):1116-1124. [CrossRef] [Medline]
    26. Rosenthal SL, Weiss TW, Zimet GD, Ma L, Good MB, Vichnin MD. Predictors of HPV vaccine uptake among women aged 19-26: importance of a physician's recommendation. Vaccine 2011 Jan 29;29(5):890-895. [CrossRef] [Medline]
    27. Saraiya M, Rosser JI, Cooper CP. Cancers that U.S. physicians believe the HPV vaccine prevents: findings from a physician survey, 2009. J Womens Health (Larchmt) 2012 Feb;21(2):111-117. [CrossRef] [Medline]
    28. Gowda C, Schaffer S, Dombkowski K, Dempsey A. Understanding attitudes toward adolescent vaccination and the decision-making dynamic among adolescents, parents and providers. BMC Public Health 2012;12:509. [Medline]
    29. Bruno DM, Wilson TE, Gany F, Aragones A. Identifying human papillomavirus vaccination practices among primary care providers of minority, low-income and immigrant patient populations. Vaccine 2014;32(33):4149-4154. [Medline]
    30. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med 2010 May;85(5):909-922. [CrossRef] [Medline]
    31. Olivieri JJ, Knoll MB, Arn PH. Education format and resource preferences among registrants of a pediatric-focused CME website. Med Teach 2009;31(8):e333-e337. [Medline]
    32. Shaw T, Barnet S, Mcgregor D, Avery J. Using the knowledge, process, practice (KPP) model for driving the design and development of online postgraduate medical education. Med Teach 2015;37(1):53-58. [Medline]
    33. Goldberg H, McKhann G. Students test scores are improved in a virtual learning environment. Adv Physiol Educ 2000;23(1):59-66. [Medline]
    34. Harris JM, Novalis-Marine C, Harris RB. Women physicians are early adopters of on-line continuing medical education. J Contin Educ Health Prof 2003;23(4):221-228. [Medline]
    35. Neuhauser L, Paul K. Readability, comprehension and usability. In: Fischhoff B, Brewer NT, Downs JS, editors. Communicating Risks and Benefits: An Evidence Based User's Guide. Silver Spring, MD: US Department of Health and Human Services; 2011:129-148.
    36. Kreps GL. Strategic use of communication to market cancer prevention and control to vulnerable populations. Health Mark Q 2008;25(1-2):204-216. [CrossRef] [Medline]
    37. Fordis M, King JE, Ballantyne CM, Jones PH, Schneider KH, Spann SJ, et al. Comparison of the instructional efficacy of Internet-based CME with live interactive CME workshops: a randomized controlled trial. J Am Med Assoc 2005 Sep 7;294(9):1043-1051. [CrossRef] [Medline]
    38. Allison JJ, Kiefe CI, Wall T, Casebeer L, Ray MN, Spettell CM, et al. Multicomponent Internet continuing medical education to promote chlamydia screening. Am J Prev Med 2005 Apr;28(3):285-290. [CrossRef] [Medline]
    39. Casebeer LL, Strasser SM, Spettell CM, Wall TC, Weissman N, Ray MN, et al. Designing tailored Web-based instruction to improve practicing physicians' preventive practices. J Med Internet Res 2003;5(3):e20 [FREE Full text] [CrossRef] [Medline]
    40. Carney PA, Dietrich AJ, Freeman Jr DH, Mott LA. A standardized-patient assessment of a continuing medical education program to improve physicians' cancer-control clinical skills. Acad Med 1995 Jan;70(1):52-58. [Medline]
    41. Harris Jr JM, Kutob RM, Surprenant ZJ, Maiuro RD, Delate TA. Can Internet-based education improve physician confidence in dealing with domestic violence? Fam Med 2002 Apr;34(4):287-292. [Medline]
    42. Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, et al. AHRQ. 2007. Effectiveness of Continuing Medical Education   URL: https://archive.ahrq.gov/downloads/pub/evidence/pdf/cme/cme.pdf [accessed 2018-02-07] [WebCite Cache]
    43. Rosen BL, Kreps G, Shepard A. Content and usability evaluation of HPV vaccine online educational interventions for clinicians. 2017 Presented at: Proceeding of the American Academy of Health Behavior 2017 Annual Scientific Meeting; March 19-22, 2017; Tucson, Arizona.
    44. Kreps GL. Translating health communication research into practice: the importance of implementing and sustaining evidence-based health communication interventions. Atlantic J Commun 2012 Jan 25;20(1):5-15. [CrossRef]
    45. Kreps GL. Evaluating new health information technologies: expanding the frontiers of health care delivery and health promotion. Stud Health Technol Inform 2002;80:205-212. [Medline]
    46. Kreps GL. Translating health communication research into practice: the influence of health communication scholarship on health policy, pratice, and outcomes. In: Thompson TL, Parrott R, Nussbaum JF, editors. The handbook of health communication. 2nd edition. New York, NY: Routledge; 2011:595-608.
    47. Kreps GL. Epilogue: lessons learned about evaluating health communication programs. J Health Commun 2014 Dec;19(12):1510-1514. [CrossRef] [Medline]
    48. Cho H, Salmon CT. Unintended effects of health communication campaigns. J Commun 2007 Jun;57(2):293-317. [CrossRef]
    49. Ringold DJ. Boomerang effects in response to public health interventions: some unintended consequences in the alcoholic beverage market. J Consumer Policy 2002;25(1):27-63. [CrossRef]
    50. Neuhauser L, Kreps GL. Integrating design science theory and methods to improve the development and evaluation of health communication programs. J Health Commun 2014;19(12):1460-1471. [CrossRef] [Medline]
    51. Centers for Medicare and Medicaid Services. National health care expenditures data. Report. Baltimore, MD: CMS. Office of the Actuary, National Health Statistics Group; 2012.
    52. Kreps GL. Evaluating health communication programs to enhance health care and health promotion. J Health Commun 2014;19(12):1449-1459. [CrossRef] [Medline]
    53. Kim P, Eng TR, Deering MJ, Maxfield A. Review of published criteria for evaluating health-related websites. West J Med 1999;170(6):329-332 [FREE Full text] [Medline]
    54. Evers KE, Prochaska JM, Prochaska JO, Driskell M, Cummins CO, Velicer WF. Strengths and weaknesses of health behavior change programs on the internet. J Health Psychol 2003 Jan;8(1):63-70. [CrossRef] [Medline]
    55. Cummins CO, Prochaska JO, Driskell MM, Evers KE, Wright JA, Prochaska JM, et al. Development of review criteria to evaluate health behavior change websites. J Health Psychol 2003 Jan;8(1):55-62. [CrossRef] [Medline]
    56. Yen P, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012 Jun;19(3):413-422 [FREE Full text] [CrossRef] [Medline]
    57. De Swert K. Calculating inter-coder reliability in media content analysis using Krippendorff’s Alpha   URL: http://www.polcomm.org/wp-content/uploads/ICR01022012.pdf [accessed 2017-09-21] [WebCite Cache]
    58. Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas 2007 Apr;1(1):77-89. [CrossRef]
    59. Caskey R, Lindau ST, Alexander GC. Knowledge and early adoption of the HPV vaccine among girls and young women: results of a national survey. J Adolesc Health 2009 Nov;45(5):453-462. [CrossRef] [Medline]
    60. Conroy K, Rosenthal SL, Zimet GD, Jin Y, Bernstein DI, Glynn S, et al. Human papillomavirus vaccine uptake, predictors of vaccination, and self-reported barriers to vaccination. J Womens Health (Larchmt) 2009 Oct;18(10):1679-1686. [CrossRef] [Medline]
    61. Gamble HL, Klosky JL, Parra GR, Randolph ME. Factors influencing familial decision-making regarding human papillomavirus vaccination. J Pediatr Psychol 2010 Aug;35(7):704-715 [FREE Full text] [CrossRef] [Medline]
    62. Do H, Seng P, Talbot J, Acorda E, Coronado GD, Taylor VM. HPV vaccine knowledge and beliefs among Cambodian American parents and community leaders. Asian Pac J Cancer Prev 2009;10(3):339-344 [FREE Full text] [Medline]
    63. Pickering LK, Baker CJ, Freed GL, Gall SA, Grogg SE, Poland GA, Infectious Diseases Society of America. Immunization programs for infants, children, adolescents, and adults: clinical practice guidelines by the Infectious Diseases Society of America. Clin Infect Dis 2009 Sep 15;49(6):817-840. [CrossRef] [Medline]
    64. Hillenbrand KM. What is going on with vaccines: keeping up with the childhood immunization schedule. J Public Health Manag Pract 2007;13(6):544-552. [CrossRef] [Medline]
    65. Bynum SA, Malo TL, Lee J, Guiliano AR, Vadaparampil ST. HPV vaccine information-seeking behaviors among US physicians: government, media, or colleagues? Vaccine 2011 Jul 18;29(32):5090-5093 [FREE Full text] [CrossRef] [Medline]
    66. Verhoeven F, Steehouder MF, Hendrix RM, van Germert-Pijnen JE. Factors affecting health care workers' adoption of a website with infection control guidelines. Int J Med Inform 2009 Oct;78(10):663-678. [CrossRef] [Medline]
    67. Harris JK, Provan KG, Johnson KJ, Leischow SJ. Drawbacks and benefits associated with inter-organizational collaboration along the discovery-development-delivery continuum: a cancer research network case study. Implement Sci 2012 Jul 25;7:69 [FREE Full text] [CrossRef] [Medline]
    68. Rosen BL, Ashwood D, Richardson GB. School nurses' professional practice in the HPV vaccine decision making process. J Sch Nurs 2016 Apr;32(2):138-148. [CrossRef] [Medline]
    69. Alexander AB, Best C, Stupiansky N, Zimet GD. A model of health care provider decision making about HPV vaccination in adolescent males. Vaccine 2015 Aug 7;33(33):4081-4086. [CrossRef] [Medline]
    70. Young JL, Bernheim RG, Korte JE, Stoler MH, Guterbock TM, Rice LW. Human papillomavirus vaccination recommendation may be linked to reimbursement: a survey of Virginia family practitioners and gynecologists. J Pediatr Adolesc Gynecol 2011 Dec;24(6):380-385. [CrossRef] [Medline]
    71. Henrikson NB, Tuzzio L, Gilkey MB, McRee AL. “You're never really off time”: Healthcare providers' interpretations of optimal timing for HPV vaccination. Prev Med Rep 2016;4:94-97 [FREE Full text] [CrossRef] [Medline]
    72. Boom JA, Nelson CS, Laufman LE, Kohrt AE, Kozinetz CA. Improvement in provider immunization knowledge and behaviors following a peer education intervention. Clin Pediatr (Phila) 2007 Oct;46(8):706-717. [CrossRef] [Medline]
    73. Berenson AB, Rahman M, Hirth JM, Rupp RE, Sarpong KO. A brief educational intervention increases providers' human papillomavirus vaccine knowledge. Hum Vaccin Immunother 2015;11(6):1331-1336 [FREE Full text] [CrossRef] [Medline]
    74. Moss JL, Reiter PL, Dayton A, Brewer NT. Increasing adolescent immunization by webinar: a brief provider intervention at federally qualified health centers. Vaccine 2012 Jul 13;30(33):4960-4963. [CrossRef] [Medline]
    75. Gilkey MB, Dayton AM, Moss JL, Sparks AC, Grimshaw AH, Bowling JM, et al. Increasing provision of adolescent vaccines in primary care: a randomized controlled trial. Pediatrics 2014 Aug;134(2):e346-e353 [FREE Full text] [CrossRef] [Medline]
    76. Kreps GL, Neuhauser L. New directions in eHealth communication: opportunities and challenges. Patient Educ Couns 2010 Mar;78(3):329-336. [CrossRef] [Medline]
    77. Neuhauser L, Kreps GL, Morrison K, Athanasoulis M, Kirienko N, Van Brunt D. Using design science and artificial intelligence to improve health communication: chronologyMD case example. Patient Educ Couns 2013 Aug;92(2):211-217. [CrossRef] [Medline]

    ‎
    AAP: American Academy of Pediatrics
    ACIP: Advisory Committee on Immunization Practices
    CDC: Centers for Disease Control and Prevention
    CE: continuing education
    HPV: human papillomavirus

    Edited by G Eysenbach; submitted 04.10.17; peer-reviewed by TW Chien, MR Dela Cruz; comments to author 29.11.17; revised version received 13.12.17; accepted 13.12.17; published 16.02.18

    Copyright

    ©Brittany L Rosen, James M Bishop, Skye L McDonald, Jessica A Kahn, Gary L Kreps. Originally published in JMIR Cancer (http://cancer.jmir.org), 16.02.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on http://cancer.jmir.org/, as well as this copyright and license information must be included.