A multimethod approach was used to evaluate the feasibility, acceptability, and preliminary efficacy of a mobile web app, HPVVaxFacts, among parents of unvaccinated children aged 9 to 17 years [32]. A pilot, 2-arm, multisite RCT was conducted at 2 urban pediatric clinics in Tennessee to determine feasibility and acceptability. Clinic Site 1 is an urban pediatric clinic serving more than 8000 children with a broader demographic of families with varied socioeconomic backgrounds. There were 421 unvaccinated children at Site 1 from June to September 2023. In contrast, Clinic Site 2 is an urban pediatric clinic serving more than 5000 children primarily uninsured, underinsured, and marginalized children. A total of 23 unvaccinated children were at Site 2 for 1 week in August 2023. Clinic providers and staff stated they had higher vaccination rates, so they chose a 1-week back-to-school event where vaccinations were offered. This time was perceived to have the highest volume of unvaccinated patients. While operational workflows (ie, administrative protocols, patient volume, and technology adoption) and catchment areas differ, both had patient intake management platforms for recruitment and followed the standardized protocols to be discussed in detail below.

Parents in both arms who agreed to follow-up were interviewed to explore acceptability and feasibility of the protocol using a qualitative, descriptive study design. Intervention acceptability and parent-child discussion were examined among parents in the intervention arm only. The reporting of the pilot RCT follows the CONSORT (Consolidated Standards of Reporting Trials) 2010 guidelines to report a pilot or feasibility trial (Multimedia Appendix 1) and the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist to evaluate web-based apps (CONSORT-HEALTH checklist in Checklist 1) [33].

The Institutional Review Board (IRB) of Meharry Medical College (Protocol Number: 18-12-890) approved this study as expedited as it was considered no more than minimal risk. The study was registered at ClinicalTrials.gov (NCT04380623) and performed in line with the principles of the Declaration of Helsinki. Electronic consent was obtained from participants using REDCap (Vanderbilt University) [34,35] for the pilot RCT, and verbal consent for the semistructured interviews. To protect participant information, survey and interview data were stored via REDCap and on a password-protected server. Data were deidentified prior to provision to statisticians and research team members for data analysis. Dissemination of results was aggregate and anonymous. Participants were compensated with a US $30 gift card for pilot study participation, and a US $20 gift card for completing a follow-up interview.

HPVVaxFacts provides individually tailored education on HPV and the vaccine to parents [36]. Education and video content for each concern was developed using the Health Belief Model and Theory of Reasoned Action (Figure 1). To tailor the education in the app, each parent answered a 10-item baseline survey adapted from the Vaccination Confidence Scale [37] to identify parental concerns. The survey responses are mapped to an algorithm that selects the top 3 parent concerns. Parents can change the selection if they disagree. Parents then review the content (ie, educational information, provider’s video, questions for the provider, and links to other credible sources verifying content) associated with each concern. An adolescent corner is available for parents to share with and/or have an optional discussion with their child about each concern. Finally, parents could submit concerns or additional needs via the provision of feedback. Refer to Cunningham-Erves et al [36] for full description of HPVVaxFacts development.

Parents in the attention control arm received a link to a website providing nutrition education for parents of children and/or teens [38], a program unrelated to the study outcome. This approach served two purposes: (1) ensuring all parents receive a benefit regardless of study arm assignment, and (2) balancing time and effort between groups [39]. Because parents’ perceptions of HPV vaccination significantly influence vaccination outcomes [40], an attention control group allowed us to account for potential placebo effects and isolate the specific effects of HPVVaxFacts. Furthermore, the study arm activities were comparable to the intervention activities. Parents completed a 5-item brief quiz with immediate feedback on correct or incorrect responses and explanations.

This study was conducted with parents who were caregivers for children in 2 urban pediatric clinics in Tennessee who were representative of the continuum of vaccine hesitancy. Eligible parents had a child aged 9 to 17 years who was unvaccinated against HPV, had a mobile phone, had an upcoming clinic visit, and spoke English. When multiple children per eligible caregiver existed, random selection determined which child would participate. Random numbers were first generated in Stata version 17 (StataCorp LLC). Children were ordered from youngest to oldest (or alphabetized by first names if twins or triplets). The child with the lowest corresponding random number was selected for the parent to be sent a study invitation.

Eligibility criteria were chosen based on participating providers and staff input. They stated it was not feasible to enroll only hesitant parents. Instead, they perceived it was equally beneficial to recruit parents representing the continuum of hesitancy. Exclusion criteria were parents who did not consent, or their child had initiated or completed the HPV vaccination series.

We documented whether participants complied with assessment and data collection protocols. This included (1) completion rates of the pre-post survey, (2) mobile web app tracking of parent concerns and selected top 3 concerns by date and time, and (3) the ability to obtain the necessary data from clinic administrative records, particularly age and HPV vaccination status. Feasibility was measured as 70% of parents viewing the mobile web app.

We assessed the internal reliability of the concerns, knowledge, and intervention acceptability scales. Participants were allowed to provide feedback on the HPVVaxFacts mobile web app to determine whether important concerns were missing from the app, which suggests missed questions on the concern scale.

We assessed the number of participants recruited and retained for the study.

These variables were in 2 categories—parent and child sociodemographics.

Age was a continuous variable. Sex was categorized as male, female, and prefer not to disclose. Parent sexual orientation included categories of asexual, bisexual, straight, fluid, and gay. Categories were combined to straight (heterosexual), prefer not to disclose, and other. Parents’ race or ethnic categories were non-Hispanic White, African American, Latino or Spanish origin, Native Hawaiian or Other Pacific Islander, and Prefer Not to Answer. Education categories were General Educational Development (GED) or high school diploma, associate’s degree, some college, bachelor’s degree, master’s degree, and doctoral or professional degree. Annual household income categories were less than US $20,000, US $20,001 to US $40,000, US $40,001 to US $60,000, US $60,001 to US $80,000, and more than $80,000, or did not answer.

Age, a continuous variable, was categorized to 9‐10, 11‐12, and ≥13. Sex was categorized as male, female, and prefer not to disclose. The child’s race or ethnic categories were non-Hispanic White, African American, Latino or Spanish origin, American Indian and Alaskan Native, Prefer Not to Answer, and Other. The child’s sexual orientation had several categories including aromantic, asexual, bisexual, fluid, straight, and gay. These categories were combined to straight (heterosexual), questioning or unsure, prefer not to disclose, and other.

Descriptive statistics characterized the sample: continuous variables are reported as means and SDs, and categorical variables as frequencies and percentages. Between-group comparisons were conducted using appropriate statistical tests: Fisher exact test or chi-square tests for categorical variables and Wilcoxon rank-sum tests or t tests for continuous variables. Comparison of pre-post scores of HPV vaccine knowledge, concerns, and intention by study arm was performed using exploratory visual analysis. Provider recommendation was not included as a covariate because (1) recommendations were expected to be similar across study arms and (2) are rarely systematically documented in clinical practice [49], making reliable data collection impractical. Parents took the survey in the HPVVaxFacts mobile web app and were allowed to select their top concerns. The research team categorized the top 3 concerns and summed the concerns by question. The summed scores were sorted from highest to lowest, with the top 3 values being the top concerns.

One month post intervention (October 2023), we held a 1-hour consultation (ie, an engagement strategy) with 2 clinic providers and 4 staff members over lunch to gather informal feedback on study protocol and implementation. Trained clinic staff coordinated the date and time of the consultation. The consultation explored ways to improve the study protocol and implementation procedures, and perceived barriers and facilitators to implementing HPVVaxFacts in routine clinic practice. This session was not audio-taped nor formally analyzed as qualitative research; instead, notes were recorded by coauthors JC-E and ES.

Overall, providers perceived that the program offered clinic benefits. Offering education preclinic visits to their patients was a top benefit. This complemented their recommendations and lessened clinic time for education. Another perceived benefit was the increased vaccination rates in the clinic. Regarding barriers, providers and clinic staff highlighted the need to improve logistics for large-scale implementation. Because the intervention was manual and required efforts from clinic staff and the research team to aid in recruitment and retention, providers questioned the feasibility of this study without full integration into the EHR but still perceived offering preclinic information in general as feasible. However, providers and staff stated they understood this was a pilot study and hoped their suggestions improved future implementation.

We met our goal of training 2 clinic staff to aid in recruiting participants through Phreesia. For this study, the clinic staff sent potential parents an IRB-approved REDCap link with a screener. If they screened eligible, they received the presurvey, viewed the intervention, and took the postsurvey. Of the 194 assessed for eligibility, 63 were ineligible. A total of 21 (16%) declined to participate and 9 more later refused, increasing the refusal rate to 23%. We recruited 110 participants who consented and enrolled in the study (107 were recruited from Clinic Site 1 and 3 from Clinic Site 2) with an enrollment rate of 84%. Parents were enrolled 15 to 20 minutes up to 24 hours prior to the clinic visit. In total, 71 people were allocated to the study arms, a 35.5% reduction from enrollment to randomization. A total of 36 parents were randomly assigned to the HPVVaxFacts intervention arm, of which 7 did not go to the mobile web app. Of those who viewed the mobile web app (81%), 2 did not complete the postsurvey. These participants were excluded with a retention rate of 75%. A total of 35 people were randomly assigned to the attention control arm. We were unable to determine whether participants viewed the nutrition education website. However, 2 did not complete the postsurvey with a retention rate of 94% (Figure 2).

The sociodemographic and clinical data by study arm assignment reported by parents demonstrated that most children belonged to the 11‐12 age group with 44% and 50% in the HPVVaxFacts and nutrition study arms, respectively. Children in both groups were predominantly female, non-Hispanic White, and heterosexual. Among parents who participated in the survey, most of them were heterosexual. The mean age for participants in the HPVVaxFacts intervention arm was 41.8 (SD 14.6) years compared to participants in the nutrition attention control arm at 41.6 (SD 7.5) years. In terms of education level, a higher proportion of the parents in the HPVVaxFacts intervention arm had a bachelor’s degree or higher (18/27, 66%) than those in the nutrition attention control arm (11/30, 40%). Additionally, the majority in both arms reported an annual household income of more than US $60,000. In the HPVVaxFacts intervention arm, 59% (16/27) of parents had been offered the HPV vaccine for their child by a provider compared to 70% (21/30) in the nutrition attention control arm. More than half in both arms were iPhone users (Table 1).

HPV vaccine status was obtained for 57 children aged 9‐17 years who were nonvaccinated for HPV prior to their parent participating in the clinical trial. HPV vaccine initiation (ie, those who received the first dose) rates of children whose parents participated in the HPVVaxFacts intervention arm were significantly higher than those of children whose parents were assigned to the nutrition attention control arm (48% vs 17%; Table 2).

We determined the impact of the HPVVaxFacts intervention and nutrition attention control on knowledge, concerns, and intention related to HPV vaccination. Analysis revealed that parents in the HPVVaxFacts intervention arm showed higher improvement in knowledge scores but remained largely unchanged in the nutrition attention control arm with no significant difference in knowledge before or after intervention. Similarly, parents in the HPVVaxFacts intervention arm had a larger reduction in concern toward HPV vaccination compared to those in the nutrition attention control arm. However, the difference in concern scores was not significant (Table 3).

Analysis demonstrated parents had varying degrees of intent (ie, an imperfect surrogate for vaccine hesitancy) to vaccinate their child. Before the intervention, 74% (20/27) of the parents in the HPVVaxFacts intervention arm had the intention of “Very likely” or “Somewhat likely” to get their children vaccinated compared to 60% (18/30) in the nutrition attention control arm. Post intervention, parents who reported intentions (very likely and somewhat likely) were 70% (19/27) in the HPVVaxFacts intervention arm and 63% (19/30) in the nutrition attention control arm. However, when examining pre-post changes in parents’ intentions, 3/27 (11%) parents in the HPVVaxFacts intervention arm and 1/30 (3%) parent in the nutrition attention control arm reported an increased likelihood of vaccinating their children. The pre-post intervention arm and between-arm differences were not statistically significant (Table 4).

Figures 3A and 3B show an exploratory analysis between pre- and postintervention knowledge and concern scores in both study arms. For knowledge in Figure 3A, the estimated slope was 0.76 (95% CI 0.48-1.04) for HPVVaxFacts intervention arm and 0.98 (95% CI 0.78-1.18) for the nutrition attention control arm, with no evidence that the association differed by study arm (P_interaction=.21). For concern in Figure 3B, the estimated slope was 0.70 (95% CI 0.52-0.89) for HPVVaxFacts intervention arm and 0.91 (95% CI 0.69-1.12) for the nutrition attention control arm, with no evidence of a differential association between study arms (P_interaction=.16; Figure 3). In summary, although participants in the HPVVaxFacts intervention arm showed lower postintervention knowledge and concern scores than those in the Nutrition attention control arm, consistent with greater knowledge and fewer concerns based on the scoring of these measures, the differences between the study arms were not significant.

Only the HPVVaxFacts mobile web app allowed parents to take the quiz and confirm or change their top concerns produced via an existing algorithm. Parents in the HPVVaxFacts intervention arm were asked to identify and select their top 3 beliefs or concerns for HPV vaccination on the web app. We determined the frequency with which a concern was selected as a top 3 concern for parents. Based on the web app data, the majority of parents identified the vaccine could cause serious health problems (20/27, 74%), too many vaccines (17/27, 63%), and questioned HPV vaccine safety (12/27, 44%). Refer to Table 5 for a summary of the top 3 concerns selected for the HPVVaxFacts intervention arm participants.

Parents who received the HPVVaxFacts (n=27) had consistently prominent levels of acceptability for the program across all the items. Over two-thirds agreed or strongly agreed that they liked the program (mean 4.1, SD 0.7), the program met their approval (mean 4.2, SD 0.8), and it was appealing (mean 4.2, SD 0.8), and they perceived that parents should use the tool for HPV vaccine education (mean 4.2, SD 0.7). Additionally, parents perceived that HPVVaxFacts was helpful and should be used to discuss the HPV vaccine with their child (mean 4.1, SD 0.7; Table 6).

A total of 19 parents completed the interviews, 9 from the HPVVaxFacts intervention arm and 10 from the nutrition attention control arm. In the HPVVaxFacts intervention arm, the mean age of the parent was 38.9 (SD 5) years. All participants were female (9/9, 100%) and non-Hispanic White (9/9, 100%). For the nutrition attention control arm, all were female (10/10, 100%), with the majority being non-Hispanic White (7/10, 70%) and almost half had an education level of GED or High School Diploma (4/10, 40%; Table S1 on parent sociodemographics in Multimedia Appendix 2).

Parents described 4 benefits from participating in the HPVVaxFacts intervention arm. We summarize these benefits below.

Most parents reported no barriers to using the program. A parent stated,

Three parents highlighted potential barriers that were not program-specific. One barrier was that some parents may lack access to the mobile web app. Another barrier older or not technologically “savvy” parents may have difficulty with web app navigation. A parent stated,

Finally, one parent noted that some parents may not trust their health care provider and the research team or the information provided. It was further recommended to use alternative distribution channels (eg, media, social media, PSA) to reach these individuals.

Overall, providers perceived the program as beneficial for their clinic. Particularly, the top benefit was offering preclinic education visits to their patients. This complemented their recommendations and lessened clinic time for education. Another perceived benefit was the increased vaccination rates in the clinic because of the intervention. Regarding barriers, providers and clinic staff highlighted the need to improve logistics for large-scale implementation. Because the intervention was manual and required efforts from clinic staff and the research team to aid in recruitment and retention, providers questioned the feasibility of recruitment and retention procedures but still perceived offering preclinic information in general as feasible. However, providers and staff stated they understood this was a pilot study and hoped their suggestions improved future implementation.

Our study assessed feasibility, acceptability, and preliminary efficacy of the theory-informed, mobile web app HPVVaxFacts designed to decrease concerns and increase knowledge, intentions, and uptake of the vaccine among children aged 9 to 17 years. This study is one of the first to use a multimethod evaluation to assess mHealth interventions to increase HPV vaccination. More importantly, HPVVaxFacts is one of the first mobile web apps to identify and tailor information to parental concerns while supporting autonomy in both parental decision-making and educating their child using an optional kid’s corner on the web app.

Our study reported feasibility across multiple metrics. The enrollment rate was 84%, and the random allocation achieved 100% recruitment. The retention rates in both study arms were 75% in the HPV program and 94% in the nutrition program. Notably, approximately 4 out of 5 (81%) parents in the intervention arm viewed HPVVaxFacts. These rates are considered good, especially with the sensitive context surrounding HPV vaccination and typical survey response rates. The engagement of clinic staff and parents in the development of the study protocol, a known factor in improving study outcomes (eg, logistics and behavior) [51], partly attributes to this success.

Additionally, the Cronbach α for concerns, knowledge, and intervention acceptability scales were considered reliable, further supporting study feasibility and potential deployment of these scales in a larger study. However, because one clinic had vaccination rates substantially increase prior to study implementation, this underscores the importance of conducting a feasibility assessment at multiple checkpoints before and during study implementation. Furthermore, both providers and participants highlighted the logistical barrier of study steps being manual, a factor which may have affected response rates at different steps in the research process. This supports the need to automate these steps for future efficacy trials to improve recruitment and retention.

Interestingly, parents identified factors that could influence use of HPVVaxFacts long-term in research and in clinical practice. For protocol application, improving the study logistics was commonly mentioned by both parents and providers, though not reflective of app functionality. This suggests that the tool alone could be applied in real-world settings with minimal human involvement, a major advantage for sustainability. Similar to past studies [52], parents identified limited access to technology (eg, mobile web app) as a potential barrier reflecting the “digital divide” in America [53]. Some parents, particularly older parents or non–English-speaking parents, may struggle to navigate the mobile web app, which has been identified as a barrier previously [54]. Some parents may mistrust a provider who delivered resources because of broader mistrust in health care. This demonstrates how mistrust can extend beyond the individual to tangibles perceived to be associated with the individual [55]. These findings suggest that feasibility of HPVVaxFacts can be improved using multilingual languages and addressing the digital divide (eg, physical infrastructure expansion, training, and tech support). Also, alternative strategies for providing health care information should be considered, including multiple dissemination channels and provider-led initiatives to build support and trust.

Acceptability was considered good with a rate of 74% or higher with room for improvement. Of interest, acceptability of HPVVaxFacts centered around the ability of the app to tailor to the parents’ needs. Particularly, parents were able to access the app when available (previsit information), identify their top vaccine concerns, and view the modality that best served their need to get their concerns addressed. This suggests the importance of a multilevel, tailored approach that is cost-effective to target parent decision-making processes for vaccination, creating a unique opportunity for this tool to have broad reach. Other acceptable components of the mobile web app, though deemed unsurprising, were its comprehensible content, ability to navigate, and being time-friendly for use. Previsit information has been supported by past studies [26], but its use in the context of a mobile app has been underexplored. Parents lauded it for reasons such as lessened time in clinic or improved preparation for the visit, an acceptable strategy we add to the literature. Our findings also support that increased acceptability of mobile web apps for parent-child discussion could be achieved via provision of parental autonomy to share select information or allow parents to use information to guide a discussion with the child without direct access. This should be further explored as mHealth continues to evolve to expand the reach of health information for caregivers and/or their children.

Not only did parents view HPVVaxFacts to improve feasibility through content being multilingual, but it was also viewed to increase acceptability. Particularly, the suggestion to expand this app content to different languages and its use in alternative settings that provide vaccination to increase acceptability reflects the altruism of these parents. Walker et al [56] found that parents with higher interpersonal altruism had higher odds of agreeing with HPV-specific state vaccine requirements, a confirmation of our qualitative findings. These findings suggest the need for inclusivity when developing strategies and tools that promote HPV vaccination among children. Exploring the app’s use in alternative immunization settings supports the HPV National Roundtables’ goal to increase HPV vaccination in these settings.

Similar to other mHealth studies [26], HPVVaxFacts increased HPV vaccine initiation rates among children at significantly higher rates compared to the nutrition program. Interestingly, this increase occurred despite no significant differences in knowledge, concerns, or intentions between the study arms. Because knowledge and attitudes had a positive trajectory through the increase in effect size in the intervention arm compared to the control arm, this could reflect the pilot study being underpowered for these secondary outcomes. This is a common outcome, as many pilot studies are powered for testing feasibility and not proving an intervention’s efficacy [57]. Also, it remains unclear whether, and to what extent, this effect was mediated through provider engagement or indirect engagement via app delivery through the EHR. Nevertheless, our qualitative findings support that parental willingness to get vaccinated was due to receipt of the information from the provider. Because parents’ most trusted information source [55,58] is providers and their recommendation is critical for uptake [59], future studies should explore in-depth whether and how provider communication modalities can be leveraged to complement provider recommendations and communication in-clinic with a goal to increase HPV vaccination compliance. Finally, these findings could also reflect a ceiling effect in which a significant portion of participants in both the HPV and nutrition programs already had positive intentions to vaccinate [60]. Future studies could either recruit parents who are “not likely” or “somewhat likely” only or use baseline status to stratify analysis, both strategies to better capture intervention effects [61]. Finally, it was not surprising that the top 3 concerns identified by these parents centered around vaccine safety, the leading barrier to vaccine hesitancy and suboptimal rates [62].

First, this study was conducted in 2 pediatric urban clinics in Tennessee in which the primary clinic for recruitment was predominantly female and non-Hispanic White. Collectively, this may limit generalizability to other populations, including those living in different geographical regions and settings along with racial and ethnic groups. Second, selection bias may have occurred if some parents with low mobile app literacy or who did not have a mobile app did not participate, which should be explored in future studies. However, we have begun to take steps to improve inclusion through the provision of HPVVaxFacts in Spanish. Third, this study had a small sample size, which limited our ability to detect a difference for some of the study outcomes (eg, concerns and intention) and whether those outcomes are a result of clinic demographics and protocol. Also, the small number of parents recruited at Clinic Site 2 made it impossible to assess whether clinic-level differences influenced results. Fourth, there were updates on the app during the feasibility study; however, study implementation metrics, including recruitment and retention, improved over time. Fifth, because parents agreed to be in the study and the invite mentioned HPV vaccination, parents may be more engaged or interested in receiving information. This should be explored in a larger study.

Sixth, some parents were lost due to their child already being vaccinated during study participation and study logistics, in which we could not recruit additional participants due to the study duration ending at both clinics. However, this reflects EHR interoperability, a commonly cited issue confirmed by clinic staff [63]. Seventh, we only studied the impact of HPVVaxFacts on initiation rates. The impact on completion rates (ie, 2 doses of HPV vaccine among children younger than 15 years old and 3 doses among those older than 15 years old) [64] will be explored in a future study, as we are exploring reasons for noncompliance and informational needs of parents to inform the update of messaging on the mobile web app to be more comprehensive. Eighth, for the interviews, there may be potential for recall bias, which increases due to the variable time between parent participation and the interview due to demanding study logistics. Ninth, some mobile web app components (ie, average session length) could not be tracked due to financial constraints. However, it is currently being updated to address this limitation. Tenth, small discrepancy between the HPVVaxFacts website content and knowledge scale could have led to a lack of significant improvement in knowledge. We were also unable to track participants’ engagement within the website that may have varied from full review to partial or no interaction with the content. It is possible that the nutrition education within the attention control arm could have influenced parents to accept and/or obtain the first dose of the HPV vaccination for their child. Finally, the nutrition attention control arm did not select their top concerns as this was not an option on the app. Therefore, we are unable to compare parental top concerns by study arm.

Our pilot study suggests HPVVaxFacts is feasible and acceptable to provide tailored information to parents’ preclinic visits to aid parent decision-making for HPV vaccination and ultimately increase uptake. This study’s outcomes, both unclear yet offer a positive trajectory, lay the foundation for a full RCT to test the efficacy of HPVVaxFacts being offered preclinic visit to increase HPV vaccine initiation and completion rates. Future studies can explore the mediating (eg, provider recommendation and communication) and moderating factors for successful intervention effects with a large sample size. Findings also reinforce that provider modalities such as mHealth are acceptable and can complement provider communication to address parental HPV vaccine concerns.