Article Text
PDF
Abstract
Previous studies have demonstrated that text message reminders can improve pediatric vaccination rates, including low income & diverse settings such as those served by federally qualified health centers. In this study, we aimed to improve compliance with routine childhood immunizations via a text message intervention in a network of urban, federally qualified health centers at a large academic medical center. We targeted parents or guardians of children aged 0-2 years who were overdue or due within 14 days for at least one routine childhood immunization without a scheduled appointment. In Round 1, two versions of a text were compared to a control (no text). In subsequent Rounds, a new text was compared to a control (no text). In each round the content, wording, and frequency of texts changed. Subjects were randomized to receive a text (treatment group(s)) or to not receive a text (control group) in each round between 2020 and 2022. The primary outcome was whether overdue vaccines had been given by 12 week follow up. The secondary outcome was appointment scheduling within the 72 hours after text messages were sent. In Round 1 (n=1203) no significant differences were found between groups in overdue vaccine administration per group or per patient at follow up, or in appointment scheduling. In Round 2 (n=251) there was no significant difference in vaccine administration per group or per patient. However, significantly more patients in the intervention group scheduled an appointment (9.1% vs. 1.7%, p=0.01). In Round 3 (n=1034), vaccine administration was significantly higher in the intervention group compared to the control overall (7.0% vs. 5.5%, 0.016) and per subject (p=0.02). Significantly more patients in the intervention group scheduled an appointment compared to the control (3.3% vs. 1.2%, p=0.02). We found that text messaging can be an effective intervention to promote health service utilization such as pediatric vaccination rates, which although improved in this study, remain low.
- Healthcare quality improvement
- Implementation science
- Paediatrics
- Randomised controlled trial
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Previous studies have demonstrated that text message reminders can improve paediatric vaccination rates, including in low-income and diverse settings such as those served by federally qualified health centres.
WHAT THIS STUDY ADDS
The study affirms that text message reminders can be effective but shows that the reminders need to be tailored to local populations and needs; even if a message was effective in one setting, it may not be in yours.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE & POLICY
The study shows the benefit of using rapid randomised testing to quickly, cheaply and robustly test multiple variations of patient-facing text messages, allowing efficient tailoring to local communities.
Background
As the COVID-19 pandemic hit New York City in the spring of 2020, many families became homebound due to a citywide lockdown. Families were less likely to bring their children in for healthcare appointments, and many young children did not receive their regularly scheduled vaccinations. A total of 64% fewer immunisations were administered to children aged under 24 months in New York City in April 2020 than in April 2019.1 Although vaccination rates in New York and across the USA approached prepandemic levels once lockdowns were lifted, administration was insufficient to compensate for missed doses, and national vaccination rates remained depressed through the 2020–2021 school year.2 3
Our health system’s federally qualified health centre (FHC) network had a very similar 64% decrease in paediatric vaccine compliance in May 2020 versus May 2019. This amounted to 55% of in-network, under 2-year-old children who were due soon or overdue for a vaccine, without a scheduled appointment in June 2020. On average, each child who was due soon or overdue for a vaccination was missing six different vaccines. Timely vaccines are essential to decreasing the chance of contracting preventable illnesses, underscored by a 2018–2019 outbreak of measles in our service area that affected over 450 unvaccinated children, caused 20 hospitalisations and cost US$8.4 million.4
Previous studies have demonstrated that text message reminders can increase immunisation rates among paediatric populations, including largely racial or ethnic minority populations and urban underserved groups that mirror the FHC patient population.5–12 Furthermore, messaging reminders are the most cost-effective intervention to promote vaccination compliance according to Community Preventive Services Task Force guidelines.13
This project therefore aimed to use reminders to improve vaccination rates and to address a gap in the wider literature by using rapid randomised testing to ensure that any reminders were effective in our local population and for our local context. This project’s primary aim was to increase the rate of early childhood immunisation among children aged 0–2 years at the FHC. A secondary aim was to use rapid iterative testing to identify which text message reminders were most effective for our population.
Methods
The FHC is one of the largest FHC networks in the nation, serving over 100 000 patients regardless of their ability to pay or insurance status. As of 2017, 81% of families served lived at or below twice the federal poverty level, a measure issued annually by the Department of Health and Human Services to define eligibility for benefits such as Medicaid. In 2017, this would have been an income of US$49 200 or less for a family of four.14
In June 2020, FHC leadership and an internal quality improvement team collaborated to promote vaccination compliance for children aged 0–2 years across the FHC network. The quality improvement team consists of a programme manager, statisticians, project coordinators, a clinical informatics analyst and a programme lead, who specialises in running rapid-cycle, randomised tests to evaluate pragmatic ideas and improve healthcare practices.15 16
Intervention
The project team identified text message vaccination reminders as an ideal intervention based on prior literature, the low resource and staff burden required for implementation and suitability for evaluation with routinely collected data and rapid iteration given the evolving and time-sensitive nature of the COVID-19 pandemic. The existing text messaging infrastructure was adapted for the project.
The project took place in three iterative rounds interspersed with qualitative interviews to inform intervention design (figure 1). Prior to the project kickoff, the FHC team established a baseline vaccination compliance rate from which improvements could be monitored. Target children were identified through electronic health record queries. Inclusion criteria for all rounds were age 0–2 years, attribution to the FHC network and due soon or overdue vaccination status for at least one routine childhood immunisation according to the Centres for Disease Control Child and Adolescent Immunisation Schedule, including the catch up schedule, without a scheduled appointment.17 Due soon vaccination status was defined as due for immunisation in the next 14 days. All texts were sent from an anonymous number but with the name of the health system as the first word in the text.
Project timeline. Family health centres.
Before each round, children’s medical record numbers, names, dates of birth, sex, race, ethnicity, language, primary payer, Medicaid number, attributed FHC provider, address, parent or guardian contact information, prior vaccination dates, vaccination status and days until their next vaccination was due for the 10 CDC-recommended early childhood immunisations were extracted. Demographic data at baseline can be seen in table 1. These variables were already captured routinely for regular auditing across the FHC. Subjects were then randomised to treatment arms via a randomisation algorithm in RStudio or SAS V.9.4 (SAS Institute, Cary, North Carolina, USA), and the randomised subject lists were sent to an internal communications team for text messaging outreach. All healthcare workers were blinded to allocation during the study, but patients and analysts were not blinded due to the nature of sending, receiving and tracking text messages.
Demographic information at baseline
Outcomes
The primary outcome measure was the administration of overdue vaccines by 12 weeks after text messages were sent. We defined this in two ways: as the mean number of overdue vaccinations delivered per child and as the percenage of overdue vaccinations delivered per treatment group. The secondary outcome, relating to the process, was subject to appointment scheduling within 72 hours after the text messages were sent. A potential balancing measure could have evaluated patient satisfaction, related to a disturbance from receiving unwanted texts. However, operational leads believed this risk to be low, and in order to adhere to our principle of minimising additional burden on staff, this outcome was not collected.
Statistical analysis
Poisson regression with an offset of the log of the number of vaccines overdue at baseline (to account for patients with different numbers of overdue vaccines) was used to compare overdue vaccines delivered per child between treatment groups. The χ2 tests were used to compare the percentage of overdue vaccines administered and appointments scheduled between treatment groups, respectively. All baseline and outcomes data were extracted directly from the electronic health record, Epic (Epic Systems, Verona, Wisconsin, USA), and analysed in RStudio or SAS V.9.4. Conditions for all tests were checked and satisfied, and two-sided statistical significance was set at 0.05 ≥α. Power calculations conducted prior to round 1 determined that to detect a reduction in patients due for vaccination without an appointment from the baseline rate of 55%–45% would require 261 patients in each group. Later rounds were each conducted with a fixed cohort, and further sample size calculations were not conducted.
During the project period, outcome definitions varied slightly between rounds as priorities shifted and methodology was refined. In round 1, we initially compared the percentages of patients who scheduled an appointment postintervention between groups, and in round 2, we initially compared the percentages of patients whose vaccinations became up to date postintervention. In order to directly compare the impacts of intervention in each round, all analyses were updated post hoc to align with the round 3 approach (with outcomes of appointments scheduled within 72 hours and vaccines administered). Conclusions did not meaningfully change after the update. See online supplemental appendix A for originally specified outcomes by round.
Supplemental material
This project met the criteria for a quality improvement project and thus did not receive institutional review board (IRB) approval, nor were patients asked to provide informed consent. This study was registered on ClinicalTrials.gov, NCT04936776. We followed Standards for Quality Improvement Reporting Excellence V.2.0 reporting guidelines.
Development of tailored texts
Round 1
In July 2020, four short, informal interviews were conducted with parents or guardians of FHC paediatric patients to inform intervention design. Interviewees were asked open-ended questions about how to aid vaccination appointment scheduling and were then asked for feedback on sample texts. The sample text messages either provided education as to the value of vaccination or emphasised the safety of coming to the clinic despite the ongoing pandemic. They also distinguished between these routine childhood vaccinations and COVID vaccinations, anticipating there might be particular resistance to COVID vaccination (online supplemental appendix B). Interviews were conducted in person by community programme representatives in the interviewees’ primary language: English, Spanish or Chinese.
Supplemental material
Interviewees reported that text messages should explain why they should visit the FHC, the importance of vaccination and the security measures that the FHC was taking to ensure cleanliness and safety in the context of the COVID-19 pandemic. When sample text messages were shown (online supplemental appendix B), parents were split between preferring texts 2, 3 and 4. These results reiterated that parents valued messaging regarding both cleanliness and safety and the importance of vaccination. Results were used to design text message A and message B (table 2). Round 1 subjects were identified on 1 January 2021 and randomised to receive message A, message B or no text in a 1:1:1 ratio. Text messages were sent on 9 February 2021.
Text message language, all rounds
Round 2
In mid-April 2021, 20 telephone interviews were conducted with parents or guardians who received round 1 text messages to assess their response and explore text message revisions. Interviews were conducted in the subjects’ primary language (online supplemental appendix C).
Supplemental material
Interview results are summarised in online supplemental appendix D. Most interviewees did not recall receiving a round 1 text message but identified text messaging as an effective means of outreach. They reported that messages should explain the purpose of the message and instructions for scheduling, that more than one reminder message would be useful and that the timing of outreach matters, most preferred evening times.
Supplemental material
Meanwhile, the rapid RCT team assessed the baseline efficacy of text message reminders by comparing a simple text message to no text (table 2). The goal was to assess whether continued revisions were necessary as the impact of the COVID-19 pandemic waned. Individuals who received round 1 text messages were excluded due to the proximity of this round to the previous one.
Subjects were identified on 12 July 2021 and randomised in a 1:1 ratio to receive text message C or no text message. Text messages were sent on 14 July 2021, a quicker turnaround than in round 1 due to workflow streamlining.
Round 3
In the fall of 2022, the rapid RCT team initiated a third round of text messaging (table 2). The design of round 3 texts drew from the results of round 2 interviews and messaging, as well as newly published findings from Milkman et al.5 That project demonstrated that repeated text messages leveraging the principle of loss aversion from prospect theory—or the idea that people prefer to avoid losses than to acquire equivalent gains—can significantly improve vaccination rates.18 Text design harnessed the principle by stating that vaccines had been reserved for recipients.
Subjects were identified on 25 August 2022 and randomised in a 1:1 ratio to receive text messages D and E two days later at noon (intervention) or no text message (control). Text messages were sent on 12 and 14 September 2022.
Appointment and vaccination outcomes data were extracted on 7 December 2022. Since two text messages were sent in this round, appointment scheduling was compared between the two texts and 24 hours after the second text, amounting to the same 3-day follow-up period applied in rounds 1 and 2.
Results
Round 1
1203 subjects who met inclusion criteria were randomised to receive text message A (n=401), text message B (n=401) or no text message (n=401) (table 3). Outcome data were not available for 37 subjects (13 in group A, 16 in group B and 8 in group C), who were excluded from analysis.
Results
No significant difference between groups was found in appointments made within 3 days of messaging (p=0.63). 18/388 (4.6%) of group A made an appointment, compared with 15/385 (3.9%) of group B and 21/393 (5.3%) of the control group (figure 2a). Eight of the 80 subjects who scheduled an appointment booked an appointment date outside the follow-up window: five in the intervention groups and three in the control group. No significant difference was found between groups in the total number of overdue vaccinations delivered within the following 12 weeks (p=0.41), with 3.1%, 3.7% and 3.6% of vaccines delivered in groups A, B and the control closed, respectively (figure 3a). Similarly, no significant difference was found between groups in overdue vaccines delivered per child (p=0.42).
Percentage of subjects who scheduled an appointment within 72 hours of text messaging.
Percentage of vaccination gaps closed per treatment group 12 weeks after text messaging.
Round 2
In round 2, we assessed the baseline efficacy of text message reminders by comparing a simple text message to no text (table 2). The goal was to assess whether continued revisions were necessary as the impact of the COVID-19 pandemic waned. Individuals who received round 1 text messages were excluded due to the proximity of this round to the previous one.
251 subjects who met inclusion criteria were randomised to receive text message C (n=131) or no text message (n=120) (table 3). This round included fewer subjects than rounds 1 or 3 because round 1 subjects were excluded, and the two rounds were conducted in quick succession.
Significantly more subjects in the intervention group (n=12, 9.2%) scheduled an appointment within 72 hours of messaging compared with the control group (n=2, 1.7%, p=0.01; figure 2b). One of the 14 subjects who scheduled an appointment booked a date outside of the follow-up window and was in the intervention group. There was no significant difference in the number of overdue vaccinations delivered between groups at 12 weeks (p=0.34), with 22.6% of vaccinations delivered in group C and 20.4% in the control group (figure 3b). Similarly, no significant difference was found between groups in vaccinations delivered per child (p=0.40). Vaccination administration for this round was overall higher than rounds 1 or 3.
Round 3
1034 subjects who met inclusion criteria were randomised to receive text messages D and E (n=517) or no text messages (n=517) (table 3).
There was a non-significant increase in appointment scheduling 24 hours after message D was sent (p=0.18) and a significant increase in appointment scheduling 24 hours after message E was sent (p=0.02), at which point 3.3% of the intervention group (n=17) and 1.2% of the control group (n=6) had scheduled an appointment (figure 2c). Two of the 23 subjects who scheduled an appointment booked an appointment date outside of the follow-up window, both in the intervention group.
Significantly more overdue vaccinations were delivered in the intervention group (7.0%) compared with the control group (5.5%) at the 12-week follow-up (p=0.016) (figure 3c). Additionally, significantly more vaccinations were delivered per child in the intervention group than in the control group (p=0.02), with 1.27 times more vaccines delivered in the intervention group than in the control group.
Discussion
In this series of randomised controlled tests, we iteratively trialled the design of text message reminders to improve uptake of overdue or imminently due routine paediatric immunisations among children aged 0–2 years visiting a network of urban FHCs. To our knowledge, this is the first randomised controlled trial to iteratively optimise text message reminders for routine childhood immunisation.
While our result affirms prior work demonstrating the feasibility of positively impacting vaccination compliance with reminder messages,5–7 9 10 19–33 the inefficacy of our first two rounds implies that not just any text will work. Instead, text design, including content, timing and repetition, matters, and text messaging programmes should be tested, iterated upon and monitored to best suit local needs and context.5 In other words, no one text message can be assumed to be effective for a given population until tested, even when employing evidence-based design. Even if population demographics are exactly the same, other factors, such as a pandemic, the current rate of vaccination (and therefore who remains to be vaccinated) and wider state policies such as vaccine mandates, will all shape what is most effective, often in ways that cannot be predicted or understood by a priori hypotheses. This conclusion is supported by systematic and narrative reviews that uncover sizeable heterogeneity in the effects of text message vaccination reminders7 8 19 21 and the mixed results of US studies targeting vaccination compliance with text messaging.12 34 35
Given the inconsistent impact of reminder interventions, a few studies have explored message design as a mediator of vaccination outcomes.5 26 36 These studies unanimously show that message design does impact efficacy, in line with our results. However, all focus on influenza vaccination, which families conceptualise differently than often mandatory childhood immunisation,37 and two required very large sample sizes.38 In addition, they texted patients who already had appointments, while we texted patients who were already overdue and therefore likely harder to influence. Our iterative approach offers a more pragmatic alternative for health systems or practices that may lack the reach to enrol thousands of participants.
We learnt several lessons through this course of work that may be helpful to others looking to implement messaging programmes. First, merging insights from multiple sources was beneficial. Round 3 texts, for instance, integrated local qualitative findings from subject interviews with behavioural science, an approach that was not taken in other studies. The former was crucial to understand the attitudes we aimed to influence, while the latter maximised that influence.5 26 36
Second, assembling an interdisciplinary team, including clinical champions, communications specialists, an information management liaison and quality improvement experts, was essential. Clinical champions facilitated qualitative interviews and had the best understanding of clinically meaningful impact. They were also crucial in advocating for resource allocation to the project and, eventually, the change in practice that it recommended. A close tie to communications enabled us to quickly assess the feasibility of different intervention ideas while providing the technical know-how to randomise them.
Last, information management provided rapid access to the most relevant data, allowing alignment of project findings with FHC priorities.
Third, iterative randomisation was an effective and low-resource tool to optimise messaging. Randomisation helped us to be certain about which texts were effective, and because each round could be completed in under 4 months, we were able to look at multiple versions of texts until we were successful. The quick turnaround between iterations also meant that we could adjust our analytic approach to meet changing FHC priorities (online supplemental appendix A). Meanwhile, selecting outcomes routinely collected for auditing meant no extra data were collected from patients, meaning frontline staff had no extra tasks added to their workload while maximising the practical relevance of our analysis. However, several conditions must be met for this methodology to work: the volume of events (here, texts sent) must be high, outcomes must be routinely collected and accessible and the intervention must be randomisable. To our knowledge, we are one of the few sites in the country routinely employing this kind of rapid iterative randomisation.
Lastly, while our final text does show significant improvements in vaccinations given among our target population, children in our system still have gaps in their vaccination requirements. Future research might look to see how this could be further improved, either through additional rounds of randomised testing of texts, potentially tailored to specific communities or further tests comparing timing and deadlines (eg, a pre-emptive ‘your child will become overdue for their vaccine in X days, please schedule now’) or through wider, more resource-intensive work within the FQHC.
Limitations
This project has several limitations. First, the results of each round cannot necessarily be compared with one another. Rounds were conducted at different times of the year, in different years, with different individuals and in varying proximity to the COVID-19 pandemic, which likely impacted attitudes towards vaccination. Second, round 2 may have been underpowered to detect a difference in primary and secondary outcomes, given the smaller sample size and higher starting vaccination administration rate compared with rounds 1 and 3. We suspect that the increased response rate was due to families’ return to healthcare services following the most acute phase of the pandemic. Future iterative trials like this one should consider the possibility of shifting baseline vaccination rates between rounds. Third, our primary outcome, vaccine administration, was assessed at a 12-week follow-up and missed patients whose scheduled appointments fell outside of that time frame; however, few appointments were made outside this window. Fourth, subjects were selected from an urban underserved population in New York City that may not be generalisable to other groups. Fifth, children targeted in this project faced fewer external requirements for vaccination than school-age children (for whom vaccination is required). While daycares and grade schools in New York City have the same vaccination requirements, daycare is not compulsory and some children in this project may not attend.38 Consequently, our interventions may be less effective in increasing vaccination rates among school-age children, whose parents have other extrinsic motivations to pursue vaccination. Sixth, the ability to rapidly iterate in our trial is based on the low cost of our interventions (texts are essentially free to us); this might not be the case for some practices.
Conclusion
Our findings suggest that, while text messaging can be an effective means to increase compliance with routine childhood immunisations, some care is needed in design. In particular, reminders and the use of wording such as ‘reserved’ can increase efficacy. It also shows that repeated testing of different text message designs can be useful. Just because a text message has worked in one setting does not mean it will work in another. We found that leveraging a multidisciplinary team, drawing from behavioural science and local insights and using a randomised study design was helpful in identifying the most useful text message content and timing. Key areas for future exploration and testing include the efficacy of our reminder text messages in different contexts, the sustainability of text messaging impact over time and cost-effectiveness analyses to identify the cost and benefit of text messaging campaigns compared with other interventions.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants, but an Ethics Committee(s) or Institutional Board(s) exempted this study.
References
Footnotes
X @kyra_rosen, @leorahorwitzmd
Contributors KR served as the primary study lead and contributor to this manuscript. Additional study and manuscript support included HK, WK and ST. Data acquisition and analysis were completed by NK and WK. Study support included FM, AB, JS and JA. Study conception and design were led by LIH and PP. LIH is the senior author and guarantor of this paper. She can confirm that all listed authors have contributed to the research and/or manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.