Evaluation strategies for measuring pneumococcal conjugate vaccine impact in low-resource settings

ABSTRACT Objectives: Pneumococcal conjugate vaccines (PCVs) are effective in reducing pneumococcal disease. We measured 13-valent PCV (PCV13) effect on different pneumococcal outcomes using diverse studies in Lao People’s Democratic Republic. Methods: Studies included: pre-PCV13 population-based record review of hospitalized childhood pneumonia cases; acute respiratory infection (ARI) study post-PCV13 to demonstrate effectiveness (VE) against hypoxic pneumonia; invasive pneumococcal disease (IPD) surveillance in all ages (2004–2018); carriage studies in children hospitalized with ARI (2013–2019); community carriage surveys pre- and post-PCV13. Results: Annual pneumonia incidence rate in children pre-PCV13 was 1,530 (95% confidence interval [CI] 1,477–1,584) per 100,000. Adjusted VE against hypoxic pneumonia was 37% (95% CI 6–57%). For IPD, 85% (11/13) of cases were due to vaccine-types pre-PCV13, and 43% (3/7) post-PCV13 in children aged <5 years; for ≥5 years, 61% (27/44) and 42% (17/40), respectively. For ARI cases, adjusted VE for vaccine-type carriage was 39% (95% CI 4–60) in <5 year olds; slightly higher than community surveys (23% [95% CI 4–39%] in 12–23 month olds). Conclusions: Despite limited baseline data, we found evidence of PCV13 impact on disease and carriage. Our approach could be used in similar settings to augment existing WHO PCV evaluation guidelines.


Introduction
Pneumococcal disease is a common cause of childhood mortality, with 318,000 deaths estimated globally in young children in 2015 [1]. Most pneumococcal deaths are due to pneumonia, with the highest rates in Africa and South-East Asia [1]. Pneumococcal conjugate vaccines (PCVs) are effective against pneumococcal disease and pneumonia in a variety of settings but there are few published studies from low-or middle-income country (LMIC) settings in Asia [2].
PCV impact studies should be conducted in different countries to provide geographical representation and show effectiveness in specific populations, as serotype distribution [3] and pneumonia etiology may vary. The World Health Organization (WHO) has published guidelines for countries introducing PCV to assess vaccine impact using a standardized framework and different strategies [4]. At least 2 years of preand 5 years of post-PCV introduction surveillance data is recommended to determine the population effects of PCV on IPD, in order to account for secular disease trends [4]. Moreover, the measurement of PCV's impact on childhood pneumonia hospitalizations requires a large sample size of many thousands of admissions each year with a defined catchment to have sufficient power to show evidence of a reduction in disease rates. LMICs may not have robust and complete datasets accessible to undertake this evaluation and establishing surveillance may be costly and resource intensive. It is therefore important to explore other assessment approaches that are feasible in resource-poor settings and with limited baseline data. Advances in implementation science methodology, especially for LMICs, are required. Governments often want to measure the impact of PCV on pneumonia due to the impact of this condition on health sector human resources, government budgets and child mortality. Measuring impact of PCV on vaccine-serotypes demonstrates evidence of a biological effect specific to the vaccine. We have been undertaking PCV13 impact evaluation studies in Lao People's Democratic Republic (Laos) since 2013 with the Laos Ministry of Health (MoH) and partners. Here, we describe methodological approaches to evaluate PCV13 impact on disease and serotypes in Laos as a case study for other countries. The Laos MoH introduced the 13-valent PCV (PCV13) in October 2013 [5]. The MoH requested a PCV13 impact evaluation as Laos was undergoing transition from Gavi financing, and they required local evidence of impact particularly given the lack of available data from similar settings in the region. An impact evaluation was not able to be undertaken as per the WHO guidelines as there were no baseline data and vaccine implementation was imminent. In this context, we designed and implemented a series of studies to evaluate the impact of PCV13 in Laos on childhood pneumonia and circulating pneumococcal serotypes.

Burden of hospitalized pneumonia
We undertook a population-based study to estimate the incidence of hospitalizations due to severe pneumonia among children aged 2-59 months. We conducted a retrospective review of children admitted to all hospitals in a defined catchment of the capital city, Vientiane: five central and nine district hospitals in the pre-PCV13 (2011-2013) period. Vientiane was the only site in Laos with sufficient case numbers to undertake a population-based study. We determined the percentage of all admissions due to pneumonia, clinical features and the percentage with hypoxic pneumonia (see supplementary material for full methods).
The 2013 WHO case definition was used to define pneumonia [6,7]. Data to determine all-cause and pneumonia hospitalization numbers for children aged 2-59 months were extracted from pediatric admission books for central hospitals and from hospital administrative data for district hospitals. At central hospitals available individual medical records were reviewed to ascertain pneumonia severity and outcomes. Admission books were not available for all facilities for varying durations. To account for missing data, we assumed that the number and pattern of pneumonia and allcause hospitalizations were stable within hospitals throughout each three-year period (see supplementary material for assumptions).
Data from admission books and administrative data were used to estimate the annual incidence of hospitalized pneumonia in the pre-PCV13 period. For incidence calculations, patients with a known residential address within the defined Vientiane Capital catchment area, and a percentage of those with missing data (based on preexisting data), were included in the numerator from central hospitals. All admissions to district hospitals within Vientiane Capital were included in the incidence calculations for hospitalized pneumonia. The denominator was based on population statistics and annual population growth data from the MoH.

Vaccine effectiveness against hypoxic pneumonia
We undertook a prospective hospital-based acute respiratory infection (ARI) study in children 2-59 months admitted to any pediatric or intensive care ward from Monday to Friday in a single hospital (Mahosot Hospital) in Vientiane to determine the effectiveness of PCV against hypoxic pneumonia. This hospital admits approximately 400 childhood pneumonia cases annually. This study, using data up until July 2018, has been published [8] and was nested within an ongoing ARI surveillance program which commenced in December 2013 [9]. Participants had information recorded on clinical presentation, potential risk factors for carriage and documented PCV13 vaccination status. Children were classified as vaccinated if they had received ≥2 doses of PCV13 (for those aged 0-11 months), or ≥1 PCV13 dose (for those aged ≥12 months) [10]. Children were considered undervaccinated if they did not meet these dose criteria [11].
A test-negative design was used with hypoxic pneumonia (test-positive) defined as pneumonia (2013 WHO definition [6]) with an oxygen saturation of <90% in room air on admission, or requiring oxygen supplementation during hospitalization [8], and all other non-hypoxic enrolled pneumonia cases were the hospital controls (test-negative). To determine the PCV13 vaccine effectiveness (VE) against hypoxic pneumonia compared to non-hypoxic pneumonia admissions, we used odds ratios from a logistic regression model [VE = (1-OR) *100]. Propensity score matching using inverse probability of treatment weighting was used to adjust for confounding variables and multiple imputation was used to handle missing data. An additional analysis to estimate PCV13 VE against a control condition of total pneumococcal carriage was also conducted. A sample size of 256 cases and 511 controls was determined to have 90% power to detect a VE of 40% assuming that 60% of controls were vaccinated and a 2:1 control-tocase ratio [8].

IPD surveillance
Invasive pneumococcal disease (IPD) surveillance was commenced at Mahosot Hospital, Vientiane in all age groups in 2004 by the Lao Oxford Mahosot Hospital Wellcome Trust Research Unit (LOMWRU) [12,13]. Blood cultures, lumbar punctures and/or other clinically relevant samples are taken with consent from patients (or their families) admitted with suspected sepsis and meningitis in all age groups who enrolled in the surveillance [12]. A case of IPD was defined as isolation of pneumococcus from a normally sterile site [14]. A case of pneumococcal meningitis was defined as laboratoryconfirmation by culture, or identification (i.e. by Gram stain, antigen detection methods) of pneumococcus in the cerebrospinal fluid (CSF) or blood in a child with a clinical syndrome consistent with meningitis [14]. IPD isolates were sent to the Murdoch Children's Research Institute in Australia for serotyping by latex agglutination/Quellung reaction [15,16]. IPD case numbers were summarized by year and serotype (PCV13 vaccine type [VT]/non-vaccine type [NVT]) for children <5 years and older individuals. Case numbers were compared between the pre-and post-PCV period

Hospital-based pneumococcal carriage surveillance
A hospital-based observational study was conducted using clinical and pneumococcal carriage data [17] from children 2-59 months of age admitted with ARI (fever and one or more respiratory symptoms) as part of ongoing hospitalbased surveillance [9]. Data for children admitted to Mahosot Hospital with ARI, using recruitment methods as previously described for hypoxic pneumonia, were used for children enrolled from December 2013 to June 2019 [17]. Study staff collected data from medical records and immunization registers for enrolled children.
Nasopharyngeal swabs were tested by lytA qPCR to screen for pneumococcal carriage and determine pneumococcal load. Samples that were positive (or equivocal) by lytA qPCR were cultured on selective agar (horse blood agar containing 5 µg/ ml gentamicin). Samples with alpha-hemolytic growth had DNA extracted, and molecular serotyping performed by microarray [18].
To demonstrate trends in VT carriage among vaccinated and under-vaccinated children over time, moving carriage prevalence rates within 7-month rolling intervals were calculated using a direct adjustment method [17]. Associations between both VE and village-level PCV13 coverage and VT carriage were estimated using generalized estimating equations, adjusted for potential confounders. VE was calculated as one minus the odds ratio of PCV13 carriage in vaccinated vs. under-vaccinated children, as previously defined [17].

Community pneumococcal carriage surveys in children
Two cross-sectional community nasopharyngeal carriage surveys were conducted and have been published [19]. These were undertaken at the start of PCV13 introduction (November 2013 -February 2014) and two years following (November 2015 -February 2016) introduction to assess VT and NVT serotype changes among healthy children aged 12-23 months and infants too young to be vaccinated (aged 5-8 weeks). A convenience sample of healthy infants and children were recruited from urban and rural immunization clinics. Both surveys were performed over the same months in the relevant years to account for seasonality [19].

Burden of hospitalized pneumonia
In the pre-PCV period, 2011-2013, hospital admission books were available from three of the central hospitals for the entire period (Table 1). Administrative data were available for eight out of nine district hospitals. Four district hospitals (Pak Ngum, Santhong, Naxaithong and Hadxaifong District Hospitals) were included in the calculations of the percentage of all-cause hospitalizations due to pneumonia, and in annual incidence calculations of hospitalized pneumonia in Vientiane. Four district hospitals (Sikkotabong, Sisatanak, Xaysettha and Chanthabouly District Hospitals) had no pneumonia hospitalizations recorded for children <5 years for 2011-2013. Based on assumptions using identified data (see supplement), we estimated a total of 18,687 all-cause hospitalizations with 3,801 (20%) due to pneumonia over 3 years (Table 1). The district hospitals contributed very few numbers to the total allcause hospitalizations (n = 1,154, 6%) and total pneumonia hospitalizations (n = 165, 4%).
In the post-PCV13 period, 5% (95% CI 3-8) of infants (n = 502) and 20% (95% CI 16-24) of children (n = 507) carried a PCV13 serotype. Although there was no significant reduction between the two surveys in VT carriage rates in the infant age group, there was a 23% (95% CI 4-39%) reduction in the carriage of VT serotypes in the toddler age group. There was no significant change in NVT carriage in either age group [19].

Discussion
Using a variety of methods and limited preceding baseline data, we were able to demonstrate that PCV13 had direct and indirect effects on pneumonia, and pneumococcal vaccine-type carriage and disease in Laos. Combined, the results from the different studies provide sufficient evidence of the impact of PCV13 introduction in the context of limited baseline data.
In LMICs, determining the effect of PCV introduction on nonspecific outcomes such as pneumonia using impact or VE studies is challenging. Advances in implementation science methodology, especially for LMICs, are needed. Countries often rely on routinely collected administrative data on pneumonia hospitalizations to document the pneumonia burden before and after PCV introduction. There are few published studies demonstrating the use of administrative data in LMICs to assess the impact of PCV. A study from Rwanda [20] collected meningitis and pneumonia pediatric admission information retrospectively from hospital registers and a meningitis surveillance database, demonstrating feasibility of this approach with appropriate data. Similarly, in Brazil the use of administrative data was shown to be a feasible approach to document a decline in pneumonia hospitalizations post-PCV10 introduction [21]. In preparation for a PCV impact evaluation in Fiji, an audit was undertaken to assess the completeness and consistency of routinely collected hospital admission data [22]. The audit found that completeness of admission data and coding was high, and suitable for impact evaluation, although consistency of complete reporting varied, with the highest rates in tertiary hospitals using an electronic database. Missing data is common in routine administrative data. A study from Fiji using administrative data imputed missing data by multiple imputation [23].
Care needs to be taken with using routinely collected data, however, given that admission criteria and clinician practice may change over time. We have previously found [24] that the change in WHO pneumonia case definition resulted in an apparent reduction in the percentage of all-cause pneumonia hospitalizations attributable to severe pneumonia, as well as a lower annual incidence of severe pneumonia hospitalizations, when comparing the 2013 definition with the 2005 definition. This has implications for the interpretation of pneumonia intervention impact evaluations [24]. Missing data may also affect study results and effectiveness estimates. Despite having missing data in Laos, we were able to estimate, for the first time in this setting, the incidence of hospitalized pneumonia using local administrative data. We also demonstrated how common pneumonia is as a cause for all childhood hospitalizations and the burden in the youngest age groups. This information provides important supportive evidence to Estimated data: accounted for missing data from hospital admission books using assumptions based on a number of consistent intra-hospital and interhospital patterns in monthly and annual admissions from the available records. It was assumed that the number of pneumonia and all-cause hospitalizations followed the same pattern within hospitals throughout the threeyear period. N/A: not available recommend the ongoing use of PCV in Laos. However, we were not able to do a hospital administrative review for pneumonia cases post-PCV with a PCV impact evaluation as we had insufficient power to undertake this; a simulation analysis based on pre-PCV13 data suggested that with the case load in the population selected, we had only 63% power to detect a 20% decline in pneumonia admissions in the 3 years post-PCV13, using an interrupted time series analysis [25]. It is important to undertake power calculations prior to embarking on pneumonia impact evaluations as there is a danger of drawing erroneous conclusions if the study is undertaken but underpowered [26].
Case-control studies are often used to evaluate vaccine effectiveness following vaccine introduction. There are however a number of challenges and limitations when conducting these types of studies [27,28]. Using cases and controls Table 2. All-cause hospitalizations (n = 9,674) and severity classification and clinical characteristics of pneumonia hospitalizations with medical records (n = 931) by age group, for children aged 2-59 months in central hospitals, Vientiane, 2011-2013. Includes patients discharged for the following documented reasons: discharged against medical advice, child remained unwell or in severe condition, evidence of severe pneumonia at time of discharge or in preceding 24 hours (cyanosis or hypoxia, oxygen use at time of discharge, severe respiratory distress, inability to drink, lethargy or reduced consciousness or convulsions) admitted to a single hospital and a novel approach using a test negative design, we were able to determine the effect of PCV on pneumonia in a low-resource setting by determining PCV13 VE against hypoxic pneumonia compared with nonhypoxic pneumonia admissions in a single hospital [8]. This was the first study in Asia showing PCV VE against hypoxic pneumonia. It employed simple methodology and is feasible to be undertaken in other LMICs using a single hospital. To adjust for confounding we used propensity score analysis to balance covariates between exposure groups, which strengthened internal validity of results. These results are important as child mortality in Laos is one of the highest in the region, pneumonia is common, and hypoxic pneumonia is the usual cause of death. Thus, this vaccine is likely to contribute to improved child survival in Laos. In addition, PCV introduction has been found to coincide with a decline in all-cause pneumonia mortality which is an added benefit [29]. IPD surveillance has been ongoing in Laos since 2004, however the reported IPD numbers are few. These numbers are likely to be a considerable underestimate of the true disease burden in the population, as there are few available bacteriology laboratories, and challenges to accessing health care [30] with antibiotics frequently given prior to hospital admission [31]. The surveillance is also sentinel at one hospital in the capital city. This makes it difficult to draw any firm conclusions from the data, although PCV13 serotypes have declined since PCV13 introduction. There are a number of obstacles to establishing robust surveillance; these include both the cost and difficulty in starting up and maintaining standardized clinical and laboratory surveillance practices [32].
The impact of PCV on VT serotypes in IPD can be augmented using carriage studies. A study from Cambodia [33] using data from six outpatient clinic carriage surveys and a hospital-based IPD program demonstrated a VE of 39.2% (95% CI 26.7-46.1%) for VT carriage and a 42.3% (95% CI 67.3 to −1.5%) decline in VT IPD. Combining colonization and IPD results, translated into a VT IPD decline of 36.3% (95% CI 23.8-46.9) [33]. In Laos, using data from two crosssectional carriage surveys, we demonstrated impact on vaccine-type nasopharyngeal carriage in children aged 12-23 months, but no indirect effects in infants too young to be vaccinated. However, given the post-PCV13 survey was undertaken two years following vaccine introduction, this may be too early to show indirect effects [19]. Infants too young to be vaccinated are at high risk of pneumococcal disease, especially meningitis. Carriage surveys allow us to measure the impact of PCV13 in this young age group and reflect herd protection [34]. It is assumed that a reduction in pneumococcal carriage would translate into a reduced incidence of pneumococcal disease. Vaccine schedules, immunization coverage and catch-up campaigns play a role in producing indirect effects in a population [35]. Catch-up campaigns accelerate indirect PCV effects through the reduction of VT carriage in toddlers who are the major reservoir for VT transmission to unvaccinated and undervaccinated groups [36]. Laos had a limited catch-up campaign to 12 months of age which relied on children being brought to the health center (for 9 month immunization visits and other non-immunization visits). Uptake was therefore limited in the older age group, allowing for continued circulation of VTs in the community. For VT pneumococcal carriage to be interrupted in the community with a suboptimal catch-up campaign requires approximately 3-4 years with vaccination of multiple birth cohorts.
In our hospital surveillance program, vaccinated ARI cases had lower VT pneumococcal carriage rates than undervaccinated cases. In contrast total pneumococcal carriage rates were unchanged, which is to be expected as VTs decline while NVTs increase following PCV13 introduction [37]. While we did not observe indirect effects in infants from the community carriage surveys, our hospitalbased carriage surveillance indicates increasing indirect effects as vaccine coverage increases -with reductions in VT carriage from 20% to 12.8% among ARI cases as PCV13 coverage increases from 0-60% four years after PCV introduction.
PCV causes an increase in disease from NVT serotypes (serotype replacement). The extent to which replacement occurs in LMICs, especially in Asia, is not well described. It is also unclear whether replacement disease could threaten pneumococcal disease control. Given the challenges with conducting robust IPD surveillance in LMICs, it is important to show whether pneumococcal carriage can predict replacement disease in these settings.
The strength of evidence for PCV impact provided by different study designs, study endpoints and sample types are variable. Although national comprehensive IPD surveillance is considered the 'gold standard,' IPD is a rare event and case identification is usually through culture-based detection methods which results in the under-detection of cases. Detection may be improved through the use of molecular methods but often this is not available. Pneumonia is the most common cause of pneumococcal disease mortality, however undertaking pneumonia surveillance to determine the population effect of PCV is challenging and resource intensive. We present an alternative implementation science method using the test negative design that despite not demonstrating the population effect of PCV, shows that PCV is effective against the most severe form of pneumonia, hypoxic pneumonia, using a single hospital approach. Additionally, to demonstrate a biological impact of PCV on circulating pneumococcal serotypes, monitoring pneumococcal nasopharyngeal carriage in children with pneumonia or healthy people from the community may be a useful and relatively inexpensive way to monitor vaccine impact in children and demonstrate the indirect effects in adults and other age groups.
Laos introduced PCV13 in a 3 + 0 schedule in 2013 based on the WHO recommendations for introduction strategies in LMICs which were in place at this time. This schedule has previously been shown in other countries to have indirect effects on vaccine-type IPD when introduced as part of a national campaign. The described methods could be used to augment existing WHO PCV evaluation guidelines for countries with limited resources and baseline data, and imminent vaccine introduction [4]. This is important for LMICs, and countries in the Asian region with limited access to resources to establish surveillance and undertake impact evaluations. However, further work is required to understand how to measure and monitor replacement in LMICs in the post-PCV era with imperfect surveillance systems.
Our evaluation provides important data to support the ongoing use of PCV in Laos as it was used for evidencebased decision-making by government decision makers to continue to fund the continuation of PCV13 as Lao PDR transitions from Gavi support. It demonstrates how vaccine impact can be measured using methods that do not require preexisting surveillance. Most program components are relatively simple and could be applied to document vaccine impact in other LMIC settings without complex laboratory-based studies. We do not recommend that these methods replace WHO guidelines but augment them for countries that are unable to undertake traditional evaluations. On their own, some components, such as the retrospective record review, did not have sufficient power to demonstrate a population effect on allcause hospitalized pneumonia but this could be alleviated in other settings with higher case numbers. However, in conjunction with other supporting data, the impact of PCV13 in this setting could be determined and has provided sufficient evidence to support the ongoing use of PCV in Laos.

Conclusions
Despite limited baseline data in Laos, we found adequate evidence of PCV13 impact on disease and vaccine-type serotypes using a combination of studies. Our evaluation approach could be used in similar settings to augment existing WHO PCV evaluation guidelines and document vaccine effects.

Declaration of interest
C von Mollendorf, C Satzke, EM Dunne, CD Nguyen and KE Mulholland are investigators on an unrelated collaborative research project funded by Pfizer. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript.

Reviewer disclosures
A reviewer on this manuscript acknowledges receiving consulting fees from Merck Sharp & Dohme, consulting and speaking fees, and funds for unrestricted research grants from Pfizer, and funds for unrestricted research grants from GlaxoSmithKline. All these fees and funds have been paid directly to her/his home institution. Peer reviewers on this manuscript have no other relevant financial or other relationships to disclose.

Author contributions
RL, MC, VS, KV, JYRL, SP, KAM, AG, MM, RP, SSD, KF, PNN, KEM, DABD and FMR supported the development of country-specific protocols and study implementation. CS and EMD devised the microbiological approach and protocols. BDO managed and conducted microbiological testing. CvM, RL, JYRL, JC, EMD, RW, CDN and CS conducted the analysis. CvM and RL drafted the manuscript. All authors provided feedback to the draft manuscript and have read and approved the final version.

Funding
This study is funded by Gavi Alliance (PP34411213A1) and the Bill & Melinda Gates Foundation (OPP1115490), with support from the Victorian Government's Operational Infrastructure Support Program. FR was supported by a NHMRC Early Career and TRIP Fellowships. CS was supported by a NHMRC Career Development Fellowship (1087957)