aes22 International Evaluation Conference

Naveen Tenneti (Victorian Department of Health), Naomi Bromley (Victorian Department of Health)

The rollout of COVID-19 vaccines has progressed at a pace and coverage that is unprecedented in the history of vaccination programs. In Victoria, as of 13 March 2022, 80.9% of the population has had two doses and 61.6% of the eligible population has received a third dose. This translates to over 14.5 million doses administered in just 12 months, 41% (6 million) of which has been delivered by the State system, at a rate of approximately 38,000 doses per day.  

From the outset, robust evaluation and continuous improvement methodologies were embedded in the Victorian Department of Health's COVID-19 vaccination program. A key feature of the evaluation approach was a rapid review platform which involved critical appraisal of 'work in progress' efforts and rapidly identifying practical recommendations for improvement. This work was supplemented by a knowledge management structure supporting evidence synthesis and insight, as well as systems for analysis of consumer sentiment and feedback.

The summative evaluation of the program explored equity and access, trust and confidence, and safety and quality. It employed a mixed methods approach including document review, data monitoring and stakeholder interviews. Three key stages of the program were identified to frame the evaluation based on vaccine demand, vaccine supply and delivery capacity; high demand:low supply, high demand:high supply and low demand:high supply. These stages are also characterised by external influences including COVID-19 case numbers, vaccine mandates, and vaccine safety signals.

Alongside, an overview of the evaluation framework and the evidence appraisal and rapid review methodologies, practical recommendations regarding these methods' utility in evaluation of future public health programs will be discussed.  The approach provides a useful case study for funding bodies and policymakers on how evaluation can be prioritised and integrated into program design, even during high-pressure environments such as a pandemic.