Evaluating public service outcomes
Designing and running evaluations to assess public service outcomes
The challenge
Without systematic evaluation, change programs will rarely deliver the intended benefits
- Orchestrating and implementing change in public services is challenging and costly.
- It is crucial that investments in new pathways and information technologies deliver the returns expected in the business case for change and these returns are experienced by consumers and public service professionals.
- When one change cycle completes, it is tempting to move on to the next, but the harsh reality is that without systematic evaluation, learning and improvement, change programs will rarely deliver their intended benefits.
Combining formative and summative evaluations enables lessons to be learnt both during and after the change
Formative evaluation starts at the outset of the program and seeks to learn lessons ‘in flight’ rather than wait to make improvements at the end. The approach ensures constant reference to the goals of the project and enables careful observation of unforeseen challenges and opportunities.
Summative evaluation considers and compares the realised benefits with those that were expected. It frames a ‘before and after’ picture and guides interventions to improve the marginal gains.
System leaders and change sponsors need to be ambitious and focused on their approach to evaluation. This is not to provide the evidence for criticism, but rather to maximise the significant marginal benefits that are gained from learning lessons.
Structured evaluations enable system leaders to address serious questions about the change program and the intended outcomes, such as:
- What is the return on investment?
- Who is benefiting from that ‘return’? Are consumers recognising a better service experience and are outcomes improving?
- If it is too early to observe long term improvements in outcomes, what indicators do we have that we are on route to seize these benefits? Are public service processes more integrated?
What we do
We conduct formative and summative evaluations that assess the outcomes and costs of public service investments
All major investments in service and system change need systematic evaluation to optimise the outcomes for public service consumers, populations and professionals.
Such evaluation needs to be carefully planned so there is appropriate combination of summative and formative evaluation.
We work with system leaders and program sponsors to focus evaluations on the issues that matter most rather than to attempt to learn every lesson.
Our evaluation focus evolves through the lifespan of the program
- As the program progresses, the focus of the evaluation needs to evolve.
- When the program starts, we are most interested in ensuring that we have the right resources in place to run the service and that the intended users are finding their way into the service.
- As the program progresses, we become increasingly focused on ensuring the activities and outputs are being carried out in the right way.
- Finally, our focus moves to understanding how well the program is leading to the achievement of the desired outcomes and impact.
Our evaluations explore leading indicators that give confidence in the improvement trajectory
Our approach is to confront the hard questions that need to be asked and creatively structure the evaluation, engage the stakeholder, collect the evidence and analyse the data.
Improvements in population outcomes take time to be realised, but our evaluations seek to explore the leading indicators of outcome improvement.
All programs seek to maximise beneficial outcomes from investments. We take care to understand these outcomes, test their validity and consider the evidence needed to give confidence about the improvement trajectory.
Get in touch to discuss how we can support you to evaluate your public service projects and programs
Jay Rebbeck
