Evaluating healthcare outcomes
Designing and running evaluations to assess outcomes
The challenge
Without systematic evaluation, change programs will rarely deliver the intended benefits
- Orchestrating and implementing change in health services and across health systems is challenging and costly in terms of time and cost.
- It is crucial that the investments in new pathways and clinical and information technologies deliver the returns expected in the business case for change and these returns are experienced by patients, carers and health professionals.
- When one change cycle completes, it is tempting to move on to the next, but the harsh reality is that without systematic evaluation, learning and improvement, change programs will rarely deliver their intended benefits.
Combining formative and summative evaluations enables lessons to be learnt both during and after the change
Formative evaluation starts at the outset of the program and seeks to learn lessons ‘in flight’ rather than wait to make improvements at the end. The approach ensures constant reference to the goals of the project and enables careful observation of unforeseen challenges and opportunities.
Summative evaluation considers and compares the realised benefits with those that were expected. It frames a ‘before and after’ picture and guides interventions to improve the marginal gains.
System leaders and change sponsors need to be ambitious and focused on their approach to evaluation. This is not to provide the evidence for criticism, but rather to maximise the significant marginal benefits that are gained from learning lessons.
Structured evaluations enable system leaders to address serious questions about the change program and the intended outcomes, such as:
- What is the return on investment?
- Who is benefiting from that ‘return’? Are patients recognising a better service experience and are outcomes improving?
- If it is too early to observe long term health gains, what indicators do we have that we have a route to seize these benefits? Are care processes more integrated? Have we increased communication in multi-disciplinary care teams? Are patients confident about their health goals, for example?
What we do
We conduct formative and summative evaluations that assess the outcomes and costs of health service investments
All major investments in service and system change need systematic evaluation to optimise the outcomes for patients, populations, professionals and the system itself from the investments made.
Such evaluation needs to be carefully planned so there is appropriate combinations of summative and formative evaluation.
We work with system leaders and program sponsors to focus evaluations on the issues that matter most rather than to attempt to learn every lesson.
Our evaluation focus evolves through the lifespan of the program
- As the program progresses, the focus of the evaluation needs to evolve.
- When the program starts, we are most interested in ensuring that we have the right resources in place to run the service and that the intended users are finding their way into the service.
- As the program progresses, we become increasingly focused on ensuring the activities and outputs are being carried out in the right way.
- Finally, our focus moves to understanding how well the program is leading to the achievement of the desired outcomes and impact.
Our evaluations explore leading indicators that give confidence in the improvement trajectory
Our approach is to not shy away from the questions that need to be asked but to add creativity and innovation in the way we structure the evaluation, engage the stakeholder, collect the evidence and analyse the data.
Health gains do take time to be realised, but our evaluations seek to explore the leading indicators of health improvement. Do patients have a better understanding of their health risks? Are they forming health goals and what evidence do we have that they are committing to them? Do they have personal support networks? Are they receptive to evidence about the consequences of their health risks?
All programs seek to maximise beneficial outcomes from investments. We take care to understand these outcomes, test their validity and consider the evidence needed to give confidence about the improvement trajectory.
Get in touch to discuss how we can support you to evaluate your projects and programs
Jay Rebbeck
