Blog: Let’s stop chasing unicorns
Patty Lozano-Casal, Evidence into Action Manager, Evaluation Support Scotland joined a panel discussion about standards of evidence at Realising Ambition’s annual Learning Event in Birmingham on 1st June which was focused on the concept and worth of Standards of Evidence. Here she write about her thoughts...
I’ve always had an active imagination. I remember as a child believing that fairies spoke to me through the sound of the wind. Children believe in things that they’ve never seen before like fairies, superheroes, dragons and unicorns. They don’t have any evidence of their existence, yet they think they’re real. As adults we no longer search for the philosopher’s stone; instead, we look for robust evidence before we form a view or make a decision. So I find myself wondering, at what age do we lose the ability to believe in what we can’t see? And is that a good or a bad thing?
In the ‘evaluation world’ some people speak of Standards of Evidence, which are frameworks to help us determine how confident we can be that an intervention is having a positive impact. The first Standards of Evidence for the UK were developed by Dartington Social Research Unit (DSRU); others like Project Oracle and NESTA use adaptations of these.
So there are different sets of standards out there, but do they work in practice or are they mythological creatures like unicorns?
Let’s look at some evidence. Project Oracle offers their funded projects to have their evaluation validated against their own set of standards for £400-£750, depending on what standard they choose.
So far, Project Oracle has done 296 validations, of which they found no robust evidence of ‘model’ and ‘system’ ready. Most evidence provided by the 296 projects fell under standards 1 and 2 (i.e. ‘project model and evaluation plan’, ‘indication of impact’). So what does this tell us?
I was recently invited to share ESS’s experience on this topic at a panel debate around the Standards of Evidence at the Realising Ambition conference on 1 June 2016. What I heard that afternoon resonated with what ESS learned from supporting third sector organisations and funders in Scotland for over 10 years.
Realising Ambition funded projects made very clear that existing standards of evidence are too theoretical and not always applicable in practice, particularly for small third sector organisations with limited capacity, expertise and/or resources. So could the Standards of Evidence as we know them really be dead? Louise Morpeth, DSRU’s Chief Executive, was open and honest about the limitations of the Standards, one being their linearity, which does not translate to reality – generation of evidence is actually an iterative process. Jonathan Breckon, Head of Alliance for Useful Evidence, noted that “not all evidence is born equal” and Tim Crabbe, using holiday-planning as an analogy for how decisions are made in practice, said that sometimes we use other than the standards of the market (e.g. TripAdvisor) to decide where / how to go for holiday (read Tim’s entertaining blog for more details!).
So if the Standards of Evidence were to be dead, what could we use to assess the quality and robustness of our evidence to make informed decisions about service delivery and policy-making?
ESS suggests the following:
1. Rather than an absolute standard, consider:
· Why do we need evidence?
· What will we use it for?
· What do we already know?
· What evidence is good enough (for the action we need to take)?
2. Don’t look just for evidence of whether something works, but how it works, how long it takes and how much it costs.
3. Triangulate your evidence - Good evidence informed decision-making needs evidence from a mix of sources (so not a hierarchy!) including:
· What people on the front line tell us – i.e. tacit knowledge
· What self-evaluation evidence tells us
· What formal research tells us.
4. Co-produce and involve the service user when generating evidence
5. Use the following framework, adapted from Levitt et. al. (2010), to judge whether your evidence is ‘good enough’:
· Transparent: clear methods, acknowledged limitations
· Relevant: up-to-date and appropriate
· Enough: strength of evidence vs proportionality
· Believable: accurate, representative and reliable
· Legitimate: coming from the right sources.
If after reading this you still think you and / or your organisation are chasing unicorns why not read the Evidence for Success guide and Realising Ambition programme insights on what works in replication and evidence use?
I better go now. Hold on, where’s my dragon…?