Evaluation tips from the Stroke Association
How many of us, I wonder, have commissioned an evaluation only to find it goes off track somewhere along the line and you end up with a report that’s less useful than you’d hoped? Or perhaps you’re an evaluator who has shared a report with a client only for them to be less pleased than you’d anticipated?
At the Stroke Association we’ve learnt a lot from evaluations that have gone really well, and some that have been less successful. If you’re new to commissioning research, these tips might help you get the most from your evaluators.
Make your brief clear by getting a review
Develop clear and specific evaluation questions. Keep them in focus so you can make sure the methods chosen really offer the best chance of addressing these. Be really clear about what’s driving your evaluation, how you see yourself using it, who will care about what it says, and who matters most when you think about audiences. Consider asking other evaluators to check your brief so you can get an insight and feedback on what you are asking.
Consider the relationship as well as the result
It can help to think about the relationship you want with your evaluators, not just the end result. Don’t just ask an evaluator what they’ll do, try and understand how they like to work and think about the kind of relationship you might want with them. Along with reading their end reports, it’s also a good idea to ask for references to find out what they are like to work with.
Be clear on role expectations for both organisations
Ensure clarity of roles and expectations upfront. Ask in the brief for them to be explicit about any assumptions they may expect from you as a client. Once expectations are clear, then think about internal communications – who needs to be involved, how and when, so that you can get people on board to ensure a successful evaluation process.
Share your vision of success
Just as you’d have success criteria for any project you were evaluating, think about success for your evaluation and make that clear to your evaluator. Sharing examples upfront of evaluation reports you like can help develop a shared vision for the final output and ensure a better report. Specify the type of report required, the expected depth or level of analysis and practicalities about tone, language, audience and whether or not recommendations are expected.
Risk assess your evaluators proposal
An initial risk assessment during scoping can be helpful. Either ask your potential evaluators to include a section on risks and assumptions in their proposal, or at least risk assess any proposals you get yourself – you need to ask 'what would we do if this aspect/activity/approach did not work?', 'do we need any plan Bs, and if so what could they be?' This helps you plan ahead where flexibility might be needed.
Don’t forget you’re an expert too
If you have doubts about what’s being proposed, trust your gut, ask for things to be explained, be upfront about concerns and questions. This can be particularly important for suggestions about data collection processes or tools. If you feel an 8-page questionnaire is not right for your group even if it’s ‘the best method’ for a particular evaluation question, talk this through and ask your evaluator to come up with other ideas.
Induct your evaluator
Supporting evaluators to learn about your client group (eg, by offering them training) can be a really worthwhile investment. At the Stroke Association, having evaluators who understood our client group and communications challenges has worked well for us on several occasions. The evaluation consultants get a free personal development opportunity and you get a better evaluation.
Consider an advisory group
An advisory group can be great for continuity in a long-running evaluation. It can give you additional evaluation expertise or user perspectives to give you confidence in making decisions. Make sure you’re all agreed on its purpose, ensure its role is meaningful and that its remit and any boundaries of influence or decision-making are clear to all concerned.
Don’t step back too far from your evaluation. You need regular updates and a flow of communication about how things are going. Build this in to your plans to help you understand how things are going and identify any issues or areas where things aren’t working at an early stage.
Sometimes clients ask for an interim and a final report without thinking about what the interim report is for and how they will use it. They can serve a number of purposes but can also end up being expensive and not particularly useful, so just be clear why you want it and that it’s coming at the right time to really be helpful. Have an honest conversation with your evaluators about how you would like to manage agreeing the final report, and if you have stakeholders who will need to review it, make sure the evaluator’s plans have taken this into account.
The Stroke Association are being supported by Nesta and The Big Lottery Fund as part of the Accelerating Ideas programme to expand and scale their Hand to Hand peer support programme.