What CSAIF funded: Smart Works was awarded £235,000 to develop the processes and systems to replicate the Smart Works model through a licensing approach. New Smart Works Licensees opened up in three new areas in England; Greater Manchester, Reading and Birmingham. They have grown to reach around 3000 people per year across the country, with further potential to grow in these new sites, supported by over 200 highly skilled volunteers around the country. Smart Works were also awarded £20,000 to evaluate the impact of their work. View the full impact evaluation.

About the evaluation

Level on Standards: Level 2 - they have captured data that shows positive change, but they cannot confirm they caused this.

Evaluator: The Social Innovation Partnership (TSIP)

Aim: The evaluation of Smart Works aimed to assess how effective the service is at helping unemployed women get back into work.

Key findings:

1. Almost half of the contacted Smart Works clients secured a job within a month of receiving the service. Of those, almost 80% said that coming to Smart Works had increased their chances of getting that job by a lot (and almost 20% by a little).
2. Amongst those yet to secure a job, just over half of the participants indicated that they felt Smart Works had increased their chances of getting a job by a lot (and almost 30% by a little).
3. The evaluation highlighted that almost all Smart Works clients felt the service had a positive impact on their confidence and interview skills.

Methodology:

• Smart Works collected data via two measurement tools: a survey that was administered to participants immediately after receiving the service, and a follow-up text message or phone call that was delivered approximately four weeks later. All tools were specifically created for this evaluation, and were not externally validated.
• 278 individuals from Smart Works' London offices took part in the evaluation (all clients starting between December 2015 and mid-February 2016). 78% of participants also completed the follow-up survey, providing a strong level of confidence in the project’s findings.

Why is this a Level 2 Evaluation?

Data on employment status was collected at baseline (implicitly) and at follow-up, showing a large positive change over time. Direct questions at follow-up strongly indicate that some of this change can be attributed to Smart Works, and the large amount of qualitative data on the clients’ positive experience also supports this claim indirectly.

About the evidence journey

Progress: This evaluation has moved Smart Works from Level 1 to Level 2 on the Standards, through refining their measurement tools and increasing their response rate at follow-up. It has also set the service up to continue collecting and analysing data sustainably, through implementing a new text message data collection system and a new automated analysis tool.

Lessons learned:

• Having a clear Theory of Change approach has proved essential in ensuring consistent outcomes are measured in the client questionnaire.
• The use of text messages to follow up with clients has proved an efficient and effective way of finding out if a client has been successful at interview.
• The process of testing and refining evaluation on the London service and then looking at how this can be taken to new locations has proven successful.

Next steps: Smart Works will use the newly refined measurement tools and process that have increased response rates with licensees to ensure that their service will also be recognised as Level 2 on the Standards of Evidence. The new survey is already in use, and the follow up text message will be used when licensees are moved on to a new database system over summer 2016. The new automated analysis tool will then also be rolled out with licensees.