What CSAIF funded:

The fund was used to review and revise the training provided to the full-time volunteers who undertake service in schools. In-house education experts were recruited and an external process evaluation of the training was commission to test the results of the training overhaul. View the impact evaluation.

About the evaluation

Level on Standards: Level 2 - you capture data that shows positive change, but you cannot confirm you caused this.

Evaluator: Renaisi

Aim: City Year’s evaluation aimed to better understand the impact of their Whole School Whole Child initiative on both pupils and corps members (i.e. volunteers).

Key findings:

1. 69% of targeted students maintained ‘good’ attendance or improved their attendance, with the ‘walking bus’ initiative contributing to a 83% decrease in late arrivals.
2. Incidents of negative student behaviour reduced by at least 50%, with incidents of serious behaviour reduced to almost zero.
3. Contact with corps members enhanced communication and relational skills in targeted pupils of all ages, with pupils becoming more engaged with the importance of career planning and preparation.
4. Targeted interventions helped ‘off-track’ pupils improve their attainment in English, with pupils receiving targeted support in English improving more than pupils not targeted for English interventions.
5. Corps members experienced increased confidence and relational skills; improved skills required for success in work or study; a greater commitment to civic roles; a sense of agency within the community and improved overall wellbeing.

Methodology:

• Method: City Year utilised pre- and post- evaluation surveys, administered to all participating schools and corps members. These were administered in a one-to-one fashion, and overseen by a City Year ‘team leader’. City Year also used data that is routinely collected by schools, for example on attendance.
• Participants: 151 corps members and 281 pupils were surveyed – this dropped to 104 corps members and 129 pupils by the time the follow-up survey was administered.

Why this Level:

City Year have delivered an evaluation that suggests a positive change over time for the people that it supports, using appropriate tools. The rate of attrition for the pupils is quite high (more than 50%), which means we cannot be confident that those findings are representative of all City Year pupils. However, the sample size at follow-up is still quite high (129), and the attrition rate for corps members is low enough for us to be confident in the representativeness of those findings.

About the evidence journey

Progress: City Year have advanced from Level 1 to Level 2 on the Standards of Evidence, reflecting the fact that they are collecting pre-and-post data in a more thorough and consistent manner. Furthermore, this data collection is embedded sustainably within the organisation, and a process evaluation delivered by Renaisi has helped City Year to refine its service and improve its quality.

Lessons learned: Since the CSAIF grant, City Year has reviewed the survey techniques and structures used in order to link the outcomes of the Theory of Change more explicitly with the questions asked of pupils and volunteers. The Renasi evaluation led to a review of in-school management practices in order to improve the school’s understanding of the City Year partnership and create more efficient data collection processes.

Next steps: City Year will undertake a review of their Theories of Change to ensure that the programme continues to pursue clear and salient outcomes. Work is being undertaken to streamline delivery and evaluation of individual components.