Introduction

The Joint UCL Public Policy & Alliance for Useful Evidence training programme was an experimental learning programme for researchers across UCL departments that aimed to support researchers to develop their understanding and skills in engaging and communicating evidence for impact. Based on the 2016 Science of Using Science project conducted by the Alliance for Useful Evidence, the EPPI-Centre at UCL, the Wellcome Trust, and the What Works Centre for Wellbeing, it sought to explore what works to increase the use of evidence in policy, and how these skills could be developed within an academic cohort. It was designed to focus on in-practice application, and was delivered in ten 2-hour sessions spread over 9 months in order to encourage the group to reflect, learn and develop their competence by problem-solving with their peers from different departments and disciplines. Given that the programme was an experimental learning offer, it was developed with a high level of flexibility in order to test various learning models in response to the needs and feedback of the cohort. It experimented with many different formats, approaches, and tools to determine what resonated with the intervention groups. As a result, a highly diverse programme was delivered over eight sessions between October 2017 and May 2018. The following report describes the programme that was delivered, provides a summary of the feedback received from the participants, a puts forward key insight and recommendations for both partners taking it forward.

The Participants

The programme included an application process that was managed by the UCL Public Policy team. This had three main aims: to manage demand for the programme given that the programme was externally funded and unique in UCL; to ascertain participants' motivations to increase the impact of their work, and to better understand the needs of those that would attend in order to shape the content. These aims were explored through an online application form that asked applicants to state their motivation for applying for the programme, their current approach to impacting policy, and the challenges that they face within this process. The shortlisting and selection process was managed by UCL and took into consideration researchers current exposure to and engagement with policy professionals as well as their expectations of the training course and the tools and skills they hoped to gain. 22 applications were received with 20 being invited to participate on the programme.

Participants' experience levels ranged from early career researchers to more experienced academics, and represented diverse disciplines, including engineering, history, education, biomedical sciences, and more. Despite the diverse backgrounds of the cohort, there were shared perceptions of the motivations and challenges faced in translating their research into policy impact that are summarized below:

Shared Motivations

1. To learn and develop a plan/strategy for demonstrating impact in
2. Determine what methods for research translation lead to the greatest impact on policy, whether it be through one-off evidence uptake or more systematic incorporation into decision-making
3. Improve ability to communicate research content tailored towards both academic and policy communities in a way that sparks action; in that communication effectively demonstrates internal validity while remaining attractive and accessible for policymakers and non-specialist audiences
4. Enhance and strengthen knowledge, experience and skills in policy engagements

Shared Challenges

1. Communicating research evidence to appropriate audiences and in an effective manner
2. Uncovering pathways for policy engagement and relationship building with decision-makers
3. Communicating the complexity of the research in an easily digestible way, including finding the most suitable language for the relative audience.
4. A Lack of formal strategy or training in strategies for research translation or policy engagement

Despite many similarities amongst the participants, the application forms revealed differences participant incentives and relationship with evidence-informed policy making processes. For example, participants demonstrated contrasting motivations in that some were interested in how the learning from the program could be applied to individual research projects, whereas others were interested in partaking in the programme to learn systems and skills in order to relay this to their teams and/or colleagues.

The diversity of experience, especially in levels of experience working with decision-makers, was particularly stark in the applications. Some participants had pre-established working relationships with policymakers, whilst others had not yet had the opportunity, nor confidence in the way in which they collated their data, to present their research at this level. A full summary of participants' background and motivations can be found here.

The Modules

The learning content was structured around 6 core modules which were drawn from UCL Public Policy's experience and the Science of Using Science research. These modules consisted of either 1 or 2 sessions that involved a mixture of research and expert input, combined with activities and worksheets that supported the application of content to their work and action planning. A diverse range of methods were used to encourage and support the group in developing practical applications of their work within these areas, as opposed to working within more abstract concepts.

The learning programme

The first module of the programme framed the programme. It provided the group with an overview of the research, introduced the action planning component of the training and crowdsourced participant needs as the basis of design for the core content, which was used and reviewed with the group at stages throughout the programme.

Crowdsourcing allowed us to more closely design and align the content of the programme with the needs of the group. A summary of what was requested during the crowdsourcing portion of the programme can be found in the following document. Key themes derived from this session included the importance of balancing complexity and clarity, understanding different levels of the ecosystem, and identifying institutional support and expertise.

Module two focused on the policy landscape and policy needs, and was delivered through two sessions supported by policy expert Paul Cairney from the University of Sterling. The first of these sessions drew from Paul's research and provided a theoretical framework for considering how to engage policy stakeholders. It emphasised the messy reality of policy-making and the role that narratives and storytelling have in effectively communicating messages. The second session put this into practice, as participants presented their research tailored towards a policy audience and received feedback on it. These insights were then added to their action plan.

Module three focused on understanding policy stakeholders, and was delivered through two sessions that explored how stakeholder engagement can be put into practice. The first session provided participants with an overview of what the research recommended around engaging policy stakeholders. Participants were asked to identify patterns of engagement within a series of case studies, then mapped their stakeholder base within their own policy areas. The second session consisted of a panel of former and current civil servants and academics, including a previous Chief Economist from the Department of Education and a current Civil Servant working within the What Works Centre Coordination unit. Panelists were asked to share what they experienced to be the biggest barriers in using research in policy and what they found worked in getting evidence used. To ensure a high level of attendance and engagement, the panel session the was opened for other academics to also attend, around 40 participated in total.

Feedback received from participants at the mid-way point demonstrated that the programme was too long (see section below on evaluation). As a result, the remaining programme was shortened, with modules consisting of one session each instead of two. A full outline of the rationale for this decision can be found here.

Module four focused on communicating evidence, with an emphasis on both audience segmentation and the importance of tailoring based on the needs of the audience. Participants were asked to prepare an email to be sent to policy stakeholder that introduced their research area, which was then reviewed within the session. The communication session drew heavily from the institutional capacity and expertise of the UCL communications department, who led presentations on branding and communications.

Module 5 focused on facilitating interactions with policy stakeholders, which was structured around the presentation of current research surrounding interaction facilitation as outlined within the Science of Using Science Report. During the session, a member of the Alliance shared case studies of successful instances of facilitation of policymakers, and led an activity in which participants developed a theory of change for their approach to influence policy.

The final session focused on how participants consider impact in their research, and how to further develop impact frameworks within their research moving forward. It involved two presentations by UCL collegues, one on the Research Evaluation Framework (REF), as was requested by participants, and another on incorporating impact into funding applications. The session finished with an action planning session, in which participants reflected on all of the content covered over the course of the programme and considered how they would put content elements into action. Participants were encouraged to keep their action plans and remain in contact with UCL Public Policy as they consider policy impact in their work in the future.

Participation

UCL to add attendance at each session as a record

Participant feedback

Participants were asked to provide feedback informally throughout the programme directly to the UCL Public Policy team, and formally through two surveys, one at the halfway point of the programme and another at the end. While overall, little feedback was received both informally and formally, the feedback that was gathered was used to influence the design of the programme and content.

Informal feedback was received on two modules: understanding the policy landscape & policy needs, and understanding policy stakeholders. For the understanding policy landscape sessions, informal participant feedback indicated a need for a more engaging and applied presentation style.

In response to this feedback, all future modules included a strong element of participation and application of concepts, whether through worksheets (such as a network map or theory of change worksheet) or through case study examples. Substantial positive feedback was received following the understanding policy stakeholders panel session on the range of contributors, their experiences, and the format of the session. Whilst it was not possible to replicate this format again in the programme due to the limited availability of similar expertise, the team sought to draw in expertise throughout the remaining modules where possible.

Formal feedback was requested from the participants through survey monkey halfway through the programme and at the end of the programme. The focus of these surveys was to ascertain participants reflections on the sessions and their utility. Five participants completed the first survey (25% of the cohort), none of which had attended all of the modules delivered up to that point.

Midpoint feedback: participant attendance

Of those that responded, 80% stated that they had found the content very useful or somewhat useful (four participants) while one participant stated had not found it useful at all, stating "I do not know anything about the UK policy environment as an overseas scholar. I am afraid I was completely lost".

At the end of the programme, formal feedback was submitted by 7 participants (35% of the cohort), non of whom had attended all sessions. None of those that responded found the sessions to be damaging, though one participant found the session with Paul Cairney un-useful. Across the modules, the first two sessions received the most positive scorings, with the expert panel session coming in at a close third.

End of programme feedback: Ratings by session

Development Team Reflections (UCL Public Policy & Alliance for Useful Evidence)

Strengths

Whilst it's difficult to draw robust conclusions from such a small sample, the feedback did highlight some strengths of the approach:

• Improved Understanding of Policy Making Context: "[I feel more confident to try to influence policy] in that understand its a complicated landscape and there are different techniques/strategies and also its mostly luck!". The programme sought to develop people's understanding of the complexities of the policy making process and the need for a multi-level approach. We wanted to bust the myth around there being a magic formula for getting research into policy. Some of the feedback forms indicated that this objective was met,"I believe I am less naive than I was before".
• Improved Awareness of Stakeholder Incentives: "I definitely got a better understanding of civil servants and how they move around”. The programme aimed to help participants understand the needs and challenges of policy makers so that they can better see how their work can meet these needs. This is driven from The Alliance's experience which indicates that the research community is prone to approaching policy makers without an awareness of the political environment in which decision-makers are embedded: "The programme has given me... more clarity about the needs of policy makers."
• Emphasis on Proactive Relationship-Building: “I think I am going to make an effort to attend events, and make connections even if they aren't obviously directly relevant". The programme put a big focus on the role of engagement and relationship building, encouraging academics to strengthen their networks. For others, the importance of being proactive stood out:, "Going forward I know who I need to talk to in my organisation about influencing and some suggestions of what influencing looks like".
• Strengthened Inner-Organizational Links: “The most valuable thing for me in this programme was to see that most academics are facing the same challenges, but that there are units at UCL that can help with many of the issues that we are confronted with." The programme design focused not only on bringing in external expertise, but also raising the profile of the available expertise from UCL Public Policy to connect and draw from the institutional capacity that already exists but isn't necessarily connected. On this point the programme seems to have been particularly effective, as one participant put it, "The key insights was the resources and expertise available through UCL Public Policy. In the future, I will be sure to their feedback on policy engagement ideas before approaching stakeholders".

Weakness

As the programme was a pilot, a key focus of its design and delivery was to test the viability and format of a programme of this nature for a diverse group of academics:

• Session Timing: 'Fewer, but possibly longer sessions with smaller gaps between them'. Whilst training over longer periods of time allows for in-practice application exercises, there was clear feedback from this cohort that this did not work for them. Many participants suggested that they would have preferred "fewer sessions", and whilst the programme was shortened half way through, there was a sense sessions could have been reduced even more. There was a diversity of opinions regarding how session length and duration could be altered, whether it should be more structured, " or something that did not require such high levels of commitment, 'I was of course not able to attend all sessions'.
• Programme Tailoring: "More tailoring to individual participants needs". Both the diversity of participant experiences in policy and the range of disciplines represented presented a challenge in the design of the programme, as the policy landscape and networks for each policy areas are highly diverse. As a result, the content had to be broadly applicable; drawing from examples from a range of sectors in the UK. As a result, a few participants highlighted that the programme didn't closely meet their need. In addition, some participants felt that, 'the international perspective was widely missing", or expressed the desire for explicit tailoring of content material, “an initial one-on-one meeting with participants at the start might help provide focus".
• Applicability of Content: "More case studies". Whilst case studies were woven into many of the sessions, there was clear feedback that participants would have liked more, and that they should be presented in greater depth that goes beyond more generalisable theory: 'Case studies could make more manifest some of the ideas that were presented in abstract form in some of the presentations, as well as show the full life cycle of engagement'. In addition, some participants wanted case studies that related more closely to their discipline and builds on the previous point around tailoring research content depending on the audience, "I would have more examples from primary scientists who have influence at UCL."
• Group Engagement: "I probably missed a lot of important insights". Inconsistent participation, commitment, and attendance within sessions was a challenge for the programme that grew incrementally as the program progressed, and was dependent on participant application to this work in between sessions. Some participants indicated that this was due to scheduling,"It would have been good to have the dates up front so that we could attend all sessions or decide that it wasn't feasible to do the programme". Others weren't able to commit due to the length of the programme However, this lack of consistent attendance brought challenges to creating an incremental learning journey: "It was therefore an issue when I missed a session too".
• Career Motivational Constraints: 'Some participants have research that clearly has policy implications whereas some don't.' The diversity of participants' experiences and needs brought challenges to the design of the programme, which reflects feedback received that the programme wasn't fully aligned with participant needs. A particular challenge for some was that influencing policy wasn't relevant to their research, or that they were not yet in a position of sufficient seniority or experience to be exposed to policy makers. For these participants, the programme was not relevant to their needs and its applicability was limited. As one participant shared, 'I wasn't in a position to take forward actions straight away and would have appreciated some more theory/practical examples that I could then apply to my own role.'

Recommendations

The Joint UCL Public Policy & Alliance for Useful Evidence training programme aimed to experiment with what a collaborative learning offer based on the Science of Using Science report would look like, and was successful in generating a multitude of learnings and insights. As one participants stated: "it was an excellent pilot programme, largely because I imagine it was clear which elements worked and which didn't." This final section will outline our recommendations based on the experience of this pilot:

1. Tailor Content to Areas of Expertise

Future learning offers need to be more tailored to learners subject areas and levels of experience. The current pilot provided an interesting test case to see what, from a learning angle, is possible to provide for a group of highly diverse academics, yet demonstrated that this leads to a lack of adequate support for participants. Tailoring content towards participants subject matter could help them feel better connected to the content and think more critically about how the learnings can be applied to their own work. Below are considerations and suggestions for how to tailor content towards the varying levels of expertise and backgrounds within audiences:

• A pre- workshop meeting with participants could help guide initial discussions surrounding how content is relevant to participant needs given varying embeddedness within different political and decision-making contexts
• Be selective with participants. Use the information provided within the programme application process, including baseline understandings surrounding topics covered in the course, participant motivations, and experiences to tailor case examples and content towards the audience.
• Segmentize workshop components based on learning needs unravelled during scoping conversations. While it's important to consider participants' backgrounds and levels of expertise when creating and delivering content, there are other targeting characteristics that can be taken into account within this process, including but not limited to: international vs. national focus, level of bureaucratic change interest, proximity to decision-makers, and familiarity with policy literature and landscape. If creating specific, tailored workshops is not feasible due to resource constraints, another possibility is having breakout sessions that reflect audience diversity.

2. Learning Offer Structure and Component Alterations

Changes can be made to the structure of the learning offer. These changes can build on what is already available across the organisation, help promote both the engagement and commitment of participants, and strengthen the ability of organisers to meet participant needs. Structural changes related to the length and frequency of workshops, content-creation processes, and learning mediums can all affect the effectiveness of the learning offer:

• If planning long terms engagements, inform participants of workshop dates beforehand in order to allow for adequate planning time and reduce attrition rates of participants. This prevents participants from missing any key content discussed in prior sessions, which allows for more holistic learning and more accurate feedback.
• Longer, more frequent sessions help to ensure continuity of content from one session to the rest, and allows adequate time for participants to do 'deeper-dives' into content. More frequent sessions allows for momentum and enthusiasm surrounding the topic to be maintained.
• Draw on behavioural insights to better encourage participant feedback. The low feedback responses of participants following workshop sessions demonstrates a need to encourage more feedback. There is an opportunity to draw on the literature in developing an engagement strategy for future learners.
• Content Variation depending on stage of learning and experience. Consider a balance of more prescriptive content (online for example), more action-oriented workshops, or one-on-one sessions depending on whether the learner is at the 'raising awareness' level or is at the 'making it happen' stage (see figure below).

• Consider how learning offers can relate to existing UCL Public Policy Offers and institutional capacity. Of those sessions that were delivered, those that drew on expert narratives and UCL departments were the most popular. UCL Public Policy already offers sessions like these on an adhoc basis and had the connections within the organisation to further elevate the expertise of UCL in this area. There is the potential to consider how these are packaged and combined together as a learning offer for cohorts of academics.

3. Embeddedness within Broader Systems

Participants, and academics more broadly, are situated within contexts of meaning-making and systems that influence their ability to participate in evidence-informed policy. These can include organizational affiliations, internal and external incentives, career mobility requirements, conceptualizations of problem and purpose, and opportunities for engagement with policy. Organisers should consider how to build bridges and target programming across these systems in order to leverage all available support when supporting academics within their learning journey and build participants motivation for considering impact:

• Promote Knowledge Sharing Across Systems on existing support networks, resources, or organizations that academics are able to tap into. For example, many participants found a lot of value add in learning about how UCL Public Policy was able to support them.
• Link Learning Approaches tailored towards academics, so that organisers are better able to target learning offers in a way that is not repetitive or counter to participants' prior learning experiences.
• Institutionalize Support Networks, which allow for the development of trust and social capital across stakeholders. Effort should be made to develop and promote both formal and informal networking and relationship building activities to allow for the co-creation of long-lasting and meaningful partnerships between academics and decision-makers.
• Tap Into Existing Incentive Schemes of participants in order to work within their contexts of understanding. For example, one session presented REF, and how the learning material can contribute towards this framework. Aligning learning offers with existing academic incentive schemes helps to ensure participant engagement and commitment.

Next Steps

The development and delivery of this pilot training programme has been an important learning experience for both the delivery partners.

UCL Public Policy plans to take the activity forward by incorporating the learning from the pilot, particularly regarding the type/variation in cohort, timing, duration and delivery of training, as it develops its own plans for a blended training programme for UCL staff in line with UCL Occupational Development.

The Alliance for Useful Evidence will use the lessons from the pilot to shape their approach in the design and delivery of its 'Research Uptake' learning programmes. It will help guide new iterations of cohort targeting, content tailoring, and course structure, while contributing to the broader strategic mission of embedding Science of Using Science learning within organisations. The Alliance intends to deliver content derived from the pilot to undergraduate and master's students participating in UCL's Global Citizenship Programme during summer, 2019