Introduction

The Digital Innovation Fund for the Arts in Wales (the fund), launched on 1st October 2015, is a strategic partnership between Arts Council of Wales and Nesta. It is the successor to the Digital R&D Fund for the Arts in Wales that ran from 2013-2015.

The fund is supporting arts organisations to experiment using digital technology to enhance audience reach or to develop their business model. The programme has four phases, as detailed in the table below.

The fund differs from a traditional grant programme in that it is designed to support arts organisations through a process of experimentation – to investigate high-risk, high-potential projects, based around clear, testable research propositions. The results of these experiments are shared through events and publications so that the wider sector is able to learn from the experience.

This report summarises the findings of our preliminary evaluation of phase one of the Digital Innovation Fund for the Arts in Wales - Identify Challenges and Opportunities. We are encouraging all funded projects to share the lessons they learn as they progress, and this report is part of our attempt to do the same. It is important, we feel, that we respond directly to the feedback we have received, and to use it to shape the programme as it progresses.

We hope that other organisations supporting the arts sector, as commissioners or sector bodies, can learn from our experience. If you would like to know more about the fund, please contact us and we will be happy to discuss in more detail.

Phases of the Digital Innovation Fund for the Arts in Wales

Fund Activities During Phase One

The fund includes a range of activity to support the development of digital capacity in the arts sector. One aspect of this is the financial and practical support for funded projects, but there is also a complementary set of activities – events, workshops, facilitated networking and one-to-one support – available to any arts organisation. Our intention is that every point of contact with the fund should be a useful opportunity to develop digital capacity, regardless of whether an organisation completes an application or receives funding.

During phase one, these complementary activities comprised:

• Application surgeries, with one-on-one and group sessions for any arts organisation interested in applying to the fund. These were provided in person, or in some cases via phone/internet. Members of the fund team would meet with prospective applicants to provide feedback on project ideas, to explain the structure and criteria for the fund, and to give general advice on developing digital capacity, enhancing audience reach and exploring new business models.
• Theory of Change workshops, introducing arts organisations to the Theory of Change tool, a logic model that provides a basis for setting out the sequential steps needed to achieve long term goals. The tool was used as the basis for the design of the application form itself. The application form is included as an annex.

There will be a further complementary program of workshops and facilitated networking as part of the third phase of the program. These will bring together arts organisations from across the sector around broad themes of common interest, such as how arts organisations are using digital technology to engage young audiences, or in a rural context. These events will also provide an opportunity for funded projects to share and review the emerging findings of their research with sector colleagues.

Data Gathering During Phase One

Data gathered during phase one included:

• Applications: We received 31 completed funding application forms.
• Theory of Change workshop feedback forms: Short, anonymised feedback forms completed on paper by the 40 individual attendees of the Theory of Change workshops.
• Application feedback forms: Longer, anonymised feedback forms completed online by individuals on behalf of 25 arts organisations, including 17 organisations who submitted applications and 8 organisations who engaged in the workshops and/or surgeries but who did not submit an application.
• Selection panel feedback: We collected scores and feedback from each of the members of the selection panel, comprising Arts Council of Wales and Nesta staff.

Unless stated otherwise, the quotes given are taken from the application feedback.

The following documents are included here as an annex:

• Transcripts of anonymised feedback - ToC workshop feedback forms and application feedback forms.
• The application form.
• The scoring criteria.

Applications

We received 31 applications from across Wales. The applications were scored by a panel of six Arts Council of Wales and Nesta staff. A maximum score from any one panel member was 14. All projects considered fundable by the selection panel received an average score of between 8 and 12.

The higher scoring applications shared some common features, which broadly reflect the scoring criteria. Conversely, where applications scored lower, they lacked one or more of those features.

Higher scoring applications:

• Made a clear link between organisational strategy and proposed activity, and gave evidence that the work would be considered a priority across the organisation.
• Identified a challenge on a scale appropriate to the organisation, its resources and networks. Crucially, they did not underestimate the scale of the challenge and the degree to which their proposal would be able to address a broader sector challenge.
• Demonstrated ownership of the challenge at all levels of the organisation, a shared sense of how important and how urgent the challenge is to address.
• Identified a challenge that had relevance both to their individual organisation(s) and the wider arts sector, rather than identifying a very specific or esoteric issue.
• Gave a clear explanation of the need and scope for 'research and development', rather than seeking support for 'business as usual' activities such as social media marketing/networking or IT system upgrades.
• Did not go too far towards identifying specific technical solutions, describing ideas and suggestions but showing a willingness to be flexible and open to new possibilities.
• Sought to build upon, rather than duplicate, other significant work already funded by Arts Council of Wales, Nesta, or others.

The scoring criteria are included as an annex.

Feedback

The feedback data gathered during phase one will be used to inform the final evaluation of the fund. At this interim stage, we have grouped together the feedback, positive and critical, into themes. The themes reflect aspects of the delivery of the fund during the first phase.

The fund activity is still ongoing, and we have not attempted to measure 'impact' or 'success' here. This paper presents a range of the feedback provided so far and our initial response to it. These emerging findings will help to inform the ongoing delivery of the fund, and to suggest areas for further study and development.

The full data set is included as an annex, including transcripts of all feedback forms received from arts organisations.

The following sections of this paper explore four themes:

• Strategic focus: making sure that digital projects are clearly linked to organisational strategy.
• Relevant, arts-based examples: using examples from the arts, providing a sector-specific context to the issues at hand.
• Sector-wide support: offering benefits at all points of contact, including to those arts organisations who do not receive funding.
• Research skills in the arts sector: supporting the development of key research skills to enable arts organisations to make effective use of their own data and of research carried out by others.

Strategic Focus

Identify challenges: an additional stage

"By focusing on the challenge or problem it was actually quite liberating and it seems a good way to start a process of learning.” - CEO/Artistic Director, Cardiff and Vale of Glamorgan Attended a workshop and an application surgery, applied to the fund

“The application was usefully set out in terms of a problem to address, rather than focusing on a detailed solution. This was particularly welcome.” - Senior/mid-level manager, Cardiff Did not attend a workshop or surgery, applied to the fund

During phase one, we invited arts organisations to approach the fund by identifying an important strategic challenge, rather than a fully-formed project idea. This was a deliberate decision, based on our experience of the predecessor programme, the Digital R&D fund for the Arts in Wales, which ran from 2013-2015.

The application process for the previous R&D fund (2013-2015) required organisations to have defined the problem they were looking to solve, formed a team and developed novel ideas to deal with the challenge. On reflection, we felt that there were many organisations who were not in a position to develop a proposal to this stage without additional support.

Many of the arts organisations who applied to the previous R&D fund had limited experience of working with digital technology, and they did not have well-developed links with the digital technology sector. Furthermore, many of the proposals we received for the R&D fund had arrived at technological solutions without clearly defining the problem that the technology would solve for their organisation, making it difficult to see what the proposed project would aim to achieve.

The first phase of the present fund, Identify Challenges and Opportunities, was included specifically to address this issue. The fund is based around two areas of impact - enhancing audience reach and exploring new business models. We were confident that almost every challenge within those broad areas would be able to identify ways in which digital technology could help. By remaining focussed on the key strategic challenge, we were also confident that range of solutions could be explored during the research phase without changing the purpose around which a team had been built.

Avoid being 'technology-led'

“The application process encouraged us to develop our idea clearly. In fact, it would be feasible to argue that the application process even prompted us to see the idea in the first place.” Attended a workshop and an application surgery, applied to the fund

“It encouraged clarity in our proposal development, particularly after having attended the [Theory of Change] introduction workshop." - Senior/mid-level manager, Cardiff Attended a workshop and an application surgery, applied to the fund

“[The application process] made us analyse and think through our objectives and what we need to achieve.” - Senior/mid-level manager, Carmarthenshire Attended an advice surgery, applied to the fund

The activities during phase one of the Digital Innovation Fund were designed to allow arts organisations to come forward whether or not they had identified a specific digital solution to meet their needs. This then provided an opportunity to fully explore the reasons why an arts organisation might look to digital technology to support its objectives.

As a result, we had arts organisations approaching the fund with excellent proposals despite having little or no previous experience of using digital technology in their work. Whilst the technology was not defined, arts organisations were able to be completely clear on what they needed the technology to do and therefore were able to have a clear basis on which to define what success would look like.

A different way of thinking

“Focusing on a challenge or need that needed to be addressed without initial regard for a solution was initially counter-intuitive... The Theory of Change model helped me to think in ways that I had not thought before. It was a real help in structuring the complexity of my ideas and focusing on the challenges or needs that needed addressing rather than on the solutions." - Senior/mid-level manager, Wrexham Attended a workshop and an application surgery, applied to the fund

“The social change innovation process was a new concept and took a short time to adjust to a different approach." - Senior/mid-level manager, Pembrokeshire Attended a workshop, didn't apply

We are aware that an organisation seeking grant funding will often define the broad outlines of a project before identifying funding opportunities. This then requires a process of 'retro-fitting' a rationale around a project idea to fit with the criteria for funding, whilst maintaining the integrity of the original idea.

The activities during phase one were instead asking organisations to re-assess their original idea, or even to approach the fund without a specific project idea in mind. Some of the feedback we received suggested that some organisations found this confusing or uncomfortable, and not what they were used to in applying for funding.

The decision to use Theory of Change was informed by Nesta's work on Standards of Evidence. The Standards of Evidence were originally developed to support impact investing. In essence, they set out a means of assessing how confident we can be that a given intervention is having its intended impact. The first stage is to provide a coherent account of how an intervention has an impact. Theory of Change is one way in which a coherent account can be developed and communicated. Subsequent stages include having robust data that confirms that outcomes are being achieved, up to the highest stages of having reproduced the same results in different circumstances through the use of set procedures and manuals.

Feedback suggests that the fund required arts organisations to approach the development of their proposals in a way that they were not used to. We believe there is an opportunity for arts organisations to benefit from exploring the use of tools like Theory of Change, and approaches to project development used in other sectors.

In response to the activities during phase one we received a wide range of quality, fundable applications from arts organisations across Wales. In this sense, we felt that arts organisations rose to the challenge, and the workshops and application surgeries did provide an opportunity for arts organisations to adapt to this different approach.

Relevant, Arts-Based Examples

Principles before tools

"The process felt very different from the usual funding applications we're involved with. It was a challenge - although useful - to approach the process by describing a problem first rather than proposing a fully thought out project of activity and outcomes. The most valuable part of the process was speaking face-to-face with Rob [a member of the fund team] during one of the workshops, as I was able to discuss the particular circumstances and context of our proposal and Rob gave very useful feedback. I found the least helpful part was 'the Theory of Change' workshop which I didn't feel was something that would be used in our organisation and it felt quite alien to a creative way of thinking and working." - Senior/mid-level arts management, Powys Attended a workshop and an application surgery, applied to the fund

Theory of Change is a well-established planning and evaluation tool used extensively in the public sector and third sector in the UK and internationally. It was used as the basis for the application form for the Digital Innovation Fund precisely because it helps to plan out the steps required to meet a longer term goal. In doing so, it encourages discussion not just about outputs from a project but how those outputs will lead to the achievement of objectives of strategic importance to an organisation.

The quotation above, taken from the application feedback, shows the importance of ensuring that tools taken from elsewhere are presented in a way that is sensitive to the context. The Theory of Change tool is clearly not something they consider useful or relevant to their organisation. However, they do not reject the principles behind it, and when able to have a discussion at the workshop based on their own context and circumstances, the logic of the application process did make sense to them.

Using examples from the arts

“An arts-based example would have been really useful as everyone could have shared their experience and expertise for problem solving." Theory of Change workshop feedback

[Suggested improvement] “Making it more relevant to arts organisations.” - Theory of Change workshop feedback

“Interesting model but need to have more worked examples.” - Theory of Change workshop feedback

“More practical examples of how the theory has been used would be good.' - Theory of Change workshop feedback

“There was a lot of information to take in within a short time. Some guidance on how to make sense of more complex Theory of Change diagrams would have helped me." Theory of Change workshop feedback

At the Theory of Change workshops and the application surgeries, we tried out different ways of using examples – being more or less generic, using examples from the arts and using examples from other fields. The feedback we received suggested that it was useful to have specific examples, but that there wasn't a consensus on which examples were the most useful, and it seems likely that a choice of examples would have been desirable.

The first impression of a completed Theory of Change diagram can itself be somewhat confusing. The reality is that an organisation would likely need to practise producing such detailed diagrams many times before being able to produce a textbook example. We therefore believe that being able to show the stages of development of a full theory of change would have aided understanding and helped to give realistic expectations of what is achievable.

Sector-Wide Support

A supportive process from the outset

“I found the application process to be helpful rather than difficult in crystalizing my original ideas on my subject area/interest for both the Company and my own creative interest in developing digital platforms... It was useful being able to attend workshop and discuss (formally and informally) the ideas." - CEO/Artistic Director, Cardiff/Vale of Glamorgan Attended a workshop and an application surgery, applied to the fund

“Learning about the Theory of Change and how to use it to better understand our problem was a good challenge, and I think it will be useful for future applications... The application process was very positive and supportive." - Engagement and learning officer, Cardiff Attended application surgery, applied to the fund

“Although I didn't complete the process, the steps I did take enabled me to refocus where needful." CEO/Artistic Director, Pembrokeshire Attended a workshop and an application surgery, didn't apply

“Although it is disappointing news we did learn a lot along the process, it was very interesting to look at a problem from such a different perspective.” - Senior/mid-level manager, Cardiff Attended application surgery, applied to the fund

The fund was designed to deliver benefits to arts organisations at every phase of the process, irrespective of whether they applied for or received funding. The complementary activities were devised with this in mind - to offer advice and skills workshops to help their organisation consider how they might better understand their strategic challenges, in order to make more effective use of digital technology.

The Arts Council of Wales is able to systematically offer this level of support to portfolio organisations, those arts organisations who receive regular funding, but are not usually in a position to make such a level of support available more widely. There are individuals, working for arts organisations or for sector umbrella bodies, who can sometimes offer this kind of advice and expertise, but these individuals are not necessarily available to help everyone, and accessing their support can come with a cost - in time and money - that not every arts organisation is able to meet.

The most useful support: discussion and feedback

"I found the sessions thorough and very helpful. I was challenged and it made me think through the concept in a proactive/real manner. The team was very honest and we could feed their insight into our final thinking.” - Freelance arts professional, Gwynedd Attended a workshop and an application surgery, applied to the fund

"Challenging for our small organisation - to identify and concentrate on a small part of our general audience... Very helpful meeting with [fund team] - they took time to run through ways of identifying audience sectors - inevitably pointing towards employing more staff. However, [the fund's] model has a disconnect - being able to deal with each audience sector and the realities of needing to juggle multiple roles creatively and on a tiny budget.” - Freelance arts professional, Cardiff Attended application surgery, didn't apply

Arts experts rather than digital experts

"Good on testing my idea, but I feel the lack of discussion on digital solutions meant a gap in the day.” -Theory of Change workshop feedback

“I really need assurance on the later developments in new technologies and their capabilities to help me here." -Theory of Change workshop feedback

Many arts organisations are still put off by the idea of digital technology – considering it to be irrelevant, inaccessible, or just too costly. Those who do have an interest can feel anxious that they do not have sufficiently detailed or up-to-date knowledge. Using the term 'Digital' in the name of the fund could in itself be off-putting to some arts organisations.

Although we would like to encourage arts organisations to develop in-house digital capacity where possible, it is our view that it is neither realistic nor desirable to require arts organisations to become experts in digital in the short term. Our intention is that by focusing on their strategy, we will prepare arts organisations to be in a better position to bring in external digital expertise where appropriate. We feel that being clear on the strategic importance of a proposed digital project will put them in a better position to communicate their needs to a digital technology partner, and to develop a shared idea of what success looks like.

Research Skills in the Arts Sector

“It was challenging and useful. I will use the Theory of Change... Maybe more exercises [needed] in the various parts of a Theory of Change." - Theory of Change workshop feedback

“I think I felt that we needed to have 'a problem' to bring with us. I didn't understand that in the run up, so it felt difficult to apply it to something specific.” - Theory of Change workshop feedback

"Personally, I felt great value in the session, but using TOC may over complicate. It's a process of "reverse engineering” and this could have made it a little easier to grasp." - Theory of Change workshop feedback

The Theory of Change workshops were provided as a way of encouraging a strategic focus that we felt would help give arts organisations a strong starting point for a digital project. Judging by the feedback from the Theory of Change workshops, even amongst those who found the tool useful, some still felt that gaps in their knowledge made it difficult to use in their context.

Some organisations simply were not used to using planning tools that are perhaps less commonly used in the arts than in other fields. As discussed in previous sections, some did not have readily available information about their audiences, and others felt they needed a greater awareness of digital technology generally.

In the applications we received, this was reflected in the relative strength of applicants responses to questions about strategy compared to other criteria.

The following scores, relating to two of the seven scoring criteria, illustrate this point. Each of the six panel members were asked to score the all questions within the applications 0 if the criteria had not been met, 1 if the criteria had been partially met, and 2 if the criteria had been fully met.

• 'Is there a clear link between the stated objective and the applicant's strategy?'
• 16 applications (52 percent) received full marks (2/2) from three or more panel members.
• 7 applications (23 percent) received no marks (0/2) from one or more panel members.
• Three or more panel members gave the same score to 29 of the 31 applications (94 percent of cases).
• 'Is the problem clearly defined, with a measurable outcome?'
• 6 applications (19 percent) received full marks from three or more panel members.
• 24 applications (77 percent) received zero marks from one or more panel members.
• Three or more panel members gave the same score to 18 of the 31 applications (58 percent of cases).

It was clear that, whilst arts organisations have high levels of knowledge and skills in many areas, they were not as well equipped to clearly define problems in a way that is measurable, and that draws on data regularly produced and gathered internally. Without readily available data in order to inform decision-making - data from their own organisation or gathered from research by other bodies, such as the Arts Council of Wales - it is more of a challenge for organisations to make full use of tools like Theory of Change, and to undertake effective strategic planning more generally.

Further Study and Development

We are currently in the process of analysing data gathered during phase two of the fund, and in the early stages of delivery of phase three.

We continue to adapt delivery of the support for funded projects and the complementary activities for the wider sector, according to the feedback received to date. One example of this is to ensure we have a clear emphasis on the principles that underpin the tools and techniques we present, as we learned from the Theory of Change workshops.

In order to assess the impact and the value of the programme overall, the evaluation will focus on the following areas:

• What support arts organisations consider most useful in developing digital capacity and undertaking R&D.
• What impact there has been across the funded projects, on audience reach and business models.
• The extent to which insights from the funded projects have been picked up and applied by the wider arts sector.
• What the potential impact could be if these insights are applied by other organisations in the arts sector in Wales.

We will continue to assess the digital capacity and research skills of the organisations we have funded, and of the wider arts sector in Wales.

This should help us to develop recommendations that are applicable to arts organisations who have different levels of capacity and skills available to them, including those who have little or no prior experience of using digital technology in their work. We will also explore this subject in more detail with technology companies, to better understand how arts organisations can effectively collaborate with those who have digital expertise.

We will follow up with those applicants who engaged in phase one activities but that have not received funding, to see if and how they have been able to apply what they've learned, whether they have used Theory of Change in other planning contexts, whether they have continued in their efforts to develop digital capacity, and the barriers they have encountered.

In order to address the lack of arts-based examples of the use of Theory of Change, we intend to ask each of the funded arts organisations to produce both early-stage and fully developed Theory of Change diagrams as part of the research outputs from their project. These will be available for others to use as templates or for training purposes.

Further evaluation will allow us to draw comparisons with other funding streams that have introduced a stage of their process similar to the 'identify challenges' phase. We are in the process of identifying existing examples, and working directly with other funders to help them apply the same method in their context.

We would be very keen to discuss our continued research and evaluation with others who are supporting arts organisations to explore innovative use of digital technology. You can find out more about the ongoing activities of the fund by visiting http://innovation.arts.wales or http://arloesi.celf.cymru.

ANNEX

APPLICATION FORM

To be considered for the next stage of the Digital Innovation Fund for the Arts in Wales you need to complete the following application form. It asks you to tell us about the challenge or opportunity you want to tackle, why you're the right people to tackle it, the audience affected and what you might do during the next stage. It's divided into four sections and all questions are mandatory. To see how this fits into the wider programme of work, please refer back to our timetable.

At present, you can't save and return to this at a later date. If you want to draft your answers first, please download and fill in a copy of this form from the Arts Council of Wales website: http://innovation.arts.wales

Your Information

Standard contact information to be included here.

Part 1 The Problem or Opportunity

Any innovation project should have a clear objective a reason why it needs to be undertaken or a goal that it wants to achieve. It should also make sure that this is a long term objective, part of the ongoing vision of the organisation undertaking the project. Innovation happens because change is necessary. We want to know what the change is that you're looking to instigate and why.

Tell us about the problem you want to solve or the opportunity you want to take advantage of. We're looking for clearly articulated challenges that could have a broad impact on the wider arts sector. Remember that we're looking for projects that want to either:

• Enhance their audience reach or engagement; or
• Develop new business models.

What's the long term aim that you're trying to achieve? (100 words) This should be a SMART objective, e.g we want to increase attendance by people with visual impairments at arts events by making arts venues more accessible. By the end of next year, we want to reduce our reliance on public subsidy by finding new ways to increase our ticket yield.

What needs to change or happen for you to achieve your aim? (400 words) Be as specific as you can. There may be more than one thing that you need to do to affect the change that you want to happen (bullet points may help here). What assumptions are you making that need to be tested? What might stop you from achieving your objective? What are you assuming that you can't currently prove or backup? For example we're assuming that people with visual impairments want to attend more often and that they can afford to do so.

How will you know if you've achieved your goal? (200 words) What metrics, measurements or other information might you use? E.g. visits to arts venues by people with visual impairments increase by x percent over the next X months.

How big a problem or opportunity is this for the wider arts sector? (200 words) Do other organisations experience the same problem or could they also take advantage of the opportunity?

Part 2 Audiences, Customers and End Users

In this section we want to know more about the people that this is going to impact on your audiences, customers or end users. Regardless of your overarching objective (audience reach or new business models), you'll be interacting with customers and users throughout your project.

Who might take advantage of the new opportunity? Who is this a problem for? (200 words) Describe your potential audiences, customers or end users and the potential scale or size of the market. Is this a new set of audiences or an existing set of customers or endusers for your organisation? If they're an existing group, tell us how many of them there are and how they currently engage with you. If it's a new group, why do you want to engage with them?

What are customers, audiences or endusers currently doing to overcome this challenge? Or, why has nobody taken advantage of the opportunity? (200 words) Tell us about the hacks, workarounds or homemade solutions that people are using. Or, if this is a new opportunity, tell us why you think nobody's taken advantage of it yet.

Part 3 Your Organisation

In this section we want to know a bit more more about your organisation why you're the right people to undertake this work and how it fits with your strategic direction.

How does solving this problem fit with your organisational strategy? (200 words) We want to know whether this fits with your current objectives or if it's a diversification. If it's the latter, do you have support from your board to diversify your activities?

Why are you the right people to be tackling this challenge or opportunity? (200 words) Tell us how you might represent the wider sector when undertaking this project, how the lessons you learn will be replicable by others and why you're in the best position to share those lessons. Is there anyone else within the sector that you should be working with? We know the benefits of collaboration.

If you were to successfully make it through to the full R&D stage, who would manage your project? (100 words) Is there someone within your organisation who has both the skills and the time to undertake this or would you need to employ someone?

Part 4 The Project

In this final section, we want to know more about how you would spend the 14-week idea generation and team- building period and what sort of support you might need.

If your application was successful, how would you spend the £5,000 on offer? Please set out a brief budget, including an allocation of at least £2,000 to a technology partner. Please also note that there is no VAT payable on our grant to you. Any VAT costs that you incur must be included in the total budget.

In order to get the most out of the idea-generation and team building stage, you must attend the following workshops. Please confirm that at least two members of staff from your organisation will be able to attend workshops in Cardiff on the following dates:

• w/c 8 February 2016
• w/c 14 March 2016
• w/c 25 April 2016

(We can cover reasonable travel and accommodation expenses for this where necessary.)

APPLICATION SCORING GUIDANCE DIGITAL INNOVATION FUND FOR THE ARTS IN WALES

The following document sets out guidance for scorers assessing applications to the Digital Innovation Fund for the Arts in Wales.

The application process is set out to test how well the applicant knows the challenge, problem or opportunity they're looking to tackle; the extent to which this is of strategic importance to them; the extent to which this is important for the wider sector; and the extent to which they are the right people to tackle this on behalf of the sector.

When scoring applications we are assessing against 7 criteria. In each case you should score either 0 - the criteria has not been met; 1 - the criteria has been partially met; or 2 - the criteria has been fully met. Guidance on each of the criteria is given below.

Is the problem clearly defined, with a measurable outcome?

2 - The problem is clearly defined and a measurable outcome is logicaly linked back to a SMART objective. The applicant has demonstrated that they understand both the root cause and the symptoms of their challenge. 1 - The problem is broadly defined. The objective may be partially SMART - some but not all of the criteia for a SMART objective may be present. The application deals mainly with the symptoms but recognises that there is also a root cause. 0 - The objective isn't SMART and measurments may be superficial. There's no acknowledgement of the root cause of a challenge, just the symptoms that it creates.

How big a problem is this for the wider sector?

2 - This is a challenge faced by a significant proportion of the sector and this is acknowledged and explained by the applicant 1 - This is a challenge faced by a significant proportion of the sector but only partially articulated by the applicant and/or this is a challenge for a smaller proportion of the sector. 0 - This is a challenge faced solely by this organisation - there's likely to be few outcomes that are of benefit for the wider sector.

Does the applicant know who its audiences and end users are?

2 - Audiences and end users are clearly and specifically described. Their current behaviours are identified and articulated, backed up with some form of evidence. 1 - Audiences and end users are described but some key details may be missing (size of market, current levels of engagement). Current behaviours may be assumed, rather than evidenced, but assumptions are acknowledged. 0 - General information is provided or assumptions are made with no acknowledgment that more research is necessary. Little or no evidence is provided around current behaviours.

Is there a clear link between the stated objective and the applicant's strategy?

2 - There's a clear link between the stated objective and organisations strategy. The applicant demonstrates how achieving this objective will contribute to their organisation's strategy. If this is a diversification, there's a clear, logical rationale behind the decision to move into a different area. 1 - The link between the the stated objective and organisational strategy is broadly recognised although some ambiguity may remain. With additional clarification, a clear link between the stated objective and the organisational strategy may be made. Reasons for diversification may require further clarification. 0 - There's no link between the stated objective and organisational strategy. Diversification isn't recognised.

Are they the right people to be tackling this project on behalf of the wider sector?

2 - The organisation can clearly articulate their place in the sector, that they are well networked. Where relevant, they've found partners who add value to the project. 1 - The organisation is aware of their place in the wider sector but may not be aware of the breadth of the sector and/or may have limited engagement with it. They may have added partners but the value that they can bring may be limited or not clearly articulated. 0 - The organisation has little engagement with the wider sector. Partners have been added but there's no clear reason why, or no additional value that can be gained from the partnership.

Is the project management clearly linked back to the organisation and its strategy?

2 - The project will be managed internally and dedicated and appropriate resource is identified. The person identified has the skills to undertake the project management An external project manager will be brought in and a plan is articulated to ensure that their work is mainstreamed back into the organisation. 1 - The project is internally managed but more clarification is required around the amount of time or resource that is being offered, or the ability of the person to manage the project and embed the outcomes into the organisation. An external project manager is appointed but more clarification is required around how they will work with the organisation (some attempt is made to explain this). 0 - The person appointed to the project lacks the skills or mandate to embed the project in their organisation. There's no attempt to link an external project manager into the organisation - the project is likely to remain at arm's length.

Does the budget represent good value for money?

As well as a general sense that the budget provides value for money:

2 - The budget provides enough time for the project manager to fully engage in the next round of the programme. Day rates (where applicable) are within current accepted norms. An allocation of at least £2,000 has been made for the technology company. 1 - The budget provides some but not all the time necessary for the project manager, day rates may be higher than current accepted norms. 0 - The budget doesn't provide enough time for the project manager. Less than £2,000 is made available for the project manager.

APPLICATION FEEDBACK - DIGITAL INNOVATION FUND

Thank you for taking the time to give us feedback. We would like to hear from anyone who considered applying for the fund, even if you did not complete an application form.

The results of this survey will not inform funding decisions - no analysis of responses will be undertaken until after funding decisions are made.

1. About the application process

Please tell us which elements of the application process you completed:

• I completed the online application form.
• I looked at online information about the fund - on Arts Council Wales' website.
• I looked at online information about the fund - on Nesta's website.
• I received advice via email/telephone from the fund team (Rob/Lisa/Dan).
• I attended a workshop about the fund.
• I met face-to-face with the fund team (Rob/Lisa/Dan) to discuss my application.

How challenging did you find the application process?

1. 5 - Very challenging
2. 4
3. 3
4. 2
5. 1 - Not challenging at all

Please explain your answer

How useful did you find the application process?

1. 5 - Very useful
2. 4
3. 3
4. 2
5. 1 - Not useful at all

Please explain your answer

Was there anything about the application process you found helpful/unhelpful? Do you have any suggestions for improvements we could make?

How well did the Arts Council of Wales and Nesta manage your application for funding?

• Extremely well
• Well
• Not very well
• Not well at all
• N/A - I didn't apply

Please give us your feedback on our application process which will help us to improve our service.

The Digital Innovation Fund team will be delivering workshops and other events in 2016 to help arts organisations make use of digital technology. How likely is it that you will attend one of these events?

1. 5 - Very likely
2. 4
3. 3
4. 2
5. 1 - Not likely at all

2. About you

Which best describes your role?

• Board member
• Artistic Director
• Chief Executive
• Senior management
• Middle management
• Learning/engagement officer
• Freelance arts professional
• Creative consultant
• Digital technology professional
• Other:

Contact details (OPTIONAL)

If you would be happy to discuss your answers in more detail, please leave your name and contact details here. If you do not wish to be contacted, please leave this blank.

3. About your organisation

In which local authority area is your organisation based?

• Blaenau Gwent
• Bridgend
• Caerphilly
• Cardiff
• Carmarthenshire
• Ceredigion
• Conwy
• Denbighshire
• Flintshire
• Gwynedd
• Isle of Anglesey
• Merthyr Tydfil
• Monmouthshire
• Neath Port Talbot
• Newport
• Pembrokeshire
• Powys
• Rhondda Cynon Taf
• Swansea
• Torfaen
• Vale of Glamorgan
• Wrexham

How many employees does your organisation have? How many volunteers does your organisation have?

In which of the following artforms does your organisation work?

• Dance
• Drama
• Combined arts/Multi-disciplinary arts
• Music
• Opera
• Theatre entertainment
• Craft/applied art
• Visual arts
• Literature
• Public art
• Film production
• Film/Video screenings
• Other:

THEORY OF CHANGE WORKSHOP FEEDBACK