Three tips we learnt from benchmarking Claude as a quant analyst
www.nesta.org.uk/blog/three-tips-we-learnt-from-benchmarking-claude-as-a-quant-analyst/
www.nesta.org.uk/blog/three-tips-we-learnt-from-benchmarking-claude-as-a-quant-analyst/
Artificial intelligence (AI) coding agents like Claude Code are being rapidly adopted by quantitative researchers and data scientists. These tools show incredible promise to speed up our work and at Nesta we are actively experimenting with them to see how we can best boost our efficiency, speed and capacity to innovate while maintaining quality and rigour. The boost to coding efficiency and speed has been clear, as our teams harness coding agents to build out infrastructure, productionise existing work and generate analysis.
An open question we’ve been grappling with is how reliable these agents are as analysts themselves: not just building the scaffolding but setting the research direction and executing end-to-end. If successful, the time to generate insights across our work could become negligible. But the risk is that confidently asserted wrong answers propagate, and validating the findings takes more time and resources for a team who haven’t spent the time exploring and thinking about a problem themselves.
We’ve pulled various analytical work our team has undertaken into a set of internal benchmarks for coding agents, to rigorously decide where across the analysis lifecycle we should deploy these tools. Our results show which analytical jobs you could hand over to an AI quantitative analyst today and which still need a human holding the pen.
Task: Take this stack of messy administrative spreadsheets and create a single source of truth.
100% of runs cleaned, joined and standardised the stack into a coherent set of panel data.
Task: Take this dataset and build a classification model, optimising for F1 on a hold-out test set.
Our human benchmark was beaten every run, through a combination of feature engineering and model optimisation.
These tasks are well-defined and checkable, and across them the agents performed extremely well. The common thread is that by giving a clear plan and a definition of success, the model can check itself against that. If you’ve done the thinking, the AI quant analyst can do the build.
Task: Take this stack of messy administrative spreadsheets and create a single source of truth.
The same task as before, but where the dataset had a fundamental shift in reporting and methodology in the middle of the series. While prompted to “deal with changes in reporting and methodology as appropriate,” not a single run succeeded in creating a comparable series. All runs returned a clean, confident, wrong dataset. Getting more specific and specifying the way a “reporting and methodology change in [year], adjust the series as appropriate” led to slight improvements, with around 20% of runs succeeding. Only when we spelled out the exact definitional shift did every run handle it correctly.
We saw similar failures on complex, open-ended problems of data matching, cleaning and aggregation that needed deeper domain knowledge or external context to succeed, with success rates hovering around 20 to 30%. These failures are invisible without domain knowledge and the work of a good human analyst is to dig into a problem, understand it and adapt accordingly. Coding agents don’t yet show this same thoroughness.
Which statistical model is appropriate? Which area of a research question should we dig into? A lot of quantitative analysis has no single correct answer. There are a variety of approaches that could be considered valid, whether being applied by a human or not. We gleaned insights looking across the distribution of approaches made by the agent.
When the AI quant analyst was asked to estimate the average marginal effects on features of a survey instrument, its approach differed substantially depending on how the data was used. Asked to use R, it fitted a mixed model reporting individual-level effects on every run; in Python it reported population-level effects. Neither choice is wrong, but the choice was clearly driven by the most popular libraries available in each language, not by the research question at hand.
On a separate difference-in-differences reanalysis, modelling choices swung the headline coefficient by up to 100%. This wasn’t inherently wrong, but it was worrying that our original team of analysts determined that core assumptions underpinning the model were violated, yet not a single agent run did, even when model structure and estimates were nearly identical. While there were caveats or concerns raised in some runs, even when evidence pointed towards violation most output runs framed results as reassuring and inconclusive.
This erroneous confidence was seen across other problems. For example, when asked to decompose model contributions to an ensemble, a completely incorrect but strongly asserted approach was returned by the AI quant analyst. Reasonable on its face and description, but was clearly fatally flawed when the actual code implementation was reviewed: what it claimed to do wasn’t happening in the code and what was happening in the code was not a valid approach.
Once you go beyond simple, exploratory analysis problems, the work becomes increasingly complex. Understanding the data, clarifying your assumptions and specifying your problem should be taken with care.
As tested, AI agents often make invisible and confident decisions that are not correct frequently enough. They are capable builders, not careful methodologists and you still need a human in control.
We’ve rolled out enterprise Claude Code across the organisation, and we’re exploring how to harness this and other tools to push the boundaries of safe and efficient data science.
Our team is focused on applying these tools on the types of problems where they excel.
The pace of developments in this space is rapidly accelerating. New skills, harnesses and other tools are constantly being released, and we are exploring how best practices from software engineering and other cutting-edge sectors can boost our analytical and data science work.
From our benchmarking, we aren’t handing over the analytical thinking to AI just yet. Questions that require judgement, domain knowledge, or methodological expertise still need a human hand.
If you’re using agents on analytical problems, have a clear analysis plan and iterate the methods and approach yourself based on your own exploration of the data and knowledge of the problem. Let them focus on the build, but ensure you’re doing the thinking. Ensure that you can quality assure whatever they produce.
Conversely, use agents as first-stop quality assurance on your own work. Ask it to reproduce your results, and investigate where it can’t. Use it to challenge your thinking, without fully handing over the work.
AI performance isn’t static. We created these benchmarks so we can not just test once, but continually assess the performance of new models and tools. We originally ran these tests on Opus 4.6. We’ve now tested it on Fable 5, with no visible shifts in results.
However, we expect that analytical harnesses for coding agents will be developed and mature similar to the recent glut of developer skillsets for Claude, AI-native IDEs and other software engineering tools. Our team is already experimenting with new skills, subagents and complex swarms of agents. We expect these could meaningfully push the needle on what’s possible, more likely than new frontier model developments. Our benchmarks enable us to continually measure progress and confidently push ourselves to test the limits of these new tools.
If the work your team is doing is complex and you’re starting to lean on AI to streamline this work, it might be time to build your own set of benchmarks. Traditional frontier model benchmarking doesn’t reflect the messy reality of our work, and that might be true for your domain, whatever it is. A suite of tests on discrete tasks can give you the confidence on where to go hard with AI and where to press the brakes.
Our team is now exploring how more structured use of coding agents could improve performance on these benchmarks. A coherent flow of skills or a swarm of subagents might create more competent and effective AI analysts. These benchmarks give us the freedom to develop and test new internal tools and approaches.
If you'd like to compare notes or use our approach, get in touch here.
