Crowd Predictions: The forecasts in full
www.nesta.org.uk/feature/crowd-predictions-results/crowd-predictions-forecasts-full/
www.nesta.org.uk/feature/crowd-predictions-results/crowd-predictions-forecasts-full/
In the second part of our crowd predictions results series, we provide the results to all of the questions we asked the crowd. We report on the 13 they got right and what we can learn from the 6 they got wrong.
Over the course of 2019, our forecasters answered 11 questions about Brexit and 8 questions on other topics ranging from the likelihood of CRISPR babies to the incidence of measles in the US. Of these, they predicted correctly on eight Brexit questions and five others. In this piece we keep it simple and let the results speak for themselves.
Prediction polling, the method used by the Good Judgment Open platform, invites participants to make probabilistic estimates about the likelihood of different outcomes as related to a particular event. The outcomes that the participants choose between are mutually exclusive and so the total percentage across all of the outcomes needs to add up to 100%. The figure below shows the interface our participants used to make their predictions. The final crowd forecast was generated by aggregating individual estimates, based on a formula developed by Good Judgment during their original experiments. The crowd forecast automatically updates as more forecasts are added or older predictions are updated. In prediction polling, forecasters are encouraged to change their predictions frequently as new information becomes available. This flexible forecasting behaviour correlates with actively open-minded thinking and has been shown to increase accuracy and reduce polarisation.
The interface that forecasters use to make their forecasts
In late 2018, NASA announced its plans for the first commercial crewed missions to the International Space Station. Flights for SpaceX and Boeing were due to be launched for June and August 2019, respectively.
In the first 3 months of 2019, the US was facing one the most significant outbreak of measles that it had seen since declaring itself measles free.
The hurricane season in the US officially extends from June 1 to November 30. In May, a “near-normal” hurricane season was predicted by experts and statistical models. On average, a normal hurricane season has between 2 and 4 major hurricanes.
In May 2017, the acting US attorney general appointed Robert Mueller as special counsel to lead an investigation into the links between Donald Trump and Russian interference in the 2016 US presidential election. Throughout the investigation it was rumoured that the results could lead to presidential impeachment.
Facebook faced mounting criticism throughout 2018, culminating in a call by investors for Mark Zuckerberg to step down from his roles.
The forecasters got off to a great start to the challenge by predicting the most likely outcome on the first five Brexit-related questions that we posted. These included questions about the fate of Article 50 on 30 March, the results of the European Parliamentary elections in May and whether Theresa May would still be Prime Minister in July. We wrote about these in detail in The Conversation.
Below, we present the results of Brexit-related questions that haven’t been published before now.
After March 2019, the UK found itself facing a new Brexit deadline at the end of October. Following an announcement by the acting prime minister, Theresa May, that she would resign by the summer, the uncertainty about Brexit seemed higher than ever before. We gave the crowd seven options to choose between, all of which were relevant to the way that the UK might or might not exit the EU. These ranged from leaving the EU with or without a deal in various guises to calling a general election. What we wanted to know was which of these outcomes was most likely to happen first?
Following many months of parliamentary stalemate and increasingly fraught exchanges between the main political parties, the UK Parliament voted by a majority of 418 to hold an early general election on 12 December. Experts claimed that the election would be difficult to predict due to the unstable and dynamic nature of voter preferences, many of whom had switched their political allegiances in the previous two national elections.
Boris Johnson replaced Theresa May as the UK’s prime minister on 24 July following a month-long leadership competition between Conservative politicians. In early September, the prime minister declared that he would “rather be dead in a ditch than delay Brexit”. As it became increasingly clear that the October Brexit deadline would be extended, we asked whether the prime minister would be able to retain his position into the new year.
Despite the impressive performance of the forecasters, there were six questions where they got it wrong. These results also provide interesting insights into the potential and the limits of crowd predictions. We explore each of them in detail below.
The second deadliest outbreak of the viral disease Ebola first started in the Democratic Republic of Congo (DRC) in August 2018. Imported cases of the disease were first confirmed in Uganda, which borders the DRC, in June 2019.
Even though the forecasters pivoted to the correct answer in the final stages, their estimates leading up to this moment were wrong. This effect is related to a forecasting idea known as temporal scope sensitivity. The smaller the time window for a novel event to occur, the less likely it is to happen so the value of the prediction decreases closer to the closing date.
The result may partly be explained by the distance of most of the participating forecasters from the issue at hand. After all, the majority of forecasters were based in the UK and US. They could only experience developments related to Ebola through news reports rather than learning about the dynamics of the epidemic through their own lived experience. This stands in contrast with the ability of our crowd to accurately predict the total number of US measles cases in 2019, a topic much closer to home, where they had already made a decisive choice for the correct outcome by mid-September, three months before the question closed.
In late 2018, the Chinese scientist He Jiankui made headline news as the first to undertake human embryonic gene-editing using CRISPR technology. Already there were rumours of a second ongoing pregnancy that was due to come to full term in 2019.
The inability of the crowd to predict the correct outcome may have been due to the paucity of relevant information in the English language media and a low number of forecasters (only 3) based in China.
Impossible Food Inc and Beyond Meat were just two of the alternative meat brands that saw an uptick in their value in 2019. Many popular U.S. burger chains already offered alternative meat options by the middle of 2019, and with veganism reportedly on the rise in 2019, we wondered whether we would see the world’s most popular food chain introduce an alternative meat burger.
In September, McDonald’s announced that they would test a meat-free option using Beyond Meat’s burgers in 28 locations in Canada, but not in the U.S. Either based on this changing tide or perhaps signalling a deeper shift in consumer attitudes to food, our forecasting crowd were relatively certain (with ~70% likelihood or more until early December) that they would see an alternative protein burger on McDonald’s menus before 2020. Although they were ultimately proved wrong, their high level of certainty shows another potential function of crowd predictions: companies could use them to infer the level of public acceptance of certain outcomes over time. This result may show that sometimes public attitudes shift earlier than industry realises.
Throughout 2018 and into 2019, the exchange rate for the British Pound remained sensitive to developments in the Brexit negotiations. Expert forecasts differed depending on the anticipated outcome and how “soft” the final Brexit deal would be.
Early in 2019, there were concerns that Brexit uncertainty would have a negative impact on the UK housing market. September was chosen as the cut-off date as it was 6 months after the original Brexit deadline.
For both of these questions, the final consensus our forecasters arrived at were more pessimistic than the actual result. But even though the crowd considered these outcomes most likely relative to the other options, they both had probabilities of less than 50%. This represented a change from the first run of the currency question, which was tied to the original Article 50 deadline in March 2019. At that time our crowd decisively predicted the fate of the pound (with 96% likelihood). So what changed in the second half of the year?
Thinking back to the political upheaval over the summer months, it is easy to understand why our forecasters might have cast a more gloomy lens on events. July 2019 saw the appointment of a new Prime Minister whose provocative stance on Brexit seemed to spell the end of all prospects for a deal, parliament seemed even more divided but also determined to reign in the Government after an unexpected prorogation in early September.
Throughout the summer months, changes to UK house prices showed a noticeable dip to around 1% (see Figure 1) and the pound was similarly fluctuating around the 1.10 mark. Both of these values were on the boundary between outcomes that we were asking the crowd to choose between. In terms of currency, a decisive uptick occurred only on October 9th. On this day frontpage headlines included “That’s it then! PMs angry clash spells end of deal” [Daily Express] to “Day the deal was doomed” [Guardian]. This signal that the UK would need to continue negotiations with the EU beyond 31 October, might have acted as a trigger to reassure the markets and bring up the value of the pound.
Figure 1: Conversion rates between GBP and EUR. Adapted from screenshot of Bloomberg Online [Accessed 7 April 2020] and Changes to house prices, made by the author using data from the ONS/gov.uk.
In the build up to the General Election in December 2019, there were many discussions about which key seats were in danger of switching hands. We chose 10 of these (including Kensington, North East Fife and Workington), with incumbents and challengers across the broadest geographical range and political party spectrum. Forecasters had to choose between 5 different outcomes, from ‘None’ to ‘7 or more’
This question was probably the most demanding of all the ones we tried throughout the year. It required making a multi-stage estimate, first about the likelihood of each of the ten marginal seats moving between parties and then integrating these into an overall estimate of the total number of seats. As the chosen seats were evenly split between different incumbent parties and far and wide across the UK, it was far from a simple task. In the end, the crowd’s forecast of 5 or 6 as the most probable outcome was just one seat short of the reality.
For more discussion and reflection on lessons learned, we recommend reading other sections of the crowd results series.
In the final part of this series, we share what we learned about running large scale crowd predictions experiments.
Crowd Predictions: Lessons and advice
