Over the course of 2019, our forecasters answered 11 questions about Brexit and 8 questions on other topics ranging from the likelihood of CRISPR babies to the incidence of measles in the US. Of these, they predicted correctly on eight Brexit questions and five others. In this piece we keep it simple and let the results speak for themselves.

The method

Prediction polling, the method used by the Good Judgment Open platform, invites participants to make probabilistic estimates about the likelihood of different outcomes as related to a particular event. The outcomes that the participants choose between are mutually exclusive and so the total percentage across all of the outcomes needs to add up to 100%. The figure below shows the interface our participants used to make their predictions. The final crowd forecast was generated by aggregating individual estimates, based on a formula developed by Good Judgment during their original experiments. The crowd forecast automatically updates as more forecasts are added or older predictions are updated. In prediction polling, forecasters are encouraged to change their predictions frequently as new information becomes available. This flexible forecasting behaviour correlates with actively open-minded thinking and has been shown to increase accuracy and reduce polarisation.

The interface that forecasters use to make their forecasts.

The interface that forecasters use to make their forecasts

The 13 questions where the crowd got it right

1. Will either SpaceX or Boeing launch its first crewed space mission in 2019?

In late 2018, NASA announced its plans for the first commercial crewed missions to the International Space Station. Flights for SpaceX and Boeing were due to be launched for June and August 2019, respectively.

  • What the crowd predicted: The first reports of changes to the original flight schedule started appearing in late March and continued throughout the year. The crowd assigned a high likelihood to ‘No’ from the second week of April (final forecast: 96% likelihood).
  • What happened: Both SpaceX and Boeing missions faced equipment faults and rescheduling of test flights throughout 2019. Towards the end of the year it was announced that the mission would be delayed to 2020.
Results graph

2. How many cases of measles will there be in the U.S. in 2019?

In the first 3 months of 2019, the US was facing one the most significant outbreak of measles that it had seen since declaring itself measles free.

  • What the crowd predicted: From mid-September the crowd forecast the most likely outcome as ‘Between 1,250 and 1,500 inclusive’ (final forecast: 49% likelihood)
  • What happened: The final number of measles cases reported by the U.S. Centre for Disease Control in early 2020 was 1,282, the highest number the country had seen since 1992.
Results graph

3. How many Category 4 or above hurricanes will occur in the Atlantic Ocean by 30 November 2019?

The hurricane season in the US officially extends from June 1 to November 30. In May, a “near-normal” hurricane season was predicted by experts and statistical models. On average, a normal hurricane season has between 2 and 4 major hurricanes.

  • What the crowd predicted: The forecasters assigned the highest likelihood to the ‘2’ hurricanes outcome from July although this was closely followed by ‘3’ until approximately the second week of October. (final forecast: 49% likelihood)
  • What happened: 2019 was the fourth consecutive year of above-average hurricane activity in the US. The National Hurricane Centre recorded two Category 5 hurricanes that developed in the Atlantic basin, Dorian (August-September) and Lorenzo (September-October).
Results graph

4. Will the U.S. House of Representatives impeach President Trump before 1 October 2019?

In May 2017, the acting US attorney general appointed Robert Mueller as special counsel to lead an investigation into the links between Donald Trump and Russian interference in the 2016 US presidential election. Throughout the investigation it was rumoured that the results could lead to presidential impeachment.

  • What the crowd predicted: The crowd assigned a low likelihood to the president getting impeached in that timeframe by mid-April, with the final crowd forecast of just 5% likelihood.
  • What happened: The Mueller report was published in April 2019 and did not result in any action against the president. Donald Trump was served the articles of impeachment by the House of Representatives in December 2019 under the alleged offence of soliciting foreign interference for the US presidential election.
Results graph

5. Will Mark Zuckerberg cease to be the sole Chairman or CEO of Facebook by 1 August 2019?

Facebook faced mounting criticism throughout 2018, culminating in a call by investors for Mark Zuckerberg to step down from his roles.

  • What the crowd predicted: The crowd correctly assigned a low likelihood to a change in Mark Zuckerburg’s roles at Facebook, with the final consensus at just 3% probability.
  • What happened: In April 2020, Mark Zuckerbeg remains the sole Chairman and CEO of Facebook.
Results graph

The forecasters got off to a great start to the challenge by predicting the most likely outcome on the first five Brexit-related questions that we posted. These included questions about the fate of Article 50 on 30 March, the results of the European Parliamentary elections in May and whether Theresa May would still be Prime Minister in July. We wrote about these in detail in The Conversation.

6. What will be the closing value for the pound against the euro on 1 April 2019?

Results graph

  • What the crowd predicted: The closing value would be between €1.10 - €1.20 (final forecast: 96% likelihood).
  • What happened: The closing value was €1.17.

7. Before 30 March 2019, what will happen next with regard to the UK's notification of Article 50?

Results graph

  • What the crowd predicted: Already in the first week of January 2019, the crowd forecast showed that an extension of Article 50 was the most likely outcome, versus Article 50 being revoked or the UK meeting the deadline to leave the EU by March 30, 2019. (Final forecast: 83% likelihood).
  • What happened: The European Commission granted a conditional extension of Article 50 until October 31, 2019.

8. If the UK participates in European Parliament elections, what percentage of votes will the Change UK Party win?

9. If the UK participates in European Parliament elections, what percentage of votes will the Brexit Party win?

Results graph

  • What the crowd predicted: The most likely vote share for the Brexit Party would be between 30% and 35%. A vote share of less than 5% was most probable for Change UK. These two questions had the quickest turnaround, they were open for the three weeks leading up to the May 22 election date. In both cases, after initial periods of high fluctuation, the crowd assigned the highest probability to the option containing the “winning” vote-share percentage almost a whole week before the public vote on May 22.
    We saw some interesting differences when we compared our crowd’s predictions for the Brexit Party vote share to the betting exchange platform, Smarkets (which uses the prediction market approach to forecasting). The Smarkets crowd assigned a much higher likelihood to a 35%+ vote share for the Brexit Party (40% at closing) whereas our crowd were much more conservative and only estimated a 17% probability for that outcome.
    On the other hand, the Smarkets crowd was considerably more confident than our crowd when it came to Change UK and closed with a 77% probability that they would win less than 5% vote share (our crowd said 55%).
  • What happened: the Brexit Party and Change UK received 30.74% and 3.31% of the vote share respectively.

10. Will there be a new prime minister of the United Kingdom before 1 July 2019?

Results graph

  • What the crowd predicted: This question, first posted on December 21, 2018 and live for six months, was a race of two halves. Our crowd of more than 2,500 forecasters made a decisive push for “No” (70% probability) by the second week in April 2019. This followed a two-week period of uncertainty where Yes and No were forecast as almost equally probable after the original deadline for Article 50. (final forecast: 82% likelihood).
  • What happened: Boris Johnson replaced Theresa May as the UK’s prime minister on 24 July following a month-long leadership competition between Conservative politicians.

Below, we present the results of Brexit-related questions that haven’t been published before now.

11. Between 3 May 2019 and 1 November 2019, which of the following will happen first in relation to the Brexit process?

After March 2019, the UK found itself facing a new Brexit deadline at the end of October. Following an announcement by the acting prime minister, Theresa May, that she would resign by the summer, the uncertainty about Brexit seemed higher than ever before. We gave the crowd seven options to choose between, all of which were relevant to the way that the UK might or might not exit the EU. These ranged from leaving the EU with or without a deal in various guises to calling a general election. What we wanted to know was which of these outcomes was most likely to happen first?

  • What the crowd predicted: This question featured 7 possible outcomes yet the crowd correctly predicted that Article 50 would be extended beyond 31 October 2019. This option was the one favoured by the crowd from early June, excluding a turbulent period in early September. During this time, the likelihood of a “No-deal Brexit” was briefly forecast as the most likely (final forecast: 36% likelihood).
  • What happened: On 19 October 2019 Boris Johnson, requested an extension of Article 50 until 31 January 2020. This 3-month extension was granted by the European Council on 28 October 2019.
Results graph

12. Will a political party gain an overall majority of seats in the House of Commons at the next UK General Election?

Following many months of parliamentary stalemate and increasingly fraught exchanges between the main political parties, the UK Parliament voted by a majority of 418 to hold an early general election on 12 December. Experts claimed that the election would be difficult to predict due to the unstable and dynamic nature of voter preferences, many of whom had switched their political allegiances in the previous two national elections.

  • What the crowd predicted: The forecasters already considered a Conservative majority as the most likely outcome to the election by mid-November. This estimate reached a peak of 75% likelihood by the first week of December. (final forecast: 67% likelihood)
  • What happened: The Conservative party secured a majority by gaining 66 new seats across the country.
Results graph

13. Will there be a new Prime Minister of the United Kingdom before 1 January 2020?

Boris Johnson replaced Theresa May as the UK’s prime minister on 24 July following a month-long leadership competition between Conservative politicians. In early September, the prime minister declared that he would “rather be dead in a ditch than delay Brexit”. As it became increasingly clear that the October Brexit deadline would be extended, we asked whether the prime minister would be able to retain his position into the new year.

  • What the crowd predicted: From early October, our forecasters assigned the highest likelihood to the outcome of Boris Johnson remaining as Prime Minister into the new year. (final forecast: 100% likelihood)
  • What happened: Boris Johnson fought a general election campaign at the end of 2019 as the leader Conservative Party. His party was elected with a majority on 12 December 2019, allowing him to retain the role of Prime Minister.
Results graph

The 6 questions the forecasters got wrong

Despite the impressive performance of the forecasters, there were six questions where they got it wrong. These results also provide interesting insights into the potential and the limits of crowd predictions. We explore each of them in detail below.

1. Will there be a case of Ebola outside the Democratic Republic of the Congo (DRC) and Uganda by 31 December 2019?

The second deadliest outbreak of the viral disease Ebola first started in the Democratic Republic of Congo (DRC) in August 2018. Imported cases of the disease were first confirmed in Uganda, which borders the DRC, in June 2019.

  • What the crowd predicted: For most of the period this question was online, the crowd thought it was most likely there would be a transmission to ‘a country bordering the DRC other than Uganda’. Although the likelihood of ‘No’ started to increase from November, it only became the most likely outcome in the final days of December.
  • What happened: To date there have been no further transmissions of Ebola reported beyond the DRC and Uganda.
Results graph

Even though the forecasters pivoted to the correct answer in the final stages, their estimates leading up to this moment were wrong. This effect is related to a forecasting idea known as temporal scope sensitivity. The smaller the time window for a novel event to occur, the less likely it is to happen so the value of the prediction decreases closer to the closing date.

The result may partly be explained by the distance of most of the participating forecasters from the issue at hand. After all, the majority of forecasters were based in the UK and US. They could only experience developments related to Ebola through news reports rather than learning about the dynamics of the epidemic through their own lived experience. This stands in contrast with the ability of our crowd to accurately predict the total number of US measles cases in 2019, a topic much closer to home, where they had already made a decisive choice for the correct outcome by mid-September, three months before the question closed.

2. How many CRISPR gene-edited babies will be born in 2019?

In late 2018, the Chinese scientist He Jiankui made headline news as the first to undertake human embryonic gene-editing using CRISPR technology. Already there were rumours of a second ongoing pregnancy that was due to come to full term in 2019.

  • What the crowd predicted: Although the crowd had fluctuated between three possible outcomes throughout the year, in the final months they gravitated towards the incorrect outcome of ‘0’ as the most likely, with a final consensus estimate of 54%.
  • What happened: The final answer was that one CRISPR gene-edited baby was born . This question proved challenging to resolve but using a combination of scientific journals and Chinese language reports on the outcome of He Jiankui’s trial in China, we determined that the pregnancy had been carried to term.

The inability of the crowd to predict the correct outcome may have been due to the paucity of relevant information in the English language media and a low number of forecasters (only 3) based in China.

Results graph

3. Will McDonald's announce an alternative meat burger in the United States?

Impossible Food Inc and Beyond Meat were just two of the alternative meat brands that saw an uptick in their value in 2019. Many popular U.S. burger chains already offered alternative meat options by the middle of 2019, and with veganism reportedly on the rise in 2019, we wondered whether we would see the world’s most popular food chain introduce an alternative meat burger.

  • What the crowd predicted: ‘Yes’ (final forecast: 55% probability)
  • What happened: McDonald’s did not introduce a meat-free burger in their U.S. restaurants in 2019.
Results graph

In September, McDonald’s announced that they would test a meat-free option using Beyond Meat’s burgers in 28 locations in Canada, but not in the U.S. Either based on this changing tide or perhaps signalling a deeper shift in consumer attitudes to food, our forecasting crowd were relatively certain (with ~70% likelihood or more until early December) that they would see an alternative protein burger on McDonald’s menus before 2020. Although they were ultimately proved wrong, their high level of certainty shows another potential function of crowd predictions: companies could use them to infer the level of public acceptance of certain outcomes over time. This result may show that sometimes public attitudes shift earlier than industry realises.

4. What will be the closing value for the pound against the euro on 1 November 2019?

Throughout 2018 and into 2019, the exchange rate for the British Pound remained sensitive to developments in the Brexit negotiations. Expert forecasts differed depending on the anticipated outcome and how “soft” the final Brexit deal would be.

  • What the crowd predicted: The closing value would be between €1.00 - €1.10 (final forecast: 41% probability).
  • What happened: The closing value was €1.15. [More than 1.10 but less than 1.20]
Results graph

5. What will be the percentage change in average UK house prices in September 2019?

Early in 2019, there were concerns that Brexit uncertainty would have a negative impact on the UK housing market. September was chosen as the cut-off date as it was 6 months after the original Brexit deadline.

  • What the crowd predicted: The percentage change would be More than -1.0% but less than 1.0% (final forecast: 38% probability).
  • What happened: The 12-month percentage change in September 2019 was 1.4% [Between 1.0% and 3.0%, inclusive]
Results graph

For both of these questions, the final consensus our forecasters arrived at were more pessimistic than the actual result. But even though the crowd considered these outcomes most likely relative to the other options, they both had probabilities of less than 50%. This represented a change from the first run of the currency question, which was tied to the original Article 50 deadline in March 2019. At that time our crowd decisively predicted the fate of the pound (with 96% likelihood). So what changed in the second half of the year?

Thinking back to the political upheaval over the summer months, it is easy to understand why our forecasters might have cast a more gloomy lens on events. July 2019 saw the appointment of a new Prime Minister whose provocative stance on Brexit seemed to spell the end of all prospects for a deal, parliament seemed even more divided but also determined to reign in the Government after an unexpected prorogation in early September.

Throughout the summer months, changes to UK house prices showed a noticeable dip to around 1% (see Figure 1) and the pound was similarly fluctuating around the 1.10 mark. Both of these values were on the boundary between outcomes that we were asking the crowd to choose between. In terms of currency, a decisive uptick occurred only on October 9th. On this day frontpage headlines included “That’s it then! PMs angry clash spells end of deal” [Daily Express] to “Day the deal was doomed” [Guardian]. This signal that the UK would need to continue negotiations with the EU beyond 31 October, might have acted as a trigger to reassure the markets and bring up the value of the pound.

Results graph

Figure 1: Conversion rates between GBP and EUR. Adapted from screenshot of Bloomberg Online [Accessed 7 April 2020] and Changes to house prices, made by the author using data from the ONS/gov.uk.

6. How many of 10 marginal seats in the House of Commons will change parties in the upcoming 2019 UK elections?

In the build up to the General Election in December 2019, there were many discussions about which key seats were in danger of switching hands. We chose 10 of these (including Kensington, North East Fife and Workington), with incumbents and challengers across the broadest geographical range and political party spectrum. Forecasters had to choose between 5 different outcomes, from ‘None’ to ‘7 or more’

  • What the crowd predicted: 5 or 6 seats would change hands (Final forecast: 39% probability).
  • What happened: 7 of the seats in question changed hands.
Results graph

This question was probably the most demanding of all the ones we tried throughout the year. It required making a multi-stage estimate, first about the likelihood of each of the ten marginal seats moving between parties and then integrating these into an overall estimate of the total number of seats. As the chosen seats were evenly split between different incumbent parties and far and wide across the UK, it was far from a simple task. In the end, the crowd’s forecast of 5 or 6 as the most probable outcome was just one seat short of the reality.

For more discussion and reflection on lessons learned, we recommend reading other sections of the crowd results series.

In the final part of this series, we share what we learned about running large scale crowd predictions experiments.

Crowd Predictions: Lessons and advice