Introduction

Every day, in every walk of life and every organisation, people work together in groups to make decisions. The reasons for doing this are many and fairly obvious – the access to more information, expertise and ideas can give the group advantages over individual decision makers. Teams that get it right are more productive, innovative and happier as a result. But making good decisions as a group can be hard.

When people work together on a problem to become more than the sum of their parts, it is known as collective intelligence. Research shows that a group's collective intelligence cannot be explained by the average IQ of its members and is better predicted by factors like the group diversity and social sensitivity. This is important because groups with high collective intelligence consistently outperform other teams when it comes to problem solving. During the 21st century, technology has transformed collective intelligence through methods like crowdsourcing that allow organisations to tap into more diverse expertise to support decision making. Drawing on the 'wisdom of crowds' can lead to better performance on tasks that require judgement, like forecasting risk, as well as creativity, like service innovation. Both types of tasks are important for decision making.

And yet, we can all think of examples of poor decisions made by groups: we may have been part of those groups ourselves, or observed them from outside. It is not unusual for organisations to fail to make the most of the skills and information distributed across their teams. Indeed, there is evidence that high profile disasters like the crash of the Challenger Space Shuttle in 1986 and the Deepwater Horizon Oil Spill in 2010 may have been avoided if the decision makers at NASA and BP had heeded the advice of engineers and workers on the ground.

Some of the factors that prevent groups from reaching their full potential are well known, such as social biases (like groupthink) or poor communication and competition between employees. There are also emerging challenges such as understanding how the rise of the remote workforce impacts collaboration and decision making. These problems are further complicated by increased uncertainty – as decision makers find themselves facing a combination of unexpected crises like the COVID-19 pandemic and complex long-term issues like climate change.

Despite the proven potential of group problem solving, the majority of research on decision making tends to focus on the individual rather than exploring how teams and organisations make decisions. In this report, we bring together some of the most accessible evidence from research about group decisions. We hope it can equip managers with the practical tactics they need to tap into the collective intelligence inside their organisations and beyond.

What is collective intelligence?

Collective intelligence is created when a group of diverse people work together, often with the help of technology, to mobilise a wider range of information, ideas and insights in order to solve a problem. It's based on the premise that intelligence is distributed. Different people hold different pieces of information and contribute different skills that, when combined, create a more complete picture of a problem and how to solve it.

About this report

The report is divided into five standalone sections, which correspond to different dimensions of group decisions. The first section provides an overview of the six most common decision rules and when to use them. After this, we turn to the fundamentals of group decision making: group composition, group dynamics, the decision making process and how to make robust decisions under uncertainty. You can read the sections in order or dip in and out. The resources at the end include five collective intelligence tools to get you started, a quick reference tool and a glossary of important terms.

In each section, we describe common pitfalls of group decisions alongside the practical 'tactics' for overcoming them. The tactics draw on evidence from a range of disciplines including management studies, computer science, social science and psychology, as well as behavioural insights. We provide key references at the end of each section.

We initially carried out this research to help Nesta explore how we might make our own decision-making processes and meetings as effective as possible. To do this, we undertook a rapid review of the literature on collective decision making over 12 days in December 2020. We prioritised well-established and widely replicated research but also included some tactics inspired by emerging findings. These are marked by. Although initially intended for internal use, we are sharing what we have learnt in case others might also find the insights useful.

Acknowledgements

David Robson, Imre Bard, Kathy Peach, Peter Baeck, Celia Hannon and Victoria Bew for conversations and advice that helped to shape the direction of this work and the final content of this report.

Who this report is for

We created this resource for managers (and their teams!) in the public sector, social enterprises, charities and foundations. It can help managers make the most of the talent in their teams, and tap into the diversity of experience, information and skills both within and external to their organisations.

The changes to decision making don't have to be overwhelming – it's possible to start small.

When viewing this document online, readers can scroll between different sections by clicking on the corresponding colour.

How to use it

Key takeaways: collective intelligence for good decisions

1. Diversity leads to better problem solving and decision making in groups. It's important to have diverse members, in terms of identity (demographics) and cognitive style (approaches to problem solving), as well as having different levels of experience and expertise represented. Groups made up of solely high-status or high-performing individuals find it difficult to collaborate, share information less effectively and struggle to stay focused on the task. In well-constructed mixed ability teams, the skills gap can be a powerful motivator for less able members, leading to overall productivity gains for the team.
2. A quick win for decision makers is to focus on developing cross-cutting skills within teams. For example, there is training to help individuals cultivate actively open minded thinking (AOMT), probabilistic reasoning, and perspective taking. These three skills increase an organisation's collective intelligence and transfer across decision making contexts. Groups with high cognitive flexibility or AOMT can adapt quickly and make better-informed decisions, while perspective taking and the ability to accurately estimate future outcomes help to correct for assumptions.
3. It's not always efficient for groups to push themselves to find the optimal solution or group consensus, and in many cases they don't need to. Using a majority decision rule is known to be reasonably robust for finding a good solution across many decision making contexts. 'Satisficing' helps to maintain quality under pressure by agreeing what is 'good enough' to make a minimum positive impact. An exception to this is decision making under uncertainty, where groups should reassess if the 'good enough' standard can still apply.
4. Increasing the size of the decision making group can help to increase diversity, skills and creativity. Evidence suggests a group size of five to seven people is optimal for decisions that require discussion and information exchange. Organisations could be much better at leveraging the wisdom of the crowd (groups of more than ten people) for certain decision making tasks where there's evidence of strong benefits. These include idea generation, prioritisation of options (especially eliminating bad options), and accurate forecasts.
5. Introducing intermittent breaks where group members work independently is known to improve problem solving for complex tasks. The best performing teams tend to have periods of intense communication with little or no interaction in between. Good communication is most important when sharing information during the early stages of decision making to make sure that all unique insights about the problem are shared. But when it comes to generating ideas or strategies, it's better for group members to work independently at first to avoid converging too early on a solution. Deliberately structuring group activities in this way may help to get the benefits of social learning while minimising biases like herding and the 'need for closure'.
6. When the external world is unstable, like during a financial crisis or political elections, traditional sources of expertise often fail due to overconfidence. Probability and forecasting training can help groups become more aware of their tendency to over- or underestimate. This will help them make more accurate judgements about the likelihood of different outcomes. Other techniques to manage uncertainty include taking a portfolio approach to decisions and using Robust Decision Making (RDM) to identify 'no-regret' options that yield benefits irrespective of what ends up happening.

The different dimensions of group decisions

Group composition: Who is in the group and what roles the different members play is a vital part of decision making. Group composition includes topics like diversity, skills, leadership, expertise, and group size.

Group dynamics: The behaviours that occur between members of the group impact on how effectively the group functions across different tasks, including decision making.

Decision making process: The decision making process can be broken down into sequential steps with many different tasks that range from creative to analytic. This report breaks down the decision making process into six stages, from goal setting to implementation.

Decision making rule: A decision rule describes the method used by the group to make the final decision. The choice of decision rule depends on the priorities of the group, some rules optimise for speed or accuracy, while others legitimacy and representation.

Uncertainty: Decisions can be affected by multiple sources of uncertainty. Learning to adapt the decision making process to make it more robust to uncertainty is a vital skill for decision makers.

Part 1 – Choosing a decision rule

Economic forecasts are used to inform strategy and planning across a variety of different sectors. Governments rely on them to set fiscal policy, while the public sector and charities draw on them to predict demand for services and infrastructure like housing or healthcare. Typically, decision makers rely on forecasts made by individual economists or some combination of individuals. But in 2019 researchers showed that even an economist with a good historical record cannot outperform the simple average of their peers. In other words, the aggregate forecasts of a group of economists (known as the 'wisdom of crowds') are consistently more accurate than any individual's estimate.

For any decision making process to work teams need to agree on a decision rule - the method used to make the final decision. The example above illustrates the use of the 'wisdom of crowds' aggregation rule.

It's important to choose your approach before you start and to make sure everyone involved knows what their role is. The choice of decision rule depends on the priorities of the group, as well as time and resourcing.

Six ways of arriving at the decision

Inspired by the Decider App developed by NOBL:

1. Consensus: everyone agrees
2. Democratic: majority rule
3. Consent: no one objects
4. Consultation: gathering advice
5. Aggregation: wisdom of crowds
6. Delegation: identify the best

Some rules optimise for speed or accuracy, while others prioritise legitimacy and representation. This section describes six different ways groups can arrive at a decision and how to choose between them.

See page 37 for a quick reference tool that will help your team choose the right decision rule for your problem.

1. Consensus: everyone agrees

Consensus decision making asks everyone in the group to help shape the decision so that the final solution incorporates everyone's perspectives and needs. Consensus decisions often take longer than other approaches to allow sufficient time for everyone to express their views.

When should I use it?

It works well when a decision will impact lots of people and those people have both valuable insight and a shared commitment to finding a solution that works for everyone. It tends to be the decision rule favoured by communities who have a set of common principles or goals like neighbourhood cooperatives or communities of practice.

2. Democratic: majority rule

Democratic decision making is when all members of the group vote on a set of options and the most popular choice is accepted as final. Democratic decisions can follow majority rules, where over 50 per cent support is required, or plurality rules, where the most popular option wins even if it doesn't clear the 50 per cent threshold.

When should I use it?

Democratic decisions work when the options are clear, all group members are well informed and everyone is willing to accept the majority rule. Research shows that majority voting is reasonably robust across many decision making contexts. An example of its use in everyday decision making is when members of a group independently rank or rate ideas before implementing the ones that receive the most support.

3. Consent: no one objects

Consent can be considered a version of consensus – it requires the whole group to be aligned for the decision to be approved. But instead of asking all members to agree, consent decisions can proceed if no one disagrees. One approach, known as integrative consent, invites group members to raise objections to a proposal. These cannot just be personal preferences and must demonstrate that the proposed approach will not achieve the goals of the group. All valid objections are then used to adapt the proposal and the decision is reached when there are no more objections.

When should I use it?

Consent works well when speed is a priority, a proposed action is clearly defined, and ideally, when the decision is reversible. Consent mechanisms prioritise clearing a minimum threshold rather than optimising for the best solution – this is known as satisficing. Consent is popular among engineering and technology firms because it attempts to combine both speed and inclusiveness.

4. Consultation: gathering advice

Consultative decision making gathers information and ideas about a decision from a relevant group of stakeholders. This input is then considered by a smaller group of decision makers, usually a senior leadership team or a group of elected officials. During consultation, advice is usually sourced independently and then integrated, but it can also involve inviting experts to provide consensus advice in a small group setting.

When should I use it?

Consultation is the basis of many citizen engagement processes across the public sector, where members of the public and organisations are invited to submit written responses to predefined surveys. (instead of questionnaires). It works well when unknowns can only be resolved by tapping into new sources of expertise and experience or to add legitimacy to "a decision". More recently, policy consultations have taken a deliberative turn with representative groups of citizens being brought together to discuss an issue before making their recommendations.

5. Aggregation: wisdom of crowds

Aggregating multiple judgements from many different people is more accurate than an average individual, particularly when the participating 'crowd' is diverse. Popular methods include prediction markets and prediction polls. In the former, individuals invest money in different outcomes and the option with the highest investment is taken as the most likely. In prediction polls, individuals assign a probability to each outcome, and the crowd prediction is determined by aggregating the probabilities.

When should I use it?

This method works best for estimating the likelihood of clearly defined future events, like election results or whether an extreme weather event, like a flood, will occur within a specified timeframe. Organisations are increasingly turning to the 'wisdom of crowds' of employees to help decide strategy or guide product launches, but they are still mostly used as an input to decision making by leaders. When taken seriously by leadership they can contribute to increasing buy-in and building valuable skills across the employee base.

6. Delegation: identify the best

Delegation means giving someone in the group explicit authority over making a decision, often with some constraints. This is often easier for decisions that operate in stable contexts where high-ability level and experience are a useful proxy for identifying the person best suited to making the decision.

When should I use it?

Delegation can work well when everyone agrees that a single member of the group has the best information or is best suited to make the decision. It is a good choice when there is little time, but the solution needs to pass a quality threshold. Giving one member of the group the authority to direct strategy helps groups to act faster and can free up time for group members to focus on other tasks or to throw their effort behind optimising the success of the chosen solution.

Key references

• Borek, A. J. and Abraham, C. (2018). How do Small Groups Promote Behaviour Change? An Integrative Conceptual Review of Explanatory Mechanisms. Applied Psychology: Health and Well-Being 10, pp. 30-61. doi: 10.1111/aphw.12120.
• Hastie, R., Kameda T. (2005). The robust beauty of majority rules in group decisions. Psychological Review, 112 (2), pp. 494-508. doi: 10.1037/0033-295X.112.2.494.
• Mellers B., Ungar L., Baron, J., Ramos, J., Gurcay, B., Fincher K., Scott, S. E., Moore, D., Atanasov, P., Swift, S.A., Murray, T., Stone, E., Tetlock, P. E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25(5), pp. 1106-15. doi: 10.1177/0956797614524255.
• NOBL, Decide Better Together, Link [last accessed March 2021].
• OECD (2020), Innovative Citizen Participation and New Democratic Institutions: Catching the Deliberative Wave, OECD Publishing, Paris, https://doi.org/10.1787/339306da-en.
• Qu, R., Timmermann, A. and Zhu, Y. (2019). Do Any Economists Have Superior Forecasting Skills? CEPR Discussion Paper No. DP14112, Available at SSRN: https://ssrn.com/abstract=3496601.

Part 2 – Group composition

In the aftermath of the attacks to the Twin Towers on the 11 September 2001, multiple investigations were launched to understand why the intelligence community had failed to stop them. And this wasn't the first time in the history of the CIA that the agency was under scrutiny for missing early warning signs, all with disastrous consequences.

The decisions made by intelligence agencies are notoriously complex. And while it's normal for individuals, no matter how well trained and educated, to have gaps (i.e. blind spots) in their judgement, a group of people who share similar perspectives and backgrounds are much more liable to share the same blind spots than groups of people with diverse perspectives and backgrounds. Homogeneous groups are also more likely to reinforce each other's judgements leading to compounding errors when they're wrong.

The investigations concluded that the failure to spot early clues and subsequent errors of judgement were associated with a lack of diversity in the groups who were making decisions. Analysts overwhelmingly came from similar backgrounds and areas of expertise... and so had, in all likelihood, similar blind spots.

Unfortunately, the CIA is not unique. Many public institutions and third sector organisations lack diversity in their workforce. This is particularly true for senior leadership teams, which tend to skew white and male. Paying attention to group composition, to decide the right size, who should be in the group and what roles the different members will play can affect the quality of decisions. In this section you will find tactics to help you assemble the right group for the task at hand.

Diversity

Problems

Homogenous groups are more likely to be overconfident and suffer from biases that impair information sharing. Groups with low identity-diversity (age, ethnicity, gender) can be prone to shared-information bias and confirmation bias.

When group members have similar backgrounds they are also more likely to overestimate their abilities and make errors in similar ways, which can lead to unwarranted risks. Increasing diversity, decreases these risks as members draw on different experiences to gather information and solve problems.

Cognitive style diversity relates to differences in how people think about problems and how to solve them – for example, analytical vs intuitive styles. Mixing cognitive styles improves performance on tasks that rely on both judgement and creativity, but these groups can struggle to arrive at a strategic consensus about how to approach a task.

Tactics

Managers should be deliberate about group composition and build diverse decision making teams. It's important to have diverse members, in terms of identity (demographics) and cognitive style (approaches to problem solving), as well as different levels of experience and expertise. Pay particular attention to:

• Gender diversity for complex problem solving, as having more women in a group increases social sensitivity, turn-taking during discussions and emotional awareness among members, which leads to higher collective intelligence.
• Identity diversity when you need to gather information about a complex problem.
• Cognitive style diversity for creative tasks such as idea generation as well as tasks where individuals' judgements are aggregated.

When forming groups with high diversity, take extra time to establish shared goals and agree communication norms. This enables strategic consensus about priorities when carrying out tasks.

Group size

Problems

Increasing the size of a decision making group is an obvious way to increase access to information and expertise. A larger group size by itself offers no guarantee of improvement in collective accuracy unless diversity and multiplicity of skills are maintained.

Maximally diverse large groups may struggle to reach their full potential when it comes to deliberation, achieving consensus and complex problem solving due to the challenge of coordinating between group members.

Tactics

Optimise group size according to the decision type, as well as available resources and time.

• For simple crowd judgement tasks there are linear gains in accuracy for groups of up to 20 people and minimal gains thereafter.
• For more complex problem solving, there is some evidence that smaller groups (<5) are better at tolerating uncertainty. But even these smaller groups need to maintain diversity to be successful.
• If a task requires finding a novel creative solution, the chances of discovering something that works increase with group size, as long as there is an efficient process for sharing, filtering and evaluating solutions.
• Groups with five to seven members need little formal coordination but still allow significant interaction to help establish trust between members. This makes them optimal for decisions that rely on information exchange and discussion.

Leaders and hierarchy

Problems

Groups made up of high-status individuals (leaders) aren't good at cooperating, share information less effectively and struggle to stay focused on the task.

This is also true for chickens. Research shows that groups of 'super chickens' can peck each other to death.

A flat group hierarchy can stall from excessive focus on reaching a compromise. This inhibits information sharing and distracts from collective goals as individuals vie for status and power.

Conflicting sources of hierarchy, for example two different members with the highest status or the most power respectively, can cause confusion and poor coordination.

Tactics

Incentivise collective goals and prompt leaders to collaborate Set collective KPIs to maintain attention on shared goals. Use a facilitator to prompt group members to share relevant information and remind high-status groups to focus on the task at hand to improve collaboration.

Assign leaders to help groups navigate complexity Procedurally complex, multi-stage decision making benefits from a good leader. A group hierarchy that is recognised as legitimate by the members helps reduce interpersonal sources of conflict.

Make hierarchies explicit early on Leadership roles should be assigned early and the accountability for decisions should be made explicit so that team members know who to defer to.

Experience

Problems

The longer that groups work together, the more likely they are to suffer from overconfidence. Overconfident groups are less flexible in response to changing external circumstances. This trait can lead to unwarranted risk-taking when decision makers face uncertainty.

Most teams are made up of members with mixed ability levels. But research shows that groups become demotivated when members think the gap between themselves and others is too large. This is why high-ability members are sometimes disruptive for group problem solving. When ability levels are more closely matched, the gap can be a powerful motivator for less able members, leading to overall productivity gains for the team. This is known as the Köhler effect.

Tactics

Set limits on tenure or regularly inject new ideas To reduce overconfidence, decision making groups should regularly rotate members or use crowdsourcing to draw on views from those with different experience.

Measure overconfidence to adjust risk estimates Test group members on their ability to judge their confidence accurately using unrelated problems with known answers. This will help decision makers to become more aware of their overconfidence and teach the group to moderate risk estimates accordingly.

Level the playing field Avoid creating groups where there is a large discrepancy in ability between members, unless a facilitator can help bridge the gap. Otherwise, level the playing field by providing extra resources and training to less experienced team members. You should also regularly rotate team membership so that the relative ability of individual members changes.

'Identify the best' If time is of the essence, teams should delegate idea generation to high-ability members, while others in the group support them to optimise their idea.

Key references

• Aggarwal, I., Woolley, A. W. (2018). Team Creativity, Cognition, and Cognitive Style Diversity. Management Science, 65 (4). doi: 10.1287/mnsc.2017.3001.
• Aggarwal, I., Woolley A. W., Chabris, C. F., Malone, T. W. (2019). The Impact of Cognitive Style Diversity on Implicit Learning in Teams. Frontiers in Psychology 10, p. 112. doi: 10.3389/fpsyg.2019.00112.
• Borek, A. J. and Abraham, C. (2018). How do Small Groups Promote Behaviour Change? An Integrative Conceptual Review of Explanatory Mechanisms. Applied Psychology: Health and Well Being 10, pp. 30-61. doi: 10.1111/aphw.12120.
• Boroş, S., van Gorp, L., Cardoen, B. et al. (2017). Breaking Silos: A Field Experiment on Relational Conflict Management in Cross-Functional Teams. Group Decision and Negotiation 26, pp. 327-356. doi: 10.1007/s10726-016-9487-5.
• Halevy, N., Chou, E, D. Galinsky, A. (2011). A functional model of hierarchy: Why, how, and when vertical differentiation enhances group performance. Organizational Psychology Review. 1(1), pp. 32-52. doi: 10.1177/2041386610380991.
• Hildreth, J. A. D., and Anderson, C. (2016). Failure at the top: How power undermines collaborative performance. Journal of Personality and Social Psychology, 110(2), pp. 261-286. doi: 10.1037/pspi0000045.
• Hong, L., Page, S. E. (2004). Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences of the United States of America, 101(46), pp.16385-9. doi: 10.1073/pnas.0403723101.
• Hong L., Page S., (October 2015). The Contributions of Diversity, Accuracy, and Group Size on Collective Accuracy. Available at SSRN: https://ssrn.com/abstract=3712299 or http://dx.doi.org/10.2139/ssrn.3712299 [last accessed March 2021].
• Kameda, T., Tsukasaki, T., Hastie, R., and Berg, N. (2011). Democracy under uncertainty: The wisdom of crowds and the free-rider problem in group decision making. Psychological Review, 118(1), pp. 76-96. doi: 10.1037/a0020699.
• Kahneman, D., Rosenfield, A. M., Gandhi, L., Blaser, T. (2016). Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making. Harvard Business Review. Link. [last accessed February 2021].
• Meissner, P., Schubert, M., Wulf, T. (2018). Determinants of group-level overconfidence in teams: A quasi-experimental investigation of diversity and tenure. Long Range Planning, 51(6), pp. 927-936. doi: 10.1016/j.lrp.2017.11.002.
• Meslec, N. and Curşeu, P. (2013). Too Close or Too Far Hurts: Cognitive Distance and Group Cognitive Synergy. Small Group Research. 44. doi: 10.1177/1046496413491988.
• Toyokawa, W., Whalen, A. and Laland, K.N. (2019). Social learning strategies regulate the wisdom and madness of interactive crowds. Nature Human Behaviour 3, 183-193 (2019). doi: 10.1038/s41562-018-0518-x.

Part 3 – Group dynamics

'Always on' communication is the calling card of the 21st-century workplace. Digital technology makes it easier than ever before for team members to connect with each other at any time of day, irrespective of their locations. But increasing the time that teams spend communicating isn't always beneficial for problem solving and collaboration. In fact, research shows that the best performing teams tend to have consolidated periods of intense communication with little or no interaction in between. Experiments with software development teams at Ditto, a US Fortune 500 company, showed that synchronising the activity of engineers to impose scheduled periods of 'quiet time' with no interruptions helped them to be more effective as a team, with around 60 per cent of engineers reporting above average productivity. Deciding on these patterns at the level of a team or department, as a form of collective time management, leads to better group outcomes than time management at the individual level. It helps members of a group to become more aware of the way their behaviour impacts on others, ultimately helping to build more cohesive teams.

In 2020, many workplaces were forced to transition to new remote working practices and team structures. Changes were made in record time but in many cases collaborative processes and the interaction between team members suffered as a result. Whether co-located or working as a distributed workforce, groups should not underestimate the importance of making deliberate choices about communication norms and team interactions.

Some adaptive social behaviours only occur in group settings. Social learning happens at the individual level, when group members improve their own skills by taking on new information from others or building on their ideas. Team learning is a collective process that helps groups to review their approach and avoid repeating mistakes. These behaviours are particularly important during the information sharing, idea generation and evaluation stages of decision making. This section includes tactics to help you optimise team and social learning, and describes how conflict, cohesion and communication affect group dynamics

Conflict, cohesion and communication

Problems

Conflict Disagreements are common in groups that have high cognitive diversity and, unless they are managed well, they get in the way of good decisions. This is especially true if conflict is caused by interpersonal clashes, which increase the chance of confirmation bias. But when disagreements stay focused on the task (e.g. having different views on strategy) they can lead to critical discussions that challenge assumptions and enable better decisions.

Cohesion Teams with greater task and interpersonal cohesion are more effective, and cohesion in management teams is associated with greater investment and growth. Remote working can cause lower emotional engagement between team members, leading to less cohesive groups.

Meetings also have an important part to play in creating cohesive teams. Cohesion suffers when meetings are too task-focused or don't make time for sharing non task-related information and celebrating success.

Communication It's very common to overestimate what other team members know, or to assume that everybody understands the problems or tasks in the same way, especially when teams work remotely or asynchronously. The balance between communicating too much and too little is difficult to get right but research suggests that teams with 'bursty' communication patterns lead to the best performance. Burstiness is when short intense interactions between group members are broken up by periods with little or no contact in between.

Tactics

Encourage task-related conflict but watch out for personal clashes Group leaders should learn to manage disagreements using cooperative resolution strategies. These include:

• Basing discussions on facts and multiple alternatives rather than reducing down to two opposing ideas.
• Balancing contributions to maintain a sense of fairness and avoid forcing consensus if possible.

Create 'virtual check-ins' Start each meeting with a check-in, having each member take a couple of minutes to discuss how they feel, what they are doing, what's going well and what's challenging. The online collaboration toolbox from Hyper Island suggests several check-in questions.

Celebrate success and encourage peer recognition Team leaders, and even peers, can encourage cohesion by regularly praising both individual and team performance during meetings.

Agree on communication etiquette upfront Negotiate team norms from the beginning, making sure you agree on preferred channels and frequency of communication. Also agree when you will be offline to allow group members enough time to work without interruptions.

Use collaborative note taking during meetings Making notes together in a shared document helps groups to build a common understanding of what needs to be done. It also helps to surface any differences in interpretation early on.

Adapt your approach to communication according to the task at hand Make sure that your team is well connected and communicating frequently during the information gathering phase of decision making. When you switch to idea generation, start off with independent working where there is little or no communication between members. This will help your team to come up with a greater variety of options.

Social learning

Problems

Often individuals learn the techniques they adopt from others, a behaviour known as social learning. This can be particularly useful for optimising existing solutions. But it can also bias individuals against looking for new solutions, so groups converge on an idea too soon. Larger groups are more likely to suffer from this.

Tactics

Balance exploitation and exploration to identify the best strategy Allocate sufficient time to explore new ideas independently before sharing as a group. Even after sharing, groups should keep trying to optimise or enhance ideas by recombining them in new ways.

Larger groups should be split into smaller teams with the explicit task of either exploitation (developing existing solutions) or exploration (continuing to generate new ideas), to make the most of both strategies.

Build deliberate breaks into group interactions Introducing intermittent breaks where group members work independently is known to improve problem solving for complex tasks. The best performing teams tend to have periods of intense activity with little or no interaction in between. Deliberately structuring interaction in this way helps groups to get the benefits of social learning while avoiding herding around suboptimal solutions.

Team learning

Problems

Effective decision making processes rely on team learning. This can only happen if groups take time to evaluate the success of their decisions and update their future strategy accordingly. Many teams struggle to discuss and learn from mistakes due to the absence of psychological safety and trust between members.

Tactics

Encourage team learning through team charters and inclusive leadership Leaders can increase psychological safety by actively asking members for input, and encouraging community members to discuss their mistakes in a constructive manner. Senior team members should encourage collective ownership of failures as well as successes. Agreeing community charters and group norms early on helps to increase psychological safety and trust.

Key references

• Berekméri, E., Zafeiris, A. (2020). Optimal collective decision making: consensus, accuracy and the effects of limited access to information. Scientific Reports 10, p. 16997. doi: 10.1038/s41598-020-73853-z.
• Bernstein, E., Shore, J., and Lazer, D. (2019). Improving the Rhythm of Your Collaboration. MIT Sloan Management Review. Link [last accessed March 2021].
• Boroş, S., van Gorp, L., Cardoen, B. et al. (2017). Breaking Silos: A Field Experiment on Relational Conflict Management in Cross-Functional Teams. Group Decision and Negotiation 26, pp. 327-356. doi: 10.1007/s10726-016-9487-5.
• Frank R.C. de Wit, F. R. C., Jehn, K. A., Scheepers, D. (2013). Task conflict, information processing, and decision-making: The damaging effect of relationship conflict. Organizational Behavior and Human Decision Processes, 122(2), pp. 177-189. doi: 10.1016/j.obhdp.2013.07.002.
• Gupta, P. and Woolley, A.W. (2018). Productivity in an era of multi-teaming: The role of information dashboards and shared cognition in team performance. Proceedings ACM Human-Computer Interaction, 2, 62. (CSCW 2018). https://doi.org/10.1145/3274331.
• Kozlowski, S. W. J. and Ilgen, D. R. (2006) Enhancing the Effectiveness of Work Groups and Teams, Psychological Science in the Interest, 7(3), pp. 77-124. doi: 10.1111/j.1529-1006.2006.00030.x.
• Maltarich, M. A., Kukenberger, M., Reilly, G., and Mathieu, J. (2018). Conflict in teams: Modeling early and late conflict states and the interactive effects of conflict processes. Group & Organization Management, 43(1), pp. 6-37. doi: 10.1177/1059601116681127.
• Perlow, L. (1999). The Time Famine: Toward a Sociology of Work Time. Administrative Science Quarterly, 44(1), pp. 57-81. doi: 10.2307/2667031.
• Perlow, L. (2014). Manage Your Team's Collective Time. Business Harvard Review. Link [last access March 2021].
• Toyokawa, W., Whalen, A. and Laland, K.N. (2019). Social learning strategies regulate the wisdom and madness of interactive crowds. Nature Human Behaviour 3, pp. 183-193. doi: 10.1038/s41562-018-0518-x.
• Watkins, M. (2013) Making Virtual Teams Work: Ten Basic Principles. Harvard Business Review. Link.
• Whillans, A., Perlow, L. and Turek, A. (2018) Experimenting during the shift to virtual team work: Learnings from how teams adapted their activities during the COVID-19 pandemic, Information and Organization, doi: 10.1016/J.infoandorg.2021.100343.

Part 4 – The decision making process

Frontline employees often have access to very different sources of information from senior teams of decision makers, yet they rarely get the chance to feed into strategic decisions. This means that organisations can miss out many relevant insights or creative ideas that could enrich the decision making process. Internal crowdsourcing and innovation platforms are helping to change this across a range of different sectors.

For example, the Biometrics and Information Sciences Department at AstraZeneca uses an internal crowdfunding platform to make decisions about which new products and services they should develop. Anyone from their 400-person team can pitch an idea as long as it meets an initial set of constraints. There is also a structured idea development process that invites others in the team to iterate on the pitched ideas, helping to make them more robust. Eventually, all members of the department rank the options by allocating their share of an internal virtual funding pot. The department uses this approach alongside their typical research and development pipeline. Making the decision making process transparent and inclusive has created greater engagement among employees, as well as cultivating higher quality ideas and a culture of continuous experimentation and learning.

The decision making process is a sequence of different tasks and it's important to consider them separately if you want to optimise your group's collective intelligence. This section separates decision making into six stages, from goal setting to implementation. It might not be relevant to go through all of them for every decision and the group members contributing to each stage might be different. For example, you might draw on crowdsourcing for idea generation, but then turn to smaller scale deliberation to evaluate different options. No matter what part of the process you are focusing on, remember to draw on the general principles of group composition and group dynamics to get the most out of your teams.

Goal setting

Problems

The absence of a clear strategic direction or purpose can get in the way of achieving collective goals during decision making. Weak leadership or poor quality interactions between group members can further decrease strategic alignment.

Early team interactions can send groups down a path of emphasising either task outcomes or work processes and these have different strengths. Teams who are process-focused commit fewer errors but show less agility when facing difficulties.

Without establishing 'who knows what' (both at individual and collective levels), groups and organisations can have a hard time establishing 'who will do what'.

Tactics

Align decision making with the collective mission Decision making criteria within the organisation should be consistent with the overall mission, organisational culture and messaging from leadership. Team leaders should be clear from the outset how the group's task will contribute to the decision making process - without overpromising.

Reward collective rather than individual performance Linking incentives to clear outcomes that can only be achieved through collective effort helps to reinforce shared goals.

Set goals that emphasise outcomes to stay agile and creative Teams who focus on outcomes tend to be more creative and innovative, which is useful in the face of decisions with a lot of uncertainty. But they also commit more errors as process takes a backseat.

Prompt group members to describe their skillset Newly-formed teams must start by sharing their expertise and skills to create a team awareness of 'who knows what'. This enables collaboration and better distribution of workload according to areas of expertise. Established groups should repeat this activity regularly, particularly if members work on multiple projects simultaneously.

Information gathering

Problems

Confirmation bias makes individuals more likely to look for evidence that confirms their views. Confirmation bias spreads easily among group members, especially when it is displayed by a leader.

Groups often fail to share information optimally. Group members are also more likely to share information they have in common rather than unique insights that help to shed new light on the issue. Newly-formed groups and fluid teams are particularly vulnerable to these effects.

Errors in judgement can occur when groups fail to consider enough relevant information or do not update their beliefs in the face of changing circumstances and new insights.

Tactics

Build in opportunities to revisit assumptions Before making strong commitments, groups should establish pre-planned 'break points' where they seek feedback and review the quality of evidence. Making this process transparent and open to external scrutiny - for example by others in the organisation - also helps to guard against confirmation bias. But it needs to happen while the evidence base is still a work in progress to be effective.

Incentivise group members to share novel or divergent information Group leaders or facilitators should remind group members to share conflicting or novel information during discussions. Set incentives, such as prizes for the most surprising fact, to encourage group members to share more diverse information.

Use crowdsourcing to increase the diversity of information and ideas The scale and diversity of information can be further increased by using crowdsourcing. There are many existing online platforms that can help organisations to tap into the wisdom of both internal and external crowds.

Agree on the standards of evidence needed to change your strategy in advance Discuss what type of evidence you would need to change a current strategy or belief. Agreeing this in advance of information sharing may help decision makers to become more flexible.

Idea generation

Problems

Brainstorming in group settings can lead to social loafing (when individuals rely on others to make contributions and exert less effort themselves), evaluation apprehension (the fear of ideas being ridiculed by other group members) and production blocking (individuals spend less time generating ideas than listening to others). These cancel out the potential creative benefits of brainstorming.

Teams are often faced with too many ideas to consider or conflicting interpretations of the generated ideas. Another risk is ending up with many solutions that are underdeveloped.

Experts are most consistent in narrowing down and refining options, but they often don't have the time to go through all options.

Tactics

Separate idea generation and discussion Ask group members to work on ideas independently first and then bring everyone together as a group to discuss them. This is known as the nominal group technique.

Don't spend too long on idea generation There is some evidence that there are diminishing returns for the quality of ideas in the later stages of ideation. Set a reasonable target, for example, that everyone in the group should contribute at least one idea. This will help you to access a diversity of ideas while maintaining quality.

Use facilitation to improve the depth of discussion Facilitators should ask questions that encourage the group to get into the detail of how an idea would work. This will help them to surface problems early, develop a shared understanding of what is involved and build on the ideas suggested by others to enhance their value.

Use crowdsourcing to reduce the options before putting them to decision makers Research suggests that non-expert crowds are better at eliminating bad options than identifying good ones, so the task should be framed accordingly. All Our Ideas is a free online tool for crowdsourced ranking of different options.

Idea evaluation

Problems

Group reinforcement or herding is when people self-censor and conform to the group majority view (even if they disagree). This can also lead to extreme positions being adopted and polarisation as members reinforce (rather than challenge) each other.

Inter-group opposition is when the pull towards group identity (and conformity) makes members reject external contributions, even if they are good ones.

Tactics

Allow people to submit questions or concerns anonymously The pressure to conform is easier to resist when people are not face to face. Digital crowdsourcing tools, like Slido, have made it easier for group members to submit divergent views or questions anonymously. This allows for a more critical debate. Groups should assign a facilitator to make sure they address all of the challenges raised.

Use collaborative red teaming Red teams are tasked with challenging assumptions and finding weaknesses in a proposal to foster debate. They should be introduced before a decision is finalised. Designating part of the group as a 'collaborative red team' is more effective than assigning this role to non-group members.

Decision selection

Problems

When group decisions are made under stress such as time constraints, members feel more pressure to conform to a majority view in order to reach a consensus decision more quickly. This heightened need for closure leads to divergent views and opposing evidence being silenced.

Groupthink or the pressure to conform to the majority view can lead to minority voices being overlooked as the final decision is made.

Tactics

Apply the 'good enough' standard to maintain quality under pressure It's not always efficient for groups to try to focus on finding the 'optimal' solution, and often it doesn't exist. In these situations, groups should satisfice and aim to get over a minimum threshold of positive impact. An exception to this general rule is decision making under uncertainty, where groups should reassess if the 'good enough' standard can still apply.

Split the group, then recombine for the final decision Splitting the group into subgroups that each recommend a decision helps to surface the minority opinions based on unique expertise or evidence that other group members don't have.

Execution & evaluation

Problems

There's a danger that groups stick to old solutions even when they no longer fit the problem. This absence of behavioural flexibility is more likely to occur during challenging tasks. The rate of herding around suboptimal strategies increases with group size.

The illusion of similarity is when groups have inaccurate assumptions about what others think and how they will respond to a decision. This can lead to premature decisions that fail during implementation.

Tactics

Continue to explore alternatives and review strategy even after a decision is made Nominate a small number of members (or an external team) to act as 'explorers' who anticipate changes in the external environment. For this approach to succeed, groups should agree a mechanism through which explorers are able to challenge existing strategies.

Test, learn and refine Where possible, use pilots to test slight variations of higher stakes decisions. Monitor results to get early feedback on what works before implementing a decision on a large scale.

Include those affected by the problem in risk assessments and evaluation Include the people implementing the decision, or those directly affected by it, to help anticipate risks and surface early warnings of failure so they can be corrected.

Key references

• Askarisichani, O., Huang, E. Y., Sato, K. S., Friedkin, N. E., Bullo, F. and Singh, A. K. (2020), Expertise and confidence explain how social influence evolves along intellective tasks. arXiv: 2011.07168.
• Bazazi, S., von Zimmermann, J., Bahrami, B., Richardson, D. (2019). Self-serving incentives impair collective decisions by increasing conformity. PLOS ONE 14(11). doi: 10.1371/journal.pone.0224725.
• Cultivate Labs. AstraZeneca Case Study (2021). Link [last accessed March 2021].
• Hallsworth, M., Egan, M., Rutter, J., and McCrae, J. (2018). Behavioural Government: Using behavioural science to improve how governments make decisions. The Behavioural Insights Team. Link.
• Hastie R., and Kameda T. (2005). The Robust Beauty of Majority Rules in Group Decisions. Psychological Review, 112(2), pp. 494-508. doi: 10.1037/0033-295X.112.2.494.
• Kerr, N. L. and Tindale, R. S. (2004). Group Performance and Decision Making. Annual Review of Psychology, 55(1), pp. 623-655. doi: 10.1146/annurev.psych.55.090902.142009.
• Klein, M., Bicharra Garcia, A. C. (2015). High-speed idea filtering with the bag of lemons. Decision Support Systems, 78, pp. 39-50. doi: 10.1016/j.dss.2015.06.005.
• Kozlowski, S. W. J. and Ilgen, D. R. (2006). Enhancing the Effectiveness of Work Groups and Teams, Psychological Science in the Public Interest, 7(3), pp. 77-124. doi: 10.1111/j.1529-1006.2006.00030.x.
• Lee, Cunningham J., Gino, F., Cable, D. and Staats, B. (2020). Seeing Oneself as a Valued Contributor: Social Worth Affirmation Improves Team Information Sharing. AMJ. doi: 10.5465/amj.2018.0790.
• Lejarraga, T. and Müller-Trede, J. (2017). When Experience Meets Description: How Dyads Integrate Experiential and Descriptive Information in Risky Decisions. Management Science, 63:6, pp. 1953-1971. doi: 10.1287/mnsc.2016.2428.
• Lewis, K. and Herndon, B. (2011). Transactive Memory Systems: Current Issues and Future Research Directions. Organization Science 22(5), pp. 1254-1265. doi: 10.1287/orsc.1110.0647.
• Malone, T. W. and Bernstein, M. (2015). Handbook of Collective Intelligence. The MIT Press. Chapter 6. ISBN: 978-0-262-02981-0.
• Pasquini, L., Steynor, A., and Waagsaether, K. (2019). The Psychology of Decision-Making Under Uncertainty: A Literature Review. ATLAS report. Link.
• Pilli, L. E., Mazzon, J. A. (2016). Information overload, choice deferral, and moderating role of need for cognition: Empirical evidence. Revista de Administração, 51(1), pp. 36-55. doi: 10.5700/rausp1222.
• Reinig, B.A., Briggs, R.O. (2008) On The Relationship Between Idea-Quantity and Idea-Quality During Ideation. Group Decis Negot 17, 403 (2008). https://doi.org/10.1007/s10726-008-9105-2.
• Riedl, C. and Woolley, A. W. (2017). Teams vs. Crowds: A Field Test of the Relative Contribution of Incentives, Member Ability, and Emergent Collaboration to Crowd-Based Problem Solving Performance. AMD (3), pp. 82-403, doi: 10.5465/amd.2015.0097.
• Seeber I., de Vreede G., Maier R. and Weber B. (2017). Beyond Brainstorming: Exploring Convergence in Teams. Journal of Management Information Systems, 34:4, pp. 939-969. doi: 10.1080/07421222.2017.1393303.
• Toyokawa, W., Whalen, A. and Laland, K.N. (2019). Social learning strategies regulate the wisdom and madness of interactive crowds. Nature Human Behaviour 3, pp. 183-193. doi: 10.1038/s41562-018-0518-x.
• Wisdom, T. N., Goldstone, R. L. (2011). Innovation, imitation, and problem-solving in a networked group. Nonlinear Dynamics, Psychology and Life Sciences. 15(2), pp. 229-52. PMID: 21382262.
• Woolley, A. W. (2009). Means vs. Ends: Implications of Process and Outcome Focus for Team Adaptation and Performance. Organization Science. 20, pp. 500-515. doi: 10.1287/orsc.1080.0382.

Part 5 – Uncertainty

During the COVID-19 pandemic, public health authorities and governments around the world scrambled to understand how the disease was spreading in real-time. But modelling infectious disease is difficult, whether it is a sudden-onset pandemic or annual flare-ups of influenza. It has to account for many different factors including the biological properties of a pathogen, weather patterns and human behaviour. In 2014, the US Centre for Disease Control launched an annual competition, known as FluSight, to incentivise the research community to improve the science and methods behind seasonal flu surveillance. The competition encouraged a diversity of approaches to be developed. The most accurate forecasts were put forward by the DELPHI group at Carnegie Mellon University. Their innovation was to use a combination of three different statistical models together with crowd forecasting. Volunteers across the country submitted their predictions for the spread of flu and these were aggregated into a 'wisdom of crowds' forecast which was used alongside the computational models. This mixed-methods approach allowed the team to account for multiple sources of uncertainty and minimise the limitations of any one approach.

In the wake of the pandemic, the team created COVIDcast, a tracker that predicts the spread of COVID-19. For this new iteration, they enriched their method by including crowdsourced data that tracks symptoms from more than 50,000 US citizens daily.

It's impossible to avoid uncertainty when it comes to strategic decisions. Senior leaders often turn to external consultants to help them navigate uncertainty rather than tapping into the skills already available within their organisations, or cultivating them through dedicated training.

This section takes you through the different steps a group should take to ensure decision making processes remain robust even under uncertainty. It describes how to adapt the decision making process to avoid biases that are amplified by uncertainty, as well as suggesting the training and tools that can help prepare decision makers at all levels in the organisation.

Identify the source of uncertainty

Acknowledge uncertainty and decide whether to resolve, delay or ignore.

Problems

The first step in managing uncertainty is acknowledging that it exists and understanding what is causing it. Uncertainty can interfere with the ability of the group to make the best decision in the present moment, for example if the group doesn't have access to accurate or up-to-date data about an issue. It can also emerge after implementation to influence if a chosen strategy fails or succeeds in the future.

Groups should identify the sources of both current and future uncertainty before deciding whether to try reducing the unknowns, delay the decision or ignore the uncertainty, depending on the level of risk.

Tactics

Agree if you can and should reduce uncertainty Try to quantify the level and source of uncertainty. For example, are you missing data or is the evidence you've gathered unreliable in some way? Then discuss if the uncertainty is significant enough to warrant further attention. Start by reviewing your group norms for decision making to make sure they still apply. Common tactics to reduce uncertainty include collecting additional information and advice, or assumption-based planning.

Avoid exploring uncertainty if it won't impact the decision Groups should ask 'would perfect information change the decision?'. Sometimes the level of uncertainty and associated risk are not significant enough to justify the time and additional resources needed to explore it. Before gathering further information to resolve uncertainty, groups should agree how much new information is needed to change the decision and if it is needed at all.

Carry out a premortem or backcasting to identify future sources of uncertainty A premortem asks decision makers to imagine that a decision they have made has failed. They should then identify five reasons for the failure that were within their control and five reasons that were not. Backcasting is similar but focuses on a successful outcome, and determining which factors impacted the success. Both techniques help decision makers to identify sources of uncertainty that are within their control.

Only use the most relevant tools

Use the methods and expertise that get to the heart of the unknowns.

Problems

The number and types of uncertainty analyses will depend on the sources of uncertainty. Choosing the wrong tool can cause groups to waste valuable time or even add to the uncertainty.

Research suggests that traditional experts fail when the external world becomes unstable, like during a financial crisis or political elections. When this happens, groups should consider novel data sources and using insights from people closer to the problem.

Tactics

Choose the right tool for the source of uncertainty Use statistical modelling and structured expert judgements to reduce uncertainty on the parts of the problem that can be predicted. Uncertainties caused by ambiguity or value differences require a more reflective approach. To frame these discussions, turn to methods that consider trade-offs like multi-criteria decision making, cost-benefit analysis, and ranking. To explore multiple scenarios that account for both analytical and values-based uncertainty, use tools like agent based modelling.

Tap into new sources of knowledge and expertise Widening the search for information is key for resolving unknowns. Collective intelligence methods are particularly well suited for helping to fill evidence gaps by tapping into new sources of expertise and data. Use crowdsourcing or crowd forecasting to shed new light on the problem.

Follow a structured protocol to get the most from the experts Experts are vulnerable to bias like everyone else and they often disagree. Using a structured elicitation protocol such as the Delphi method, makes expert advice as useful as possible. Research also suggests asking experts to provide an estimate or range where they have a high level of confidence in the outcome (e.g. 90-95 per cent). Then the group should agree a collective tolerance for risk by discussing what level of confidence they are comfortable with.

Beware of the 'need for closure'

Guard against premature decisions caused by emotional responses and bias.

Problems

Emotions influence the way we make decisions. For many people, uncertainty is associated with negative feelings like fear and anxiety. While this might seem obvious, it's rarely taken into account during decision making. Taking action to reduce uncertainty is often motivated by the desire to reduce negative feelings rather than to improve understanding of a complex issue.

The need for closure describes how long people are willing to remain in a state of uncertainty before needing to settle on a decision. If group members have a high personal need for closure, they should take care that it doesn't push them towards confirmation bias or settling on a shortlist of options too quickly.

Tactics

Introduce dissenting views gradually Capture individual views first, then form groups that have a mix of different perspectives. Building groups in this way helps to reconcile and integrate different perspectives. For complex or controversial topics, opposing views should be introduced into the discussion gradually. At each stage the group should recap their overall understanding of the problem. Following a highly structured process can help groups to cope with uncertainty.

Choose a balanced, adaptive approach

Spread your risk and pursue 'no regret' options.

Problems

It isn't always possible to reduce uncertainty or to delay a decision until the best course of action becomes more clear. This is often the case during a crisis like an extreme weather event or a public health emergency. It's even more important to guard against optimism bias, overconfidence and sunk cost bias when implementing decisions in these circumstances and remain sensitive to changes in the external environment. Optimism bias often leads us to overestimate the quality of decisions and their success while overconfidence can lead to higher risk decisions based on false assumptions.

Tactics

Explore multiple options and spread your risk If it's not possible to reduce the level of uncertainty, decision makers should try a portfolio approach, where multiple options are pursued at the same time. This helps to spread risk and gives organisations the flexibility to switch between strategies as circumstances change.

Pursue 'no-regret' options 'No-regret' options are those that yield benefits irrespective of what ends up happening. This way, the group can be sure that their decision will perform well under different future scenarios but it may require sacrificing the option with the highest potential impact.

The Robust Decision Making approach developed by RAND helps evaluate options based on their resilience to future uncertainty.

Build critical skills in the team

Train groups to develop the soft and technical skills that help with interpretation of risk and probability.

Problems

There are many biases that interfere with our ability to objectively assess evidence. Uncertain outcomes and risk are often assessed in terms of probability, and yet most of us struggle with making accurate estimates about the relative likelihood of different outcomes (this is known as probabilistic reasoning). Emerging decision support tools like AI and simulations often use confidence levels and probabilities to describe outcomes. These tools can help to simplify and reduce uncertainty but only if they are interpreted correctly by the end users.

Tactics

Train decision makers in forecasting and actively open minded thinking (AOMT) Statistical literacy and the ability to engage in futures thinking are key skills for decision makers. Probability training is more effective than scenario training to improve an individual's ability to think about the future, but both are better than no training. Even basic probability training can have lasting impacts on accuracy. Research also shows that forecasting as a team with shared goals results in higher accuracy.

Develop skills in perspective taking to correct for assumptions Perspective taking is a simple technique to reduce uncertainty about the actions and beliefs of others while avoiding quick, habitual judgements. It can help groups evaluate different options in a more balanced way. Prompt group members to think about how others in their social circle would feel about the decision. This leads to more accurate predictions about social beliefs and behaviours.

Invest time into learning new tools Advances in modelling, interactive visualisations and simulations are changing the way that we understand complex problems. These new tools can help groups explore the impact of different decisions as well as challenge their assumptions. Groups need to spend time developing graphical literacy skills and discussing the outputs from these tools as a group to ensure that they are interpreting them correctly.

Key references

• Jefferson, A., Bortolotti, L., Kuzmanovic, B. (2017). What is unrealistic optimism? Consciousness and Cognition 50, pp. 3-11. doi: 10.1016/j.concog.2016.10.005.
• Bernstein, E., Shore, J., Lazer, D. (2018). How intermittent breaks in interaction improve collective intelligence. Proceedings of the National Academy of Sciences, 115 (35), pp. 8734-8739. doi: 10.1073/pnas.1802407115.
• Brodbeck, F. C. et al. (2020). Group-level integrative complexity: Enhancing differentiation and integration in group decision-making. Group Processes & Intergroup Relations. doi: 10.1177/1368430219892698.
• Feldman Hall, O., Shenhav, Α. (2019). Resolving uncertainty in a social world. Nature human behaviour, 3(5), pp. 426-435. doi: 10.1038/s41562-019-0590-х.
• French, E. (ed) (2020). Decision Support Tools for Complex Decisions under Uncertainty. AU4DM. Link.
• Galesic, M., Bruine de Bruin, W. Election polls are more accurate if they ask participants how others will vote, The Conversation November 18th, 2020. Link.
• Hallo, L., Nguyen, T., Gorod, A., Tran, P. (2020). Effectiveness of Leadership Decision-Making in Complex Systems. Systems 2020, 8, 5. doi: 10.3390/systems8010005.
• Hemming, V., Walshe, T.V., Hanea, A.M., Fidler, F., Burgman, M.A. (2018). Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management. PLOS ONE 13(6): e0198468. doi: 10.1371/journal.pone.0198468.
• Ilgen, J. S., Teunissen, P. W., de Bruin, A. B. H., Bowen, J. L., Regehr, G. (2020). Warning bells: How clinicians leverage their discomfort to manage moments of uncertainty. Medical Education. doi: 10.1111/medu.14304.
• Kerr, N. L. and Tindale, R. S. (2004). Group Performance and Decision Making. Annual Review of Psychology, 55(1), pp. 623-655. doi: 10.1146/annurev.psych.55.090902.142009.
• Levontin, P., Walton, J.L., Kleineberg, J., Barons, M., French, S., Aufegger, L., McBride, M., Smith, J.Q., Barons, E., and Houssineau, J. (2020). Visualising Uncertainty: A Short Introduction. AU4DM, London UK. Link.
• Marold, J., Wagner, R., Schöbel, M., Manzey, D. (2012). Risk, uncertainty and decision making. Decision making in groups under uncertainty. Les cahiers de la sécurité industrielle. FonCSI. Link.
• Nussbaum, D. (2020). Practical Tools for Better Decisions: A Q&A with Annie Duke on How to Decide. Behavioural Scientist. Link.
• Pasquini, L., Steynor, A., and Waagsaether, K. (2019). The Psychology of Decision-Making Under Uncertainty: A Literature Review. ATLAS report. Link.
• Rield, C., Woolley, A. (2017). Teams vs. Crowds: A Field Test of the Relative Contribution of Incentives, Member Ability, and Emergent Collaboration to Crowd-Based Problem Solving Performance. AMD (3), pp. 82-403, doi: 10.5465/amd.2015.0097.
• Runge, M. C., Converse, S. J., Lyons, J. E. (2011). Which uncertainty? Using expert elicitation and expected value of information to design an adaptive program. Biological Conservation, 144(4), pp. 1214-1223. doi: 10.1016/j.biocon.2010.12.020.
• Tetlock, P. E. et al. (2014). Forecasting Tournaments: Tools for Increasing Transparency and Improving the Quality of Debate. Current Directions in Psychological Science, 23(4), pp. 290-295. doi: 10.1177/0963721414534257.
• Toyokawa, W., Whalen, A., Laland, K.N. (2019). Social learning strategies regulate the wisdom and madness of interactive crowds. Nature Human Behaviour 3, pp. 183-193. doi: 10.1038/s41562-018-0518-x.
• Wisdom, T. N., Goldstone, R. L. (2011). Innovation, imitation, and problem-solving in a networked group. Nonlinear Dynamics, Psychology and Life Sciences. 15(2), pp. 229-52. PMID: 21382262.

Resources

Collective intelligence tools to get you started

Digital technology is making it easier than ever before to mobilise the collective intelligence of entire organisations, or even tap into external crowds, to make better decisions. This selection of off-the-shelf collective intelligence tools is not exhaustive, but it will help you get started.

Choosing a decision rule

The Decider App is an online tool that will help you choose which decision rule is most suited to your problem. It takes users through a series of simple questions and makes a recommendation, as well as providing a comprehensive overview of the pros, cons and alternatives. When using it in your teams try filling it in individually and take time to discuss any differences of opinion to make sure that everyone understands how the decision will be made and why.

Idea generation and prioritisation

All Our Ideas is an open-source tool that allows decision makers to mix existing ideas with new insights from crowdsourcing. It's an example of a Wiki survey, which produces a ranked list of options based on pairwise comparisons of the full list of ideas. It can be used to capture input from larger groups across an organisation or to engage external crowds. For example, the New York City Mayor's Office of Long-Term Planning and Sustainability used the tool to crowdsource ideas for the city's sustainability initiative from residents.

Crowd predictions

Metaculus and Good Judgment Open are two open platforms for generating predictions about real-world events in politics, science and technology. Users can track their predictions over time to earn points and access resources to improve their forecasting accuracy. Cultivate Labs develops tailored crowd forecasting programmes for organisations who want to draw on the power of the crowd to help make strategic decisions.

Consensus building

Polis is an open-source system for gathering, analysing and understanding what large groups of people think in their own words, using asynchronous participation. It uses machine learning to highlight areas where opinions are very divided, as well as areas of consensus. This helps groups to identify common priorities when discussing polarising issues. Polis can be used to support deliberation between more than 1000 participants. The Taiwanese government uses the platform to run open consultations that help to determine regulatory policy.

Knowledge management

Wikis are collaborative knowledge commons that can be public, such as Wikipedia, or used by organisations to document their internal processes and develop collective resources. They enable groups to create shared repositories that can help to increase the transparency and coordination of decision making. The global grassroots movement, Public Lab, uses an open Wiki to collect and store information about environmental projects carried out by their members worldwide.

Managing group decisions from start to finish

Loomio is an online tool that helps teams to keep all ideas and discussions about a decision in one place. All parts of the decision making process are collected together as a 'thread' and can include deliberation, interactive idea generation and ranking of options. This increases the transparency of the process and helps prevent hindsight bias when reviewing past decisions. The P2P Foundation, an international community working on open-source software and open design standards, uses Loomio for high level coordination to help them document and delegate responsibility during complex decisions.

Additional collective intelligence tools can be found online in the open repository compiled by Nesta's Centre for Collective Intelligence Design.

Choosing a decision rule: quick reference tool

Adapted from the Collective Intelligence Design Playbook and inspired by the Decider App developed by NOBL.

Glossary

The terms in this glossary are those that are used multiple times throughout this report or refer to key concepts. Definitions for other terms, such as common behaviours and biases, are provided in situ.

Actively open minded thinking (AOMT): A trait marked by the fast integration of new information into current beliefs and a willingness to change your beliefs based on evidence.

Agent based modelling: A type of computational modelling based on simulating the actions of agents (for example, individuals or organisations) in an environment, to extrapolate about their effects on the system as a whole. It uses assumptions about the agents' beliefs and preferences to model behaviours.

Anchoring: A tendency to jump to conclusions by basing decisions on information or an idea gained early on in the decision making process. Also known as first-impression bias.

Burstiness: Burstiness is a measure of the pattern and frequency of a team's communication. When teams have high burstiness, they have short periods of intense communication separated by minimal interaction in between.

Cognitive style: A concept from psychology that refers to the way an individual prefers to think and learn.

Cognitive flexibility: The ability to switch between thinking about two different concepts or to think about multiple concepts simultaneously.

Collective intelligence: A measure of the intelligence that emerges from diverse groups. It predicts how well the group will perform on a range of problem solving tasks.

Crowdsourcing: Crowdsourcing is an umbrella term for a variety of approaches that source data, information, opinions or ideas from large crowds of people.

Delphi method: A method for structuring group communication where individuals respond anonymously to a question or survey. All contributions are aggregated and shared with the group, before each individual updates and resubmits their response. A Delphi process can go through multiple rounds of aggregation and updates.

Expert elicitation: Methods that gather and synthesise expert judgements about issues that are hard to predict.

Forecasting: Forecasting is a range of techniques that asks individuals to predict what they think will happen.

Group dynamics: The study of how the actions and behaviours within a group affect the way that the group functions as a whole.

Herding: A bias commonly seen in groups, when individuals take on the opinion of the majority rather than making their own judgement. It is associated with the pressure to conform.

Nominal group technique: A structured decision making process where group members first submit their opinions independently before discussing and prioritising them as a group.

Overconfidence: A bias where a person's subjective confidence in their judgements is reliably greater than the objective accuracy of those judgements.

Probabilistic reasoning: A method for dealing with uncertainty, by assigning probabilities to represent the likelihood of different outcomes.

Robust Decision Making: An iterative analytic framework that helps groups make decisions under uncertainty. It draws on combinations of decision analysis, assumption based planning and exploratory modelling.

Social sensitivity: The ability to perceive and understand the feelings and viewpoints of others.

Wisdom of crowds: Wisdom of crowds refers to a range of methods that aggregate individual judgements from large groups of people. These include prediction markets and crowd forecasting.

Other resources

Behavioural Government: Using behavioural science to improve how governments make decisions (2018) This Behavioural Insights Team report explores how biases in decision making in groups can be addressed or mitigated, with a specific focus on governments that are using behavioural insights to design, enhance and reassess their policies and services. It focuses on three core activities of policymaking: noticing, deliberating and executing.

The Collective Intelligence Design Playbook was designed by Nesta to help teams design and deliver collective intelligence projects. It provides an introduction to collective intelligence and illustrative case studies. It includes a collective intelligence design canvas, plus prompt cards and other activities to help you structure and stretch your thinking. The playbook is a resource for teams or groups working on how to harness the best ideas, information and insights to address a complex issue.

The Hyper Island Toolbox is a collection of activities and tools designed for teams that want to do things more creatively and collaboratively.

58 Victoria Embankment London EC4Y ODS +44 (0)20 7438 2500 [email protected] @nesta_uk www.facebook.com/nesta.uk www.nesta.org.uk

designbysoapbox.com

ISBN number: 978-1-913095-32-1