Risk analysis and expert judgment

Purpose: To provide an overview of problems with judging risk and how these can be overcome by groups using integrated elicitation processes.

Description: A brief overview is provided about distortions in risk perception stemming from factors such as overconfidence, degree of personal experience and dreadfulness of the outcome. These lead to a range of ‘pathologies’ such as insensitivity to sample size, anchoring and hindsight bias.

Although peer assessments of people’s ability to make accurate judgments show a high correlation with self-assessments, there is no correlation between peer assessments and actual performance.

An integrated elicitation process that can help groups make better judgments is briefly described. It uses a structured process to elicit information from groups that is relatively accurate and well-calibrated (ie., groups have a good assessment of their own level of uncertainty). The focus is on information that can be independently verified (ie., facts that are already known – but not by the group – and predictions).

Better judgments are achieved:
when individuals…

  • consider counter-arguments
  • answer the same question in different ways (estimate the lowest probability, estimate the highest probability, use these to make a best estimate)
  • indicate their confidence
  • examine estimates made by other people (feedback)
  • revise original estimates after feedback
  • anticipate issues with conditional probabilities, base rates
  • work in groups

and when groups…

  • discuss questions to eliminate linguistic uncertainty
  • are diverse—age, gender, background and cognitive style
  • discuss and revise question meaning
  • are composed of psychologically, culturally and contextually diverse, independent people
  • avoid group think, for example by using Delphi technique, independent data or anonymity in judgments.

Notes: The video was recorded at the First Global Conference on Research Integration and Implementation. The session was chaired by Howard Gadlin and questions were asked at the conclusion of the talk by Gerald Midgley, Deborah O’Connell, Tim Moore and Linda Neuhauser. Reference is also made in the talk to two other talks, by Hilary Bradbury on action research – see https://i2s.anu.edu.au/resources/action-research-and-transformation-knowledge-creation and by Michael Smithson on unknowns – see https://i2s.anu.edu.au/resources/unknowns-inter-and-trans-disciplinary-settings.

Video (28 minutes): “Expert Judgment in Risk Analysis” presented by Mark Burgman was a plenary talk at the First Global Conference on Research Integration and Implementation held in Canberra in Australia, online and at three co-conferences (Lueneburg in Germany, The Hague in the Netherlands and Montevideo in Uruguay), 8-11 September 2013. The Microsoft PowerPoint presentation from the video is available as a PDF (1.3MB).

Related tools on this website: N/A

Related tools on the i2Insights blog: 

Related topics on Wikipedia:

Posted: November 2014
Last modified: January 2020