The reliability of aggregated probability judgments obtained through Cooke's classical model

Abstract

Purpose

The purpose of this paper is to compare various linear opinion pooling models for aggregating probability judgments and to determine whether Cooke's performance weighting model can sift out better calibrated experts and produce better aggregated distribution.

Design/methodology/approach

The leave‐one‐out cross‐validation technique is adopted to perform an out‐of‐sample comparison of Cooke's classical model, the equal weight linear pooling method, and the best expert approach.

Findings

Both aggregation models significantly outperform the best expert approach, indicating the need for inputs from multiple experts. The performance score for Cooke's classical model drops considerably in out‐of‐sample analysis, indicating that Cooke's performance weight approach might have been slightly overrated before, and the performance weight aggregation method no longer dominantly outperforms the equal weight linear opinion pool.

Research limitations/implications

The results show that using seed questions to sift out better calibrated experts may still be a feasible approach. However, because the superiority of Cooke's model as discussed in previous studies can no longer be claimed, whether the cost of extra efforts used in generating and evaluating seed questions is justifiable remains a question.

Originality/value

Understanding the performance of various models for aggregating experts' probability judgments is critical for decision and risk analysis. Furthermore, the leave‐one‐out cross‐validation technique used in this study achieves more objective evaluations than previous studies.

Citation

Lin, S. and Cheng, C. (2009), "The reliability of aggregated probability judgments obtained through Cooke's classical model", Journal of Modelling in Management, Vol. 4 No. 2, pp. 149-161. https://doi.org/10.1108/17465660910973961