A comparison of elicitation methods for probabilistic multiple hypothesis revision

Abstract

The evaluation of competing hypotheses is an essential aspect of the audit process. The method of evaluation and re-evaluation may have implications for both efficiency and effectiveness. This paper presents the results of a field experiment using a case study set in the context of a fraud investigation in which practicing auditors were required to engage in multiple hypothesis probability estimation and revision regarding the perpetrator of the fraud. The experiment examined the effect of two different methods of facilitating multiple hypothesis probability estimation and revision consistent with the completeness and complementarity norms of probability theory as it applies to the independence versus dependence of competing hypotheses and with the prescriptions of Bayes' Theorem. The first method was to have participants use linear probability elicitation scales and receive prior tutoring in probability theory emphasizing the axioms of completeness and complementarity. The second method was to provide a graphical decision aid, without prior tutoring, to aid the participants in expressing their responses. A third condition in which participants used linear probability elicitation scales but received no tutoring in probability theory, provided a benchmark against which to assess the effects of the two treatments.

Participants receiving prior tutoring in probability theory and using linear probability elicitation scales complied in their estimations and revisions with the probability axioms of completeness and complementarity. However, they engaged in frequent violations of the normative probability model and of Bayes' Theorem. They did not distribute changes in the probability of the target hypothesis to the nontarget hypotheses, and they engaged in “eliminations and resuscitations” whereby they eliminated a suspect by assigning a zero probability to that suspect at an intermediate iteration and resuscitated that suspect by reassigning him or her a positive probability at a later iteration. The participants using the graphical decision aids, by construction, did not violate the probability axioms of completeness and complementarity. However, with no imposed constraints, the patterns of their revisions were different. When they revised the probability of the target hypothesis, they revised the probabilities of the nontarget hypotheses. They did not engage in eliminations and resuscitations. These patterns are more consistent with the norms of probability theory and with Bayes' Theorem. Possible explanations of this phenomenon are proposed and discussed, including implications for audit practice and future research.

Citation

Emby, C. (2009), "A comparison of elicitation methods for probabilistic multiple hypothesis revision", Arnold, V. (Ed.) Advances in Accounting Behavioral Research (Advances in Accounting Behavioural Research, Vol. 12), Emerald Group Publishing Limited, Leeds, pp. 85-108. https://doi.org/10.1108/S1475-1488(2009)0000012007