To read the full version of this content please select one of the options below:

A panel for lemons? Positivity bias, reputation systems and data quality on MTurk

Ted Matherly (Department of Marketing, Oklahoma State University Stillwater, Stillwater, Oklahoma, USA)

European Journal of Marketing

ISSN: 0309-0566

Article publication date: 15 January 2019

Issue publication date: 5 March 2019

Abstract

Purpose

The purpose of this paper is to investigate how the effectiveness of systems for ensuring cooperation in online transactions is impacted by a positivity bias in the evaluation of the work that is produced. The presence of this bias can reduce the informativeness of the reputation system and negatively impact its ability to ensure quality.

Design/methodology/approach

This research combines survey and experimental methods, collecting data from 1,875 Mechanical Turk (MTurk) workers in five studies designed to investigate the informativeness of the MTurk reputation system.

Findings

The findings demonstrate the presence of a positivity bias in evaluations of workers on MTurk, which leaves them undifferentiated, except at the extremity of the reputation system and by status markers.

Research limitations/implications

Because MTurk workers self-select tasks, the findings are limited in that they may only be generalizable to those who are interested in research-related work. Further, the tasks used in this research are largely subjective in nature, which may decrease their sensitivity to differences in quality.

Practical implications

For researchers, the results suggest that requiring 99 per cent approval rates (rather than the previously advised 95 per cent) should be used to identify high-quality workers on MTurk.

Originality/value

The research provides insights into the design and use of reputation systems and demonstrates how design decisions can exacerbate the effect of naturally occurring biases in evaluations to reduce the utility of these systems.

Keywords

Acknowledgements

The author gratefully acknowledges the comments of Daniel Malter, Zachary G. Arens, Eunyoung Jang, Alejandra Rodriguez and Steven Shepherd, the associate editor and three anonymous reviewers on previous versions of this manuscript. The author also appreciates the feedback of April Bequette, who provided unique information and insights into the MTurk platform. However, all errors within are the author’s.

Citation

Matherly, T. (2019), "A panel for lemons? Positivity bias, reputation systems and data quality on MTurk", European Journal of Marketing, Vol. 53 No. 2, pp. 195-223. https://doi.org/10.1108/EJM-07-2017-0491

Publisher

:

Emerald Publishing Limited

Copyright © 2019, Emerald Publishing Limited