Reliable, valid and efficient evaluation of media messages: Developing a message testing protocol
Journal of Communication Management
Article publication date: 7 June 2019
Issue publication date: 21 August 2019
In the field of public relations and communication management, message evaluation has been one of the starting points for evaluation and measurement research at least since the 1970s. Reliable and valid message evaluation has a central role in message effects research and campaign design in other disciplines as well as communication science. The purpose of this paper is to offer a message testing protocol to efficiently acquire valid and reliable message evaluation data.
A message testing protocol is described in terms of how to conceptualize and evaluate the content and format of messages, in terms of procedures for acquiring and testing messages and in terms of using efficient, reliable and valid measures of perceived message effectiveness (PME) and perceived argument strength (PAS). The evidence supporting the reliability and validity of PME and PAS measures is reviewed.
The message testing protocol developed and reported is an efficient, reliable and valid approach for testing large numbers of messages.
Researchers’ ability to select candidate messages for subsequent deeper testing, for various types of communication campaigns, and for research in theory testing contexts is facilitated. Avoiding the limitations of using a single instance of a message to represent a category (also known as the case-category confound) is reduced.
Communication campaign designers are armed with tools to assess messages and campaign concepts quickly and efficiently, reducing pre-testing time and resources while identifying “best-in-show” examples and prototypes.
Message structures are conceptualized in terms of content and format features using theoretically driven constructs. Measures of PAS and PME are reviewed for their reliability, construct and predictive validity, finding that the measures are acceptable surrogates for actual effectiveness for a wide variety of messages and applications. Coupled with procedures that reduce confounding by randomly nesting messages within respondents and respondents to messages, the measures used and protocol deployed offer an efficient and utilitarian approach to message testing and modeling.
The authors wish to acknowledge the funding support of the National Cancer Institute’s Center of Excellence in Cancer Communication (CECCR: P20-CA095856) at the Annenberg School for Communication, University of Pennsylvania, and National Institute of Health’s Exceptional, Unconventional Research Enabling Knowledge Acceleration (EUREKA: R01CA160226-01). The authors wish to acknowledge the valuable comments from the editors and reviewers in tuning this manuscript to JOCM’s readership.
Kim, M. and Cappella, J.N. (2019), "Reliable, valid and efficient evaluation of media messages: Developing a message testing protocol", Journal of Communication Management, Vol. 23 No. 3, pp. 179-197. https://doi.org/10.1108/JCOM-12-2018-0132
Emerald Publishing Limited
Copyright © 2019, Emerald Publishing Limited