To read the full version of this content please select one of the options below:

Reliable, valid and efficient evaluation of media messages: Developing a message testing protocol

Minji Kim (Center for Tobacco Control Research and Education, University of California San Francisco, San Francisco, California, USA)
Joseph N. Cappella (Annenberg School for Communication, University of Pennsylvania, Philadelphia, Pennsylvania, USA)

Journal of Communication Management

ISSN: 1363-254X

Article publication date: 7 June 2019

Issue publication date: 21 August 2019

Abstract

Purpose

In the field of public relations and communication management, message evaluation has been one of the starting points for evaluation and measurement research at least since the 1970s. Reliable and valid message evaluation has a central role in message effects research and campaign design in other disciplines as well as communication science. The purpose of this paper is to offer a message testing protocol to efficiently acquire valid and reliable message evaluation data.

Design/methodology/approach

A message testing protocol is described in terms of how to conceptualize and evaluate the content and format of messages, in terms of procedures for acquiring and testing messages and in terms of using efficient, reliable and valid measures of perceived message effectiveness (PME) and perceived argument strength (PAS). The evidence supporting the reliability and validity of PME and PAS measures is reviewed.

Findings

The message testing protocol developed and reported is an efficient, reliable and valid approach for testing large numbers of messages.

Research limitations/implications

Researchers’ ability to select candidate messages for subsequent deeper testing, for various types of communication campaigns, and for research in theory testing contexts is facilitated. Avoiding the limitations of using a single instance of a message to represent a category (also known as the case-category confound) is reduced.

Practical implications

Communication campaign designers are armed with tools to assess messages and campaign concepts quickly and efficiently, reducing pre-testing time and resources while identifying “best-in-show” examples and prototypes.

Originality/value

Message structures are conceptualized in terms of content and format features using theoretically driven constructs. Measures of PAS and PME are reviewed for their reliability, construct and predictive validity, finding that the measures are acceptable surrogates for actual effectiveness for a wide variety of messages and applications. Coupled with procedures that reduce confounding by randomly nesting messages within respondents and respondents to messages, the measures used and protocol deployed offer an efficient and utilitarian approach to message testing and modeling.

Keywords

Acknowledgements

The authors wish to acknowledge the funding support of the National Cancer Institute’s Center of Excellence in Cancer Communication (CECCR: P20-CA095856) at the Annenberg School for Communication, University of Pennsylvania, and National Institute of Health’s Exceptional, Unconventional Research Enabling Knowledge Acceleration (EUREKA: R01CA160226-01). The authors wish to acknowledge the valuable comments from the editors and reviewers in tuning this manuscript to JOCM’s readership.

Citation

Kim, M. and Cappella, J.N. (2019), "Reliable, valid and efficient evaluation of media messages: Developing a message testing protocol", Journal of Communication Management, Vol. 23 No. 3, pp. 179-197. https://doi.org/10.1108/JCOM-12-2018-0132

Publisher

:

Emerald Publishing Limited

Copyright © 2019, Emerald Publishing Limited