Reliable, valid and efficient evaluation of media messages: Developing a message testing protocol

Abstract

Purpose

In the field of public relations and communication management, message evaluation has been one of the starting points for evaluation and measurement research at least since the 1970s. Reliable and valid message evaluation has a central role in message effects research and campaign design in other disciplines as well as communication science. The purpose of this paper is to offer a message testing protocol to efficiently acquire valid and reliable message evaluation data.

Design/methodology/approach

A message testing protocol is described in terms of how to conceptualize and evaluate the content and format of messages, in terms of procedures for acquiring and testing messages and in terms of using efficient, reliable and valid measures of perceived message effectiveness (PME) and perceived argument strength (PAS). The evidence supporting the reliability and validity of PME and PAS measures is reviewed.

Findings

The message testing protocol developed and reported is an efficient, reliable and valid approach for testing large numbers of messages.

Research limitations/implications

Researchers’ ability to select candidate messages for subsequent deeper testing, for various types of communication campaigns, and for research in theory testing contexts is facilitated. Avoiding the limitations of using a single instance of a message to represent a category (also known as the case-category confound) is reduced.

Practical implications

Communication campaign designers are armed with tools to assess messages and campaign concepts quickly and efficiently, reducing pre-testing time and resources while identifying “best-in-show” examples and prototypes.

Originality/value

Message structures are conceptualized in terms of content and format features using theoretically driven constructs. Measures of PAS and PME are reviewed for their reliability, construct and predictive validity, finding that the measures are acceptable surrogates for actual effectiveness for a wide variety of messages and applications. Coupled with procedures that reduce confounding by randomly nesting messages within respondents and respondents to messages, the measures used and protocol deployed offer an efficient and utilitarian approach to message testing and modeling.

Citation

Kim, M. and Cappella, J.N. (2019), "Reliable, valid and efficient evaluation of media messages: Developing a message testing protocol", Journal of Communication Management, Vol. 23 No. 3, pp. 179-197. https://doi.org/10.1108/JCOM-12-2018-0132