Impact of survey quality on composite indicators

Abstract

Purpose

The purpose of this study is to show the importance of adequately considering quality measures within the use of composite indicators (CIs). Policy support often relies on high quality indicators. Often, the underlying data of relevant indicators are coming mainly from sample surveys. Obviously, the reliability of the indicators then heavily relies on the sampling design and other quality aspects.

Design/methodology/approach

Starting from the well-known work on sensitivity analysis of indicators, this study integrates the sampling process as an additional source of variability. The methodology is evaluated in a close-to-reality simulation environment using relevant and important surveys with different sampling designs. As an example, this study uses data related to the statistics of income and living conditions (SILC). The study is based on a design-based simulation framework.

Findings

In general, the normalisation method is dominating as source of the total variance of CI. In our study, we show that the sampling process also becomes rather relevant and generally dominates the influence of different weighting methods. We show that in some scenarios approximately 40 per cent of the variability in the sensitivity analysis comes from the sampling process. The quality of ranking derived from CIs then suffers considerably from the sampling design. When using data sources from different quality, e.g. in regional comparisons, one may expect some cases with biased CI values which may become useless for applications.

Research limitations/implications

The impact of sampling heavily depends on the data gathering process. In case of sample data, the sampling designs play an important role. However, the design effect still depends on the variables taken into account and has to be considered carefully.

Practical implications

The findings show the importance of considering the quality framework the European Code of Practice also for CIs. This additional information shall foster to understand possible over- or misinterpretations of CIs, especially when deriving rankings from the indicators. Specialised statistical methods shall be integrated in future research, particularly when focusing on regional indicators.

Originality/value

CIs are often used for policy monitoring. In general, the data gathering process is not considered adequately by end-users. This becomes especially important when being interested in regional indicators. The present paper shows possible implications of the sampling designs on CI outcomes with the focus on comparative studies.

Citation

T. Münnich, R. and Georg Seger, J. (2014), "Impact of survey quality on composite indicators", Sustainability Accounting, Management and Policy Journal, Vol. 5 No. 3, pp. 268-291. https://doi.org/10.1108/SAMPJ-10-2013-0045