Performance evaluation of classifier ensembles in terms of diversity and performance of individual systems

Yun‐Sheng Chung (Department of Computer Science, National Tsing Hua University, HsinChu, Taiwan)
D. Frank Hsu (Department of Computer and Information Sciences, Fordham University, New York, USA)
Chun‐Yi Liu (Department of Computer Science, National Tsing Hua University, HsinChu, Taiwan)
Chun‐Yi Tang (Department of Computer and Information Engineering, Providence University, TaiChung, Taiwan)

International Journal of Pervasive Computing and Communications

ISSN: 1742-7371

Publication date: 22 November 2010

Abstract

Purpose

Multiple classifier systems have been used widely in computing, communications, and informatics. Combining multiple classifier systems (MCS) has been shown to outperform a single classifier system. It has been demonstrated that improvement in ensemble performance depends on either the diversity among or the performance of individual systems. A variety of diversity measures and ensemble methods have been proposed and studied. However, it remains a challenging problem to estimate the ensemble performance in terms of the performance of and the diversity among individual systems. The purpose of this paper is to study the general problem of estimating ensemble performance for various combination methods using the concept of a performance distribution pattern (PDP).

Design/methodology/approach

In particular, the paper establishes upper and lower bounds for majority voting ensemble performance with disagreement diversity measure Dis, weighted majority voting performance in terms of weighted average performance and weighted disagreement diversity, and plurality voting ensemble performance with entropy diversity measure D.

Findings

Bounds for these three cases are shown to be tight using the PDP for the input set.

Originality/value

As a consequence of the authors' previous results on diversity equivalence, the results of majority voting ensemble performance can be extended to several other diversity measures. Moreover, the paper showed in the case of majority voting ensemble performance that when the average of individual systems performance P is big enough, the ensemble performance Pm resulting from a maximum (information‐theoretic) entropy PDP is an increasing function with respect to the disagreement diversity Dis. Eight experiments using data sets from various application domains are conducted to demonstrate the complexity, richness, and diverseness of the problem in estimating the ensemble performance.

Keywords

Citation

Chung, Y., Hsu, D.F., Liu, C. and Tang, C. (2010), "Performance evaluation of classifier ensembles in terms of diversity and performance of individual systems", International Journal of Pervasive Computing and Communications, Vol. 6 No. 4, pp. 373-403. https://doi.org/10.1108/17427371011097604

Download as .RIS

Publisher

:

Emerald Group Publishing Limited

Copyright © 2010, Emerald Group Publishing Limited

To read the full version of this content please select one of the options below

You may be able to access this content by logging in via Shibboleth, Open Athens or with your Emerald account.
To rent this content from Deepdyve, please click the button.
If you think you should have access to this content, click the button to contact our support team.