
## Relationship Between Diversity And Perfomance Of Multiple Classifiers For Decision Support R. Musehane, F. Netshiongolwe, F.V. Nelwamondo***, L. Masisi And T. Marwala**

School of Electrical & Information Engineering, University of the Witwatersrand, Private Bag 3, 2050, Johannesburg, South Africa 
***Graduate School of Arts and Sciences Harvard University GSAS Mail Center Child 412, 26 Everett Street Cambridge,** 
Massachusetts, 02138 USA 
Abstract: **The paper presents the investigation and implementation of the relationship between diversity and the performance of** 
multiple classifiers on classification accuracy. The study is critical as to build classifiers that are strong and can generalize better. The parameters of the neural network within the committee were varied to induce diversity; hence structural diversity is the focus for this study. The hidden nodes and the activation function are the parameters that were varied. The diversity measures that were adopted from ecology such as Shannon and Simpson were used to quantify diversity. Genetic algorithm is used to find the optimal ensemble by using the accuracy as the cost function. The results observed shows that there is a relationship between structural diversity and accuracy. It is observed that the classification accuracy of an ensemble increases as the diversity increases. There was an increase of 3%-6% in the classification accuracy. 

Key words: **Classification, Diversity Measures, Genetic Algorithm, Multiple Classifiers, Structural Diversity.** 

## 1. **Introduction**

Computational intelligence techniques have been used in many classification problems. The literature emphasises that a group of classifiers is better than one classifier [1-5]. This is because the decision that is made by a committee of classifiers is better than the decision made by one classifier. In this paper the committee of classifiers will be referred as an ensemble. The most popular way to gain confidence on the generalisation ability of an ensemble is by introducing diversity within the ensemble [1, 2, 5]. This has led to the development of measures of diversity and various aggregation schemes for combining classifiers. However, diversity is not clearly defined [6, 7]. Thus, a proper measure of diversity that will relate diversity to accuracy is to be adopted. Current methods commonly use the outcome of the individual classifiers of an ensemble to measure diversity. Hence an ensemble is considered diverse if classifiers within the ensemble produce different outcomes as opposed to having the same outcomes [1, 6, 7]. In this paper, as opposed to looking at the outcomes of the individual classifiers, ensemble diversity is viewed as the structural variation within classifiers that form an ensemble [1, 5]. Thus, diversity will be induced by changing structural parameters of a neural network [5]. The paper investigates the relationship between structural diversity within an ensemble and the prediction accuracy of the ensemble. It has been intuitively accepted that the classifiers to be combined should be diverse [8]. This is because it has been found meaningless to combine identical classifiers because no improvement can be achieved when combining them [8, 9]. Hence, measuring structural diversity and relating it to accuracy is crucial in order to build better learning machines. However, it is necessary to find the optimal size of an ensemble that gives better generalization. Therefore, a study on the size of the ensemble was done as to find the optimal size that can be used for the investigation. The methods for measuring structural diversity are to be devised and implemented. Moreover, the outcome diversity of structurally different classifiers is critical to be measured. This is because it is essential to show how correlated the outcomes of the structurally different classifiers is. Hence, the limitations of accuracy in the structural diversity are to be justified. Different methods for creating diversity such as bagging and boosting have been explored [1, 3]. However, the aggregation methods are to be used to combine the ensemble predictions. Methods of voting and averaging have been found to be popular [9, 10] and hence are used in this study. The paper first discusses the background in section 2. 

Analysis of the data used for this study is presented in section 3. The accuracy measure and structural measures of diversity used are discussed in section 4 and section 5. The methodologies used in investigating the effect of diversity on generalization are presented in section 6. The results and future work are then discussed in section 7. 

## 2. **Background** 2.1. **Neural Networks**

Neural Networks (NN) are computational models that have the ability to learn and model linear and non-linear systems [11]. There are many types of neural networks but the most common neural network architecture is the multilayer perceptron (MLP) [11]. The neural network architecture that is used in this paper is a MLP network as shown in Figure 1. The MLP network has the input layer, the hidden layer and the output layer. An MLP network 