

![0_image_1.png](0_image_1.png)

a good-classification rate and error classification rate, and how to fuse the different expert opinions. In section 3, we propose two new distance-based measures in order to evaluate well and mis-segmented pixel taking into account the experts certainties. This evaluation is illustrated in section 4 on real sonar images, in order to evaluate a fusion of the classifiers presented in [7].

## 2 Image Classification Evaluation

In this section, we propose an original evaluation approach for classification based on a new confusion matrix taking into account the uncertainty and the possibility that one unit belongs more than one class. This evaluation approach is adapted to the image classification evaluation, but can be used for any classifier evaluation.

## 2.1 Classical Evaluation

A first step of the classical classification evaluation can be made by comparing the results of the classifier to the reality. But in order to evaluate a classification algorithm, many different configurations and tests must be considered. Classification algorithms can yield many variable results depending on the sample. Most of the time, classification algorithms evaluation is conducted by the confusion matrix.

Confusion matrix is composed by the number cmij of elements of the class i classified in the class j**. In**
order to obtain rates making it easier to compare different size of databases, we normalize this confusion

![0_image_0.png](0_image_0.png)

$$N c m_{i j}={\frac{c m_{i j}}{\sum_{j=1}^{N}c m_{i j}}}={\frac{c m_{i j}}{N_{i}}},$$
(1)  $\frac{1}{2}$ .... 
, (1)
with N the number of considered classes and Ni the number of element from the true class i**. From this** normalized confusion matrix a good-classification rate vector can be written as:
GCRi = N cmii, (2)
and an error classification rate vector as:

$$E C R_{i}={\frac{1}{2}}\left(\sum_{j=1,j\neq i}^{N}N c m_{i j}+\sum_{i=1,i\neq j}^{N}{\frac{N c m_{i j}}{N-1}}\right).\quad(3)$$
$\mathbf{r}_{\mathrm{ij}}\dot{\mathbf{i}}$
This error classification rate is the mean of the two errors corresponding to the elements from a given class i **classified in another class (first term), and corresponding to the elements classified in a given class** j being from another class i **(second term). We do not**
have to normalize the first term because of the normalization of the confusion matrix on the rows, but the second term must be normalized by the number of rows minus one (because of the N cmii **term corresponds to the good-classification).**
Thus image classification algorithms evaluation must be made not only on one image but on the whole images database. As a consequence, we have to consider a non-normalized confusion matrix on each image and normalize the sum of the matrix confusion on all images of the database.

## 2.2 Evaluation With Certainty Given By Each Expert

We consider here a general case where information is given by the expert on each pixel and the classification algorithm is made on an unit of n×n **pixels. Hence on**
each unit, more than one class can be present. Generally, the classification algorithms can find only one of these classes. In order to take into account the inhomogeneous units, consider that if the classification algorithm finds one of these classes on the unit, the algorithm is right in the proportion of this found class in the n × n **pixels-unit and it is wrong in the proportion of the other classes in the considered unit. For**
instance, imagine the case where the expert considers a tile of size 16 × **16 pixels and declares that on a part** of the unit, 50 given pixels belong to class 1, and 206 other pixels belong to class 3. If the classification algorithm finds the unit belongs to class 1, the confusion matrix will be computed by the recurrence relations:
cm11 ← cm11 + 50/256 and cm31 ← cm31 + 206/**256.**
Hence the confusion matrix is not composed of integer numbers and Ni**is also not integer; but the sums of**
column are still integers.

If the expert can give the class with a certainty grade, we must not take equally two different grades in our classification evaluation. For instance, in sonar