gave for the Netflix data set [15]. The coordinates have been generated by MATLAB's mdscale function using the metric stress criterion. Since in our data set about 14 percent of all movie–movie pairs had no raters in common, we treated the respective entries of the distance matrix as missing data.

To measure the effect of dimensionality, we generated three different coordinate spaces with each extractor by varying the parameter d.

We chose d = 10, d **= 50**, and d **= 100**.

## 4.2 Applying The Classifiers

In total, we used 14 different classifiers to evaluate each of the 12 coordinate spaces with respect to each of the 13 genres.

We implemented the two support vector machine classifiers by soft-margin SVMs with parameters C = 4 and (for SVM-RBF)
γ = 0.1, which have been determined by cross-validation to maximize classification accuracy.

Each of the four different kNN-classifiers will be applied to the data sets with three different choices of k. To measure whether movies of the same genre tend to occur in larger groups, we chose k = 1, k = 3, and k = 9. In the following, we will refer to these 12 classifiers as kNN-Eucl, kNN-sEucl, kNN-scal, and kNN-cos.

To enable comparisons among classifiers and data sets, we generated 20 pairs of training and test sets, each by randomly chosing 40%
of all movies for training and 10% (of the remaining movies) for testing. For each of the resulting 2184 combinations of coordinate spaces, classifiers and genres, we use the same 20 pairs of item sets for training and testing. In each case, we measured the classification accuracy. All results reported below are averages over the 20 runs.

## 4.3 Results

Probably the most popular way of assessing a classifier's performance is measuring its accuracy, that is, the fraction of test items which have been classified correctly. However, in our setting, this measure is not very helpful. To see this, recall that the relative frequency of genres is very different in our data set. For example, over half of all movies belong to the genre *Drama,* but there are only about 5% War movies. While attaining an accuracy of 95% would be significant for the genre *Drama,* it can easily be achieved for the genre War just by classifying any movie as *non-War*. To enable comparisons across genres, we propose to use a modified version of Cohen's kappa measure.

Any result of a binary classification task can be described by four numbers, which sum up to 1: the fraction of true positives (αtp), the fraction of false positives (αfp), the fraction of false negatives
(αfn), and the fraction of true negatives (αtn). Accuracy is defined as acc = αtp + αtn. Moreover, the accuracy of a static majoritybased classifier (which always returns the label of the more frequent class) is accmaj = max{αtp + αfn, αfp + αtn}. We propose to use this kind of naive classifier for normalizing the accuracy and define κ = (acc − accmaj)/(1 − accmaj). This measure expresses a classifier's relative performance with respect to the majority-based classifier. If acc = 1 then κ = 1, if **acc > acc**maj, then κ > 0, if acc = accmaj, then κ = 0, and if **acc < acc**maj, then κ < 0.

By measuring accuracy in terms of κ, we can average classification performance over different genres. Tables 2–5 report the mean κs over all 260 classification results obtained for each combination of coordinate space and classifier type. All entries larger than 0.10 have been marked in boldface. We can observe the following:

| SVD-10    | SVD-50                                                               | SVD-100   |       |
|-----------|----------------------------------------------------------------------|-----------|-------|
| SVM-lin   | 0.08                                                                 | 0.18      | 0.20  |
| SVM-RBF   | 0.15                                                                 | 0.23      | 0.25  |
| 1NN-Eucl  | −0.24                                                                | −0.21     | −0.19 |
| 3NN-Eucl  | 0.01                                                                 | 0.05      | 0.04  |
| 9NN-Eucl  | 0.12                                                                 | 0.16      | 0.14  |
| 1NN-sEucl | −0.25                                                                | −0.27     | −0.31 |
| 3NN-sEucl | 0.01                                                                 | 0.00      | −0.06 |
| 9NN-sEucl | 0.12                                                                 | 0.12      | 0.04  |
| 1NN-scal  | −0.42                                                                | −0.30     | −0.30 |
| 3NN-scal  | −0.16                                                                | −0.03     | −0.03 |
| 9NN-scal  | 0.01                                                                 | 0.11      | 0.12  |
| 1NN-cos   | −0.25                                                                | −0.18     | −0.16 |
| 3NN-cos   | 0.00                                                                 | 0.06      | 0.06  |
| 9NN-cos   | 0.12                                                                 | 0.17      | 0.16  |
| Table 2.  | Kappas for coordinates generated by SVD. δ-SVD-10 δ-SVD-50 δ-SVD-100 |           |       |
| SVM-lin   | 0.07                                                                 | 0.16      | 0.18  |
| SVM-RBF   | 0.13                                                                 | 0.20      | 0.23  |
| 1NN-Eucl  | −0.26                                                                | −0.26     | −0.26 |
| 3NN-Eucl  | −0.01                                                                | 0.01      | −0.02 |
| 9NN-Eucl  | 0.11                                                                 | 0.12      | 0.08  |
| 1NN-sEucl | −0.26                                                                | −0.29     | −0.36 |
| 3NN-sEucl | 0.00                                                                 | −0.03     | −0.11 |
| 9NN-sEucl | 0.11                                                                 | 0.09      | −0.01 |
| 1NN-scal  | −0.41                                                                | −0.28     | −0.22 |
| 3NN-scal  | −0.06                                                                | 0.02      | 0.06  |
| 9NN-scal  | 0.05                                                                 | 0.13      | 0.16  |
| 1NN-cos   | −0.26                                                                | −0.19     | −0.16 |
| 3NN-cos   | 0.00                                                                 | 0.07      | 0.09  |
| 9NN-cos   | 0.12                                                                 | 0.18      | 0.19  |
| Table 3.  | Kappas for coordinates generated by δ-SVD. NNMF-10 NNMF-50 NNMF-100  |           |       |
| SVM-lin   | 0.02                                                                 | 0.05      | 0.11  |
| SVM-RBF   | 0.02                                                                 | 0.09      | 0.14  |
| 1NN-Eucl  | −0.56                                                                | −0.47     | −0.41 |
| 3NN-Eucl  | −0.20                                                                | −0.16     | −0.13 |
| 9NN-Eucl  | −0.02                                                                | 0.01      | 0.02  |
| 1NN-sEucl | −0.56                                                                | −0.47     | −0.45 |
| 3NN-sEucl | −0.20                                                                | −0.16     | −0.16 |
| 9NN-sEucl | −0.02                                                                | 0.01      | 0.00  |
| 1NN-scal  | −0.37                                                                | −0.34     | −0.34 |
| 3NN-scal  | −0.11                                                                | −0.10     | −0.09 |
| 9NN-scal  | −0.02                                                                | 0.00      | 0.02  |
| 1NN-cos   | −0.56                                                                | −0.45     | −0.41 |
| 3NN-cos   | −0.20                                                                | −0.15     | −0.13 |
| 9NN-cos   | −0.03                                                                | 0.02      | 0.03  |
| Table 4.  | Kappas for coordinates generated by NNMF. MDS-10 MDS-50 MDS-100      |           |       |
| SVM-lin   | −0.16                                                                | 0.15      | 0.19  |
| SVM-RBF   | 0.03                                                                 | 0.16      | 0.17  |
| 1NN-Eucl  | −0.29                                                                | −0.19     | −0.18 |
| 3NN-Eucl  | −0.01                                                                | 0.06      | 0.06  |
| 9NN-Eucl  | 0.13                                                                 | 0.18      | 0.18  |
| 1NN-sEucl | −0.29                                                                | −0.23     | −0.29 |
| 3NN-sEucl | −0.01                                                                | 0.05      | −0.01 |
| 9NN-sEucl | 0.13                                                                 | 0.17      | 0.12  |
| 1NN-scal  | −0.29                                                                | −0.19     | −0.18 |
| 3NN-scal  | −0.01                                                                | 0.07      | 0.08  |
| 9NN-scal  | 0.12                                                                 | 0.18      | 0.18  |
| 1NN-cos   | −0.28                                                                | −0.18     | −0.16 |
| 3NN-cos   | 0.00                                                                 | 0.07      | 0.08  |
| 9NN-cos   | 0.13                                                                 | 0.19      | 0.19  |
| Table 5.  | Kappas for coordinates generated by MDS.                             |           |       |
