- Second, we believe that factor models might be a powerful tool for automatically extracting meaningful descriptions of otherwise hard-to-describe items such as movies or songs—particularly, essential features of movies cannot be characterized at all by purely technical features such as runtime, language, or release date.2 But given a coordinate representation of movies that matches human perception, the full machinery developed in preference handling research can be applied [6, 9]. For example, clustering techniques can give user an initial high-level impression of the available items, item rankings can be learnt from ordinal preference statements [10] or utilities [5], and the best items can be retrieved by means of Top-k algorithms [12].
Since our primary research interest lies in applying preferencebased retrieval techniques to item collections, in this paper we will concentrate on evaluating the semantic structures contained in the item matrix A. Performing a similar analysis of the user matrix B
may require entirely different methods.

The paper is structured as follows: After introducing notation and reviewing the most important factor models, we develop general guidelines on how to evaluate coordinate spaces for semantic information. Then, we illustrate how to apply these guidelines to the evaluation of factor spaces generated from movie rating data and perform experiments on the MovieLens 10M data set.

## 2 Preliminaries

In the following, we use the variables i and j to identify items, whereas u and v denote users. We are dealing with ratings given to I
items by U users. Let R = (ri,u) ∈ {R **∪ ∅}**I×Ube the corresponding rating matrix, where ri,u = ∅, if item i has not been rated by user u; otherwise, ri,u expresses the strength of user u's preference for item i. Ratings are usually limited to a fixed integer scale (for example, one to ten stars). Moreover, R =(i, u)| ri,u 6= ∅	is the set of all item–user pairs for which ratings are known. Let n be the total number of ratings observed (the cardinality of R). Typically, n is very small compared to the number of possible ratings I · U (for example, in the Netflix data set it is n I·U ≈ 1.4%).

Given some target dimensionality d, the basic idea underlying factor models is to find matrices A = (ai,r) ∈ R
I×dand B = (br,u) ∈
R

d×Usuch that their product Rˆ = A · B closely resembles R on all known entries. To quantify this notion of "close resemblance,"
the sum of squared errors (SSE) is popularly chosen. The SSE difference between the rating matrix R and its estimation Rˆ = (ˆri,u) is defined as

$$\operatorname{SSE}(R,{\hat{R}})=\sum_{(i,u)\in\mathcal{R}}\left(r_{i,u}-{\hat{r}}_{i,u}\right)^{2}.$$

Factor models are typically formulated as optimization problems over A and B, in which the SSE (or some other measure) is to be minimized.

Probably the most popular factor model is Brandyn Webb's regularized SVD model [16, 18], in which A and B are defined as the solution of the least squares problem

$$\operatorname*{min}_{A,B}\quad{\mathrm{SSE}}\!\left(R,A\cdot B\right)+\lambda\sum_{(i,u)\in{\mathcal{R}}}\sum_{r=1}^{d}\left(a_{i,r}^{2}+b_{r,u}^{2}\right).$$
Here, $\lambda\geq0$ is a regularization constant used to avoid overfitting. 
2 A complementary approach to closing this *semantic gap* is content-based image and video retrieval [8].

More advanced versions of the SVD model exclude systematic rating deviations from the factorization and model them explicitly using new variables. Bell and Koren [3] propose to estimate rating ri,u by

$${\hat{r}}_{i,u}=\mu+\delta_{i}+\delta_{u}+\sum_{r=1}^{d}a_{i,r}b_{r,u},$$
$\mathbf{a}\cdot\mathbf{a}^{\dagger}=\mathbf{a}\cdot\mathbf{a}^{\dagger}$. 

where the constant µ denotes the mean of all observed ratings; δi and δu are I + U new model parameters expressing systematic item and user deviations from µ. Again, the parameters are chosen according to a regularized least squares problem:

$$\operatorname*{min}_{A,B,\delta_{*}}\quad\operatorname{SSE}\!\left(R,{\hat{R}}\right)+\lambda\sum_{(i,u)\in\mathcal{R}}\left(\sum_{r=1}^{d}\left(a_{i,r}^{2}+b_{r,u}^{2}\right)+\delta_{i}+\delta_{u}\right).$$

The rationale underlying this approach—which we refer to as δ-SVD
in the following—is that the removal of item- and user-specific general trends from the factorization allows to focus on more sophisticated rating patterns.

The third basic factor model being relevant to our work performs a non-negative factorization of the rating matrix [23]. It is identical to the regularized SVD model up to the additional constraint that all entries of A and B must be non-negative. Extending this model by explicit item and users deviations is not reasonable since this would require negative entries in A and B to approximate R close enough.

The non-negative matrix factorization model aims at creating a coordinate space in which effects of different dimensions on the estimated ratings cannot cancel out each other. Henceforth, we refer to this model as NNMF.

## 3 Evaluating Coordinate Spaces

Given an item–feature matrix A ∈ R
I×dgenerated by some factor model, how can we determine whether the items' coordinates in this d-dimensional space resemble a "semantically meaningful" pattern?

The most straightforward approach consists in extending and systematizing the casual investigations described in the introduction. This could easily be done by presenting the item coordinate space to a number of different people and asking them to label its dimensions.

The correspondence between the generated item coordinates and human perception could, for example, be done by measuring the degree of consensus among people or the average time needed to come up with adequate labels.

Although this kind of investigation seems very reasonable, it contains some severe flaws, which cannot be fixed by careful study design:

1. The dimensionality chosen in most applications of factor models typically ranges between d **= 10** and d **= 100**. A comprehensive analysis of the resulting data sets would require the users to comprehend high-dimensional spaces, which is impossible even when using advanced visualization techniques.

2. Due to hindsight bias, given enough time, users will be able to assign a fitting label to almost any dimension of the coordinate space.

Chances are good that this effect accounts for rather questionable labels such as *NOT Monty Python*.

3. By using free association to name dimensions, the collection of resulting labels tend to show a high variability and reflect individual differences between users. To produce statistically significant results, either the sample size must be extended (which requires more study participants and results in higher costs), or the variability must be reduced, for example, by training participants to