use an established domain-specific vocabulary to articulate the semantic properties they recognize in the data (which also increases time and effort).

4. Typically, there are many near-optimal solutions to the above mentioned optimization problems, which can be transformed into one another by rotation of the coordinate axes. This is because, for any invertible matrix M ∈ R
d, the solution pairs (**A, B**) and
(**AM, M**−1B) produce the same SSE. Although regularization usually enforces the theoretical existence of a unique optimal solution pair, in practice the enormous problem size often allows only finding one of the many near-optimal solutions. Consequently, the direction of the coordinate axes is completely arbitrary, which makes the task of assigning labels a hopeless undertaking.

## 3.1 Some Guidelines

In this section, we devise a set of guidelines on which to base more appropriate approaches to the analysis of coordinate spaces.

- In the view of problems (1) and (4), we recommend to avoid any direct human interaction with *item coordinates.* Instead, human input should concentrate on describing *item properties,* which in turn are related to coordinates as well as compared by algorithmic means.

- The only effective way to eliminate hindsight bias (2) is collecting feedback on items before generating and presenting any information extracted by the factor models under consideration.

- To resolve problem (3), we primarily recommend to adapt a domain-specific vocabulary to allow a structurized description of items. For example, to characterize music, the rich vocubulary developed by allmusic3seems appropriate; amongst others, it includes very detailed information about genres, styles, moods, and connections between artists. Since this kind of semantic information can be (or already have been) provided by a small number of experts and usually is little prone to debate, it is easy to assemble and work with. In later stages of analysis, unrestricted user feedback may be included to reveal the position and extent of more fine-grained and rather subjective concepts in the coordinate space.
We also propose to apply a standardization procedure to the generated coordinate space. This is for the following reasons: First, recall that, for any invertible matrix M ∈ R
d, the solution pairs (**A, B**) and
(**AM, M**−1B) are equivalent; to enable comparisons between different factor models and even different runs of the same optimization algorithms, we need to define one solution pair as the standard representation. Second, to enable a better separation of different effects in the data, the axes of the item (and user) coordinate space should be chosen to be orthogonal. Moreover, axes should be ordered according to their relative importance (measured by the variance of data along each axis); that is, the first dimension should be assigned to the most important axis.

The perfect tool for matching these requirements is the singular value decomposition, a well-known matrix factorization technique from linear algebra, which inspired the SVD factor model. It is based on the fact that, for any rank-d matrix X ∈ I × U, there is a columnorthonormal matrix U ∈ R
I×d, a diagonal matrix S ∈ R
d×d, and a row-orthonormal matrix V ∈ × U such that X = USV . By reordering rows and columns, S can be chosen such that its diagonal elements are ordered by increasing magnitude. Moreover, the diagonal matrix S can be eliminated from this factorization by setting X = U
′V
′
, where U
′ = US 12 and V
′ = US 12 . The matrices U
′

and V
′
are unique if all diagonal elements of S have been mutually different.

In our setting, we will apply the singular value decomposition to transform the product X = A · B into a new product A
′
· B
′
as just described. Since rating data tends to be very "noisy," we can safely assume that (A
′
, B′) is a unique representation of (**A, B**); we did not encounter any counterexamples during our experiments on large realworld rating data. Moreover, any equivalent pair (**AM, M**−1B) also gets transformed into (A
′
, B′), which we define as the corresponding standard representation. It can be computed efficiently using the product decomposition algorithm proposed in [7, Sec. 3].

## 3.2 Use Case: Movie Ratings

Based on these guidelines, we now present a concrete method for performing a basic evaluation of coordinate spaces generated from movie ratings. Our focus rests on immediate applicability, so we relate the item coordinates to reference data that is already available.

The reference source for all kinds of movie-related information is IMDb, the Internet Movie Database4, which currently covers about 1.6 million titles. Most of IMDb's data has been created with the help of its users. Therefore, a large proportion of the available content can freely be downloaded and used for non-commercial purposes5. Based on this comprehensive data, one should be able to cross-reference any collection of movie ratings with IMDb.

For the semantic evaluations we are going to perform, the following attributes of titles may prove helpful: genres, certifications (e.g.,
USA:PG for *parental guidance suggested*), year of release, and plot keywords. To illustrate the general procedure, we will only exploit genre information in this paper. Extendig our method to other types of semantic information is straightforward. Checking the correspondence between genres and item coordinates also makes up a good first test of whether at least some basic semantic properties of movies are represented in coordinate spaces, which is exactly the purpose of the current work.

IMDb recognizes 28 different genres, from Action to *Western,*
where each movie may belong to multiple genres. The assignment of genres is done by IMDb's expert staff in cooperation with IMDb users. To enforce consistency, this process is based upon a collection of publicly available guidelines6. Therefore, this data source matches the requirements developed in the previous section.

To analyze whether the distribution of genres in coordinate space displays any significant pattern, we turn to established classification algorithms, which explicitly have been designed to exploit any relevant patterns in the data if there are any. In particular, we propose to measure the degree of adherence to a pattern by the classification accuracy shown by these algorithms when predicting the genre of movies based on their coordinates. In essence, we transform our analysis into a sequence of binary classification problems (one for each genre), which enables us to build on solid grounds. Following the common methodology, we use cross-validation; that is, accuracy is measured on a data set, which is independent of the one used to train the classifier. By applying proven techniques to counter overfitting, our approach also overcomes any possible problems related to hindsight bias.

3 http://www.allmusic.com