ticipants. Let us contrast the number of parameters needed to represent these features in both the motion-profile and force-dynamic approaches. Since, as discussed above, motion-profile features are typically continuous, hidden Markov models with continuous outputs can be used in the motionprofile approach. When the features are independent, such a model requires O(n) parameters per state to specify the output distributions. Even if one uses, say, a multivariate Gaussian to model dependent features, this requires only O(n2) parameters per state to specify the output distributions in the motion-profile approach. However, force-dynamic features are Boolean. This requires using discrete-output hidden Markov models. Such models output a stream of symbols, not feature vectors. Constructing an appropriate alphabet of output symbols requires considering all possible subsets of features. This requires O(2n) parameters per state to specify the output distributions in the force-dynamic approach. Thus continuous-output hidden Markov models appear to be better suited to an approach that uses motion-profile features while event logic appears to be better suited to an approach that uses force-dynamic features.

Humans use language for three fundamental purposes: we describe what we see, we ask others to perform actions, and we engage in conversation. The first two require grounding language in perception and action. Only the third involves disembodied use of language. Almost all research in computational linguistics has focused on such disembodied language use. Data-base query processing, information extraction and retrieval, and spoken-language dialog all use language solely to manipulate internal representations. In contrast, the work described in this paper grounds language in perception of the external world. It describes an implemented system, called LEONARD, that uses language to describe events observed in short image sequences.

Why is perceptual grounding of language important and relevant to computational linguistics?

Current approaches to lexical semantics suffer from the 'bold-face syndrome.' All too often, the meanings of words, like *throw*, are taken to be uninterpreted symbols, like **throw**, or expressions over uninterpreted symbols, like **cause to go** (Leech, 1969; Miller, 1972; Schank, 1973; Jackendoff, 1983, 1990; Pinker, 1989). Since the interpretation of such symbols is left to informal intuition, the correctness of any meaning representation constructed from such uninterpreted symbols cannot be verified. In other words, how is one to know whether **cause to go** is the correct meaning of *throw*?

Perceptual grounding offers a way to verify semantic representations. Having an implemented system use a collection of semantic representations to generate appropriate descriptions of observations gives evidence that those semantic representations are correct. This paper takes a small step in this direction. In contrast to prior work, which presents informal semantic representations whose interpretation is left to intuition, it presents perceptually-grounded semantic representations. While the system described in this paper addresses only perceptual grounding of language, the long-term goal of this research is to provide a unified semantic representation that is sufficiently powerful to support all three forms of language use: perception, action, and conversation.

Different parts of speech in language typically describe different aspects of visual percepts.

Nouns typically describe objects. Verbs typically describe events. Adjectives typically describe properties. Prepositions typically describe spatial and temporal relations. Grounding language in visual perception will require construction of semantic representations for all of these different parts of speech. It is likely that different parts of speech will require different machinery to represent their lexical semantics. In other words, whatever the ultimate representation of *apple* and *chair* are, they are likely to be based on very different principles than the ultimate representation of *pick up* and put down. These, in turn, are likely to be further different from those needed to represent in, on, red, and big. Indeed, machine vision research, at least that aspect of machine vision research that focuses