# Grounding The Lexical Semantics Of Verbs In Visual Perception Using Force Dynamics And Event Logic

Jeffrey Mark Siskind QOBI@RESEARCH.NJ.NEC.COM
NEC Research Institute, Inc. 4 Independence Way Princeton, NJ 08540 USA

## Abstract

This paper presents an implemented system for recognizing the occurrence of events described by simple spatial-motion verbs in short image sequences. The semantics of these verbs is specified with event-logic expressions that describe changes in the state of force-dynamic relations between the participants of the event. An efficient finite representation is introduced for the infinite sets of intervals that occur when describing liquid and semi-liquid events. Additionally, an efficient procedure using this representation is presented for inferring occurrences of compound events, described with event-logic expressions, from occurrences of primitive events. Using force dynamics and event logic to specify the lexical semantics of events allows the system to be more robust than prior systems based on motion profile.

## 1. Introduction

If one were to look at the image sequence in Figure 1(a), one could describe the event depicted in that sequence by saying *Someone picked the red block up off the green block*. Similarly, if one were to look at the image sequence in Figure 1(b), one could describe the event depicted in that sequence by saying *Someone put the red block down on the green block*. One way that one recognizes that the former is a *pick up*1event is that one notices a state change in the force-dynamic relations between the participant objects. Prior to Frame 13, the red block is supported by the green block by a substantiality constraint, the fact that solid objects cannot interpenetrate (Spelke, 1983; Baillargeon, Spelke, & Wasserman, 1985; Baillargeon, 1986, 1987; Spelke, 1987, 1988). From Frame 13 onward, it is supported by being attached to the hand. Similarly, one way that one recognizes that the latter is a *put down* event is that one notices the reverse state change in Frame 14.

This paper describes an implemented computer system, called LEONARD, that can produce similar event descriptions from such image sequences. A novel aspect of this system is that it produces event descriptions by recognizing state changes in force-dynamic relations between participant objects.

Force dynamics is a term introduced by Talmy (1988) to describe a variety of causal relations between participants in an event, such as allowing, preventing, and forcing. In this paper, I use force dynamics in a slightly different sense, namely, to describe the support, contact, and attachment relations between participant objects.

A number of systems have been reported that can produce event descriptions from video input.

Examples of such systems include the work reported in Yamoto, Ohya, and Ishii (1992), Starner
(1995), Siskind and Morris (1996), Brand (1996), and Bobick and Ivanov (1998). LEONARD differs