The techniques described in this paper have been implemented in a system called LEONARD.

LEONARD is a comprehensive system that takes image sequences as input and produces event descriptions as output. The overall architecture of LEONARD is shown in Figure 6. The input to LEONARD consists of a sequence of images taken by a Canon VC-C3 camera and Matrox Meteor frame grabber at 320-240 resolution at 30fps. This image sequence is first processed by a segmentation-and-tracking component. A real-time colour- and motion-based segmentation algorithm places a convex polygon around each coloured and moving object in each frame. A tracking algorithm then forms a correspondence between the polygons in each frame and those in temporally adjacent frames. The output of the segmentation-and-tracking component consists of a sequence of scenes, each scene being a set of polygons. Each polygon is represented as a sequence of image coordinates corresponding to a clockwise traversal of the polygon's vertices. The tracker guarantees that each scene contains the same number of polygons and that they are ordered so that the i th polygon in each scene corresponds to the same object. Figure 3 shows the output of the segmentation-and-tracking component on the image sequences from Figure 1. The polygons have been overlayed on the input images for ease of comprehension.

This scene sequence is passed to a model-reconstruction component. This component produces a force-dynamic model of each scene. This model specifies three types of information: which objects are grounded, i.e. are supported by an unseen mechanism that is not associated with any visible object, which objects are attached to other objects by rigid or revolute joints, and the qualitative depth of each object, i.e. a qualitative representation of the relative distance of different objects in the field of view from the observer, in the form of a same layer relation specifying which objects are at the same qualitative depth. Figure 4 shows the output of the model-reconstruction component on the scene sequences from Figure 3. The models are depicted graphically, overlayed on the input images, for ease of comprehension. The details of this depiction scheme will be described momentarily. For now, it suffices to point out that Figure 4(a) shows the red block on the same layer as the green block up through Frame 1 and attached to the hand from Frame 14 onward. Figure 4(b) shows the reverse sequence of relations, with the red block attached to the hand up through Frame 13 and on the same layer as the green block from Frame 23 onward.

This model sequence is passed to an event-classification component. This component first determines the intervals over which certain primitive event types are true. These primitive event types include SUPPORTED(x), SUPPORTS(x; y), CONTACTS(x; y), and ATTACHED (x; y). This component then uses an inference procedure to determine the intervals over which certain compound event types are true. These compound event types include PICKUP(x; y; z), PUTDOWN(x; y; z),
STACK(w; x; y; z), UNSTACK(w; x; y; z), MOVE(w; x; y; z), ASSEMBLE (w; x; y; z), and DISASSEMBLE (w; x; y; z) and are specified as expressions in event logic over the primitive event types. The output of the event-classification component consists of an indication of which compound event types occurred in the input movie as well as the subsequence(s) of frames during which those event types occurred. Figure 5 shows the output of the event-classification component on the model sequences from Figure 4. The subsequences of frames during which the events occurred are depicted as spanning intervals. Spanning intervals will be described in Section 4.1.

LEONARD is too complex to describe completely in one paper. This paper provides a detailed description of the event-classification component and, in particular, the event-logic inference procedure. The segmentation and tracking algorithms are extensions of the algorithms presented in Siskind and Morris (1996) and Siskind (1999), modified to place convex polygons around the participant objects instead of ellipses. The model-reconstruction techniques are extensions of those