a date to a reaction, or more generally to any sentence conveying background information, and placing it into the chronological stream of the main events does not produce a logical ordering. The ordering of these themes is, therefore, not covered by the CO algorithm. Furthermore, some sentences cannot be assigned any time stamp. **For instance, the sentence,**
"The vast, sparsely inhabited Xinjiang region, largely desert, has many Chinese military and nuclear installations and civilian mining." **describes a state rather than an event and,** therefore, trying to describe it in temporal terms is invalid. Thus the ordering cannot be improved at the temporal level.

The second phenomenon we observed is that Poor summaries typically contain abrupt switches of topics and are generally incoherent. For instance, in Figure 9, quotes from US officials (third and fifth sentences) are split, and sentences about the mourning (first and sixth sentences) appear too far apart in the summary. Grouping them together would increase the readability of the summary. At this point, we need to find additional constraints to improve the ordering.

## 5. Improving The Ordering: Experiments And Analysis

In the previous section, we showed that using naive ordering **algorithms does not produce** satisfactory orderings. In this section, we investigate through experiments with humans how to identify patterns of orderings that can improve the algorithm.

## 5.1 Collecting A Corpus Of Multiple Orderings

Sentences in a text can be ordered in a number of ways, and the text as a whole will still convey the same meaning. But the majority of possible orders **are likely to be unacceptable because they break conventions of information presentation. One way to identify these** conventions is to find commonalities among different acceptable orderings of the same information. Extracting regularities in several acceptable orderings can help us specify ordering constraints for a given input type. There is no naturally occurring existing collection of summaries for multiple documents that we aware of7**. But even such a collection would not**
be sufficient since we want to analyze a collection of multiple **summaries over the same set**
of articles. We created our own collection of multiple orderings produced by different humans. Using this collection, we studied common behaviors and mapped them to strategies for ordering.

Our collection of multiple orderings, along with our test corpus is available at http://www.cs.columbia.edu/~noemie/ordering/**. We collected ten sets of articles for**
this collection. Each set consisted of two to three news articles reporting the same event.

For each set, we manually selected the intersection sentences, simulating MultiGen8**. On**
average, each set contained 8.8 intersection sentences. The sentences were cleaned of explicit references (for instance, occurrences of "the President" were resolved to "President Clinton") and connectives, so that participants would not use them as clues for ordering. Ten subjects participated in the experiment, and they each built one ordering per set of