cial for further processing steps since the system mainly looks for co-occurrences of different cues. The resulting set of annotations constitutes the "working memory" of the system. 

Another set of experts then combine the initial disseminated knowledge to recognize basic segments (section 3.3) and to compute scopes and frames (section 3.4). These experts form the "inference engine" which analyzes information stored in the working memory and adds new knowledge to the database. Even when linear order is irrelevant for the inference process new information is indexed with textual clauses, to enable the system to produce the original text along with annotation. 

 A facilitator helps to determine which expert has the most information needed to solve the problem. It is the facilitator that controls, for example, the application of default rules and the revision of the default segmentation. It controls the chalk, mediating among experts competing to write on the blackboard. Finally, an XML output is produced for the document, corresponding to a candidate GEM version of the document (no XML tags overlap in the output since we produce an instance of the GEM DTD; all potential remaining conflicts must have been solved by the supervisor). To achieve optimal accuracy this output is validated and possibly modified by domain experts. 

## 5 **Evaluation**

The study is based on a corpus of 18 practice guidelines in French (several hundreds of frames), with the aid of domain experts. We evaluated the approach on a subset of the corpus that has not been used for training. 

## 5.1 **Evaluation Criteria**

In our evaluation, a sequence is considered correct if the semantics of the sequence is preserved. For example **Chez l'obèse non diabétique (accord** professionnel) (**In the case of an obese person** without any diabetes (professional approval)), recognition is correct even if **professional approval**
is not stricto sensu **part of the condition. On the** 
other hand, Chez l'obèse (**In the case of an obese** person**) is incorrect. The same criteria are applied** for recommendations. 

We evaluate the scope of condition sequences by measuring whether each recommendation is linked with the appropriate condition sequence or not. 

## 5.2 **Manual Annotation And Inter-Annotator** Agreement

The data is evaluated against practice guidelines manually annotated by two annotators: a domain expert (a doctor) and a linguist. In order to evaluate inter-annotator agreement, conditions and actions are first extracted from the text. The task of the human annotators is then to (manually) build a tree, where each action has to be linked with a condition. The output can be represented as a set of couples (condition - actions**). In the end, we calculate** 
accuracy by comparing the outputs of the two annotators (\# of common couples). 

Inter-annotator agreement is high (157 nodes out of 162, i.e. above .96 agreement). This degree of agreement is encouraging. It differs from previous experiments, usually done using more heterogeneous data, for example, narrative texts. Temporals 
(like "in 1999"**) are known to open a frame but** 
most of the time this frame has no clear boundary. 

Practice guidelines should lead to actions by the doctor and the scope of conditions needs to be clear in the text. 

In our experiment, inter-annotator agreement is high, especially considering that we required an agreement between an expert and non-expert. We thus make the simplified assumption that the scope of conditions is expressed through linguistic cues which do not require, most of the time, domainspecific or expert knowledge. Yet the very few cases where the annotations were in disagreement were clearly due to a lack of domain knowledge by the non-expert. 

## 5.3 **Evaluation Of The Automatic Recognition** Of Basic Sequences

The evaluation of basic segmentation gives the following results for the condition and the recommendation sequences. In the table, P is precision; R is recall; P&R is the harmonic mean of precision and recall (P&R = (2*P*R) / (P+R), corresponding to a F-measure with a β **factor equal to 1).** 

|           | Without domain   | With domain   |
|-----------|------------------|---------------|
| knowledge | knowledge        |               |
| P         | 1                | 1             |
| R         | .83              | .86           |
| P&R       | .91              | .92           |

## Conditions:
