

![0_image_0.png](0_image_0.png)

tain significantly more or fewer tokens than expected based on **the old number-of-tokens** distribution, are eliminated from the set of candidate extracts. The learned patterns are often too general and will match many, possibly hundreds, text segments on each page.

Among these spurious text segments is the correct example of **the data field. The rest of**
the discussion is concerned with identifying the correct examples of data on pages.

We exploit some simple *a priori* **assumptions about the structure of Web pages to help**
us separate interesting extracts from noise. We expect examples of the same data field to appear roughly in the same position and in the same context on **each page. For example,**
Fig. 6 shows fragments of two Web pages from the same source displaying restaurant information. On both pages the relevant information about the **restaurant appears after the**
heading "ALL LISTINGS" and before the phrase "Appears in the **Category:". Thus, we**
expect the same field, *e.g.***, address, to appear in the** *same place***, or slot, within the page**
template. Moreover, the information we are trying to extract will not usually be part of the page template; therefore, candidate extracts that are part of the page template can be eliminated from consideration. Restaurant address always follows restaurant name (in bold) and precedes the city and zip code, *i.e.***, it appears in the** *same context* **on every page.**
A given field is either visible to the user on every page, or it is **invisible (part of an HTML** tag) on every page. In order to use this information to separate extracts, we describe each candidate extract by a feature vector, which includes positional information, defined by the (page template) slot number and context. The context is captured by the adjacent tokens:
one token immediately preceding the candidate extract and one token immediately following it. We also use a binary feature which has the value one if the token is visible to the user, and zero if it is part of an HTML tag. Once the candidate extracts have been assigned feature vectors, we split them into groups, so that within each group, the candidate extracts are described by the same feature vector.

The next step is to score groups based on their similarity to the training examples. We expect the highest scoring group to contain correct examples of the data field. One scoring method involves assigning a rank to the groups based on how many extracts they have in common with the training examples. This technique generally works well, because at least some of the data usually remains the same when the Web page layout changes. Of