Defining the target

Defining the target

It’s easy to say rather glibly that TEI markup is a good archival format, and in many respects it is: experience shows that a TEI file can nearly always be read without too many assumptions about the platform or software needed to read it. Because a TEI document uses a very basic form of labelled bracketting, developing software to act upon the markup is a breeze; moreover because the semantics and syntax of the markup are well defined, the software can perform whatever tricks it likes on the basis of an explicit model of the document’s structure and semantics. The tricky part is deciding what exactly the components of that explicit model should be: what (to coin a phrase) is this text really?

On the screen I am currently typing at I see that the phrase “Defining the target” and the word “really” are both in an italic font. The first is a heading, and the second is a word I wish to emphasize. In neither case is it particularly helpful to state that there’s a font change here: if I lost that information for the first case you would still (probably) recognise the words as a heading on other grounds (it’s not a sentence; it’s a separate block; it’s in a place where a heading is conventionally appropriate etc.) but in the second, without the signal given by the font style change you have no easy way of noticing that this word is meant to be more salient than the others, still less of recognising the allusion it makes to a famous journal article. Is recognising (and preserving) this emphasis as essential a part of this document as distinguishing the heading from what follows, or noting the paragraph divisions ?

Although it’s tricky, it’s something I and others have been doing for decades, this business of deciding which are the “essential” components of a document, independently of its realisation on screen or paper. The claim is not just that this separation of the document from its realisation is meaningful, but that it’s also useful. Certainly it makes it much simpler to process masses of similar but different documents in a reasonably intelligent way if their structural components and semantically salient properties are explicitly and exhaustively flagged in the same way. Certainly there may be a price to pay for that simplicity: we may have to renounce the ability to visualise the document exactly as one or other of its many historical realisations did; just as we do for other cases where such a realisation depended on a specific software infrastructure. Good luck emulating a pixel-perfect WordPerfect 4.2 or WordStar view of your TEI document on the basis of its TEI archival form.

All this by way of prelude to the next stage in my attempts to recover/reconstruct a usable TEI archive of the deliberations of the TEI Council. Those deliberations currently exist (as previous blog entries have shown) in one or more of three different forms: as Google Docs, as WordPress HTML pages, or in one or other legacy TEI format. All of these formats are relatively simple to convert into XML without loss of such information as they already contain: the task is to define a minimal TEI markup scheme to which they can all be reduced, without losing anything essential. It is that classic TEI markup problem: what do you want to distinguish in your documents? With the added constraint that I’d rather not have to introduce distinctions not already explicit (one way or another) in the sources.

I started with the Word Press XML files, since these constitute the official published record, even though they have many shortcomings. I wrote another perl script to extract a list of all the different XML tags present in the files, and an XSLT stylesheet containing a default template for each of them, mapping format-oriented tags like <h1> and <b> to semantic ones like <head> and <hi>. I then spent a happy hour or three fiddling with that, before deciding that this approach was too labour intensive to be a general solution.

So I moved on to the Google Doc files. I exported them all as docx files, applied the default TEI docxtotei conversion, and started looking at my 100 or so allegedly TEI documents. The first step was to generate an ODD which described their actual tagging practice, for which I used the TEI

oddByExample utility. This is a good way of starting the process, but has some quirks (like specifying all the element classes you might use, even though you don’t actually use any of them and explicitly deleting each attribute supplied by a class rather than deleting the class), and one major drawback. The drawback is actually perhaps a virtue: the schema you get from the ODD it generates is a strictly conformant TEI subset of TEI All. So if your data has features which are not valid in TEI All, shall we say @xml:id values which are of the wrong datatype, or empty <list> elements, or <list> or <table> elements appearing directly inside <front> instead of being decently wrapped inside a <div> … it won’t be valid.

(These examples were not chosen at random, by the way: they are all the consequence of a bug– issue https://github.com/TEIC/Stylesheets/issues/604 reported this morning – in the current docxtotei tool). Anyway, this means that either the ODD needs to be adapted to be more forgiving, or the data needs to be corrected to be less weird. Doing the former would also mean tweaking the data (to avoid polluting the TEI namespace with the weirdness), so maybe choosing the latter course of action is the wiser decision. Especially since it’s not so hard to correct the aberations I have identified so far.

So my first XSLT stylesheet is simplify.xsl, which does just that. If it finds a list or a table directly inside a front it wraps them in a p and looks the other way. When it finds an anchor it sticks an extra letter in front of its identifier. After its ministrations, all 112 generated XML files (bar one, which had an empty <list> element) were valid against the generated ODD schema. Hosannah.

That leaves 51 items with no easy XML representation, or 12 items if we assume that the legacy TEI format also counts as potentially easy XML. Sadly all but 1 of those 12 are in the “ill formed” Word Press XML format, so some (more) manual tweaking will be required before I can safely apply the retagfromWP conversion to them. Then I will have to work out what to do with the legacy TEI files, some of which are still in P4. But I think I see a way forward…

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search
https://foxglove.hypotheses.org/tag/tei-archive
foxglove.hypotheses.org
CC-MAIN-2024-30
107219
application/http; msgtype=response
sha1:MICYXX3EUWNPAKDXNTK7FQBF4BAWG4RE
<urn:uuid:967a9b50-3759-4507-b9c5-c63ba244aa3a>
2024-07-21T18:47:00Z
134.158.39.132
text/html
sha1:DMQTEEUFZXGZIYID7UR2AALD76F3YJ6S
<urn:uuid:d9271b44-b328-4528-93e0-477c395837cd>
https://foxglove.hypotheses.org/tag/tei-archive
response
<urn:uuid:711e5f16-a1f8-48a0-9d5c-ee9c699952a3>
isPartOf: CC-MAIN-2024-30
publisher: Common Crawl
description: Wide crawl of the web for July 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-120
software: Apache Nutch 1.20 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/