August 2010 – Diary of a Repository Preservation Project https://blog.soton.ac.uk/keepit How digital repositories can plan for the future - by four exemplars in science, arts, research and teaching Wed, 17 Nov 2010 10:16:43 +0000 en-US hourly 1 https://wordpress.org/?v=6.2.2 KeepIt course 3: describing significant characteristics and recording provenance https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-describing-significant-characteristics-and-recording-provenance/ https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-describing-significant-characteristics-and-recording-provenance/#respond Mon, 23 Aug 2010 14:23:22 +0000 http://blog.soton.ac.uk/keepit/?p=1389 KeepIt course module 3, London, 2 March 2010
Principal topic this module: significant properties
Tools this module: PREMIS, Open Provenance Model (OPM)
Tags Find out more about: this module KeepIt course 3, the full KeepIt course
Presentations and tutorial exercises course 3 (source files)

In the original notice for this course the focus of module 3 was on description. That emphasis changed a little in the interim towards significant characteristics, as we saw in the previous session. Now that we understand which characteristics of a digital object might be significant in affecting its use both now and in the future, and how to identify those characteristics, we need to have a means to describe these characteristics and record any actions taken on the object that might alter them. Our earlier game revealed the importance of recording this information.

In this final section of module 3 we look at two means of recording information for preservation, one a well established metadata dictionary, PREMIS, the other an emerging standard, the Open Provenance Model (OPM). A participant’s view of this session can be found

On the surface there is little to connect these metadata models in terms of the people involved and the starting point for each initiative. PREMIS comes from the digital library community, while OPM could be said to have roots in the computer science and Semantic Web community. One describes over 100 elements, the other barely more than a dozen. In terms of purpose, however, there is perhaps more common ground. So might the two approaches eventually collaborate, or be in competition? This is not clear yet, although I could report my KeepIt colleague Dave Tarrant, a programmer and developer, was immediately drawn to the elegant and concise OPM, less so to the unwieldy PREMIS. I’ve always been unstinting in my praise for PREMIS.

Preservation Metadata Implementation Strategies (PREMIS)

[slideshare id=3365403&doc=keepit-course3-presmeta-100308093709-phpapp01]

When I first began looking at preservation metadata in 2004, there were many schemes. It was clear that someone would have to try and unite these disparate approaches, and thankfully PREMIS filled that gap. Yes, the PREMIS Data Dictionary for preservation metadata is extensive and initially daunting, but once you understand the structure, the interested and informed reader cannot but be struck by its coherent and comprehensive treatment. Each time I give this presentation – and I have done so previously for events by the Repositories Support Project, such as in 2008 in London and Bournemouth – I want to give those new to PREMIS a sense of this contribution.

The other key point from this presentation, invariably aimed at repository managers and administrators, is however daunting PREMIS may appear, it is very likely they are already recording PREMIS metadata. One aim of the exercise attached to the presentation (slides 8, 9) and associated preservation metadata worksheet, is to select elements from the Dictionary that will be familiar to repository managers, such as object identifier, file size and format. Having established the basic familiarity of PREMIS, the exercise then asks participants to identify the sources of information for 20 PREMIS elements listed on the worksheet.

On this day, such was the focus on significant characteristics that we did not have time for the full exercise, working in groups, and instead treated it as a discussion exercise.

One of the PREMIS elements included in the exercise is labelled significantProperties, but in terms of what we have learned about this topic in this module, this element alone is clearly not very substantial. PREMIS is doing more work to accommodate significant characteristics, and the final slide (11) here reveals some likely directions.

Open Provenance Model

“As data becomes plentiful, verifiable truth becomes scarce”. Since provenance is about records to verify the past history of an object, be it an antique or any physical or digital object, this great quote from Eric Hellman instantly reveals why provenance will become increasingly critical for digital information.

OPM is driven by an international community, and its lead author is our colleague Professor Luc Moreau at the University of Southampton. We attended a seminar by Luc on OPM, read the documentation, and met with him. Apart from the opening slide, the slides used here are selected from one of his presentations on OPM, so aside from the editing and this commentary this is all bona fide original OPM, and we are grateful to Luc for permission to reproduce this material.

Given the time available and, as we noted earlier, the origins in another community, this presentation is quite pared back, so please try to find out more about OPM from the sources provided. What we wanted to do here is introduce the general concept of provenance, put it in the context of a familiar information lifecycle, in this case the science lifecycle, and then identify the main elements of OPM.

[slideshare id=3365483&doc=keepit-course3-provenance-100308094733-phpapp01]

From a tools perspective, a critical feature of OPM is its support for ‘serialisation’ (slide 8). This allows information describing an object to be passed to different services. RSS and Atom are well known serialisation formats. OPM supports serialisation formats in XML, RDF and others, and supports the Semantic Web ontology language OWL. These formats can be used by repositories, and EPrints repository software, for example, now supports provenance information based on OPM.

OPM uses ‘nodes’ (slide 9); immediately we can start to see some parallels with PREMIS. Nodes are connected by ‘edges’ denoting the relation between nodes (slide 10); this begins to look like the graph-based approach of RDF, such as is used by OAI-ORE. Vitally, OPM graphs represent relations in time (slide 12). Finally, we see how Dublin Core, which will be familiar to repository administrators, might be mapped to OPM to represent the process of publication and versions of the published item (slides 13, 14).

By working with an open standard model, that can pass information as XML and in standard serialisation formats (e.g. RDF), it is possible to build provenance services into repository environments.

Summary of module 3

Module 3 of this course has demonstrated that actively used digital documents have critical features that affect productive use in different circumstances, and will change perceptibly and imperceptibly over time. We have recognised that we have to record this information, and have briefly used some tools to help do this.

Our arsenal of preservation tools is growing by the module. Our participant reporter noted that ‘for practical purposes there really needs to be some substantial work undertaken to integrate resources as well as applications to support content creators as well as repository managers in developing policies and practises for preservation.’

In module 4 we will see how these approaches can be built into the systems we are already using, our digital repositories.

]]>
https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-describing-significant-characteristics-and-recording-provenance/feed/ 0
KeepIt course 3: significant characteristics https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-significant-characteristics/ https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-significant-characteristics/#respond Mon, 23 Aug 2010 14:06:50 +0000 http://blog.soton.ac.uk/keepit/?p=1369 KeepIt course module 3, London, 2 March 2010
Principal topic this module: significant properties
Tools this module: PREMIS, Open Provenance Model (OPM)
Tags Find out more about: this module KeepIt course 3, the full KeepIt course
Presentations and tutorial exercises course 3 (source files)

Significant characteristics are “ the characteristics of digital objects that must be preserved over time in order to ensure the continued accessibility, usability, and meaning of the objects”.

To continue introducing the more technical aspects of preservation workflow in course module 3, we welcome Stephen Grace and Gareth Knight from the Centre for e-Research at King’s College London (KCL), to explore the practical implications and impact of significant characteristics. Alternating presentation duties, Steve and Gareth split their coverage into six self-contained presentations interspersed with practicals and discussion.

[slideshare id=3364267&doc=1sp-introducingsignificantproperties-d2-100308064856-phpapp02]

Presentation 1

One of the difficulties of digital preservation is that although we as users hope to be able see what we want in a digital object, this may not be the same as another user, or the author or creator. In other words, there is scope for interpretation. In addition, machines interpret and present the object depending on the technology framework available at a given time. As we know, that framework can change. So if we allow the possibility of creating a new version of an object – e.g. by format migration, which by definition must change something in the object, however small – to enable presentation in a contemporary technical framework, how do we decide what changes to the object are to be permitted? Such an analysis is the focus of what has become known variously as significant properties or significant characteristics (although Dappert and Farquhar argued for ‘characteristics’ and against interchangeability of these terms).

Perhaps surprisingly, given its wide scope and multi-faceted nature, the topic of significant characteristics has received far less attention than, say, file formats, as the timeline in presentation 1 recognises.

[slideshare id=3364378&doc=2sp-inspectframework-100308070838-phpapp01]

Presentation 2

A recent substantive investigation of significant characteristics was carried out by the InSPECT project, led by KCL. This project provided a framework for determining significance based on Function-Behaviour-Structure (FBS), originally developed to assist engineers and designers to create and redesign artefacts. The FBS approach is used throughout subsequent presentations in this set and in group practicals, and presentation 2 introduces the method and adapts it for data objects:

  1. Analyse structure for technical properties
  2. identify the purpose of properties and categorise
  3. determine expected behaviours
  4. and classify into functions

However, the behaviour of an object may vary depending on the user, or ‘stakeholder’ in the analysis presented. Which activities might the user wish to perform on the object? In this presentation we learn how to cross-match object functions with stakeholder functions.

Group exercises

[slideshare id=3364808&doc=3sp-practicalobjectanalysis-100308081920-phpapp02]

Presentation 3

Now we are ready to test our understanding of this framework with our first group exercise, designed to analyse the content of a familiar type of digital object, an email, and consider how it will be used. Presentation 3 invites participants to start (slide 3) with a blank FBS structure (or rather an SBF structure, since we are performing the analysis on this direction), and shows a generic example (slide 4) of what this might look like at the finish. (Note, some of the detail of these slides may be better viewed on the originals – use View on Slideshare button, lower right-hand corner of the slide viewer – or the source MS Powerpoint, see link at the top of this blog entry.) A handout sheet to support the exercise tabulates and defines the elements of an email, also providing a functional description and examples. Users were also given a sheet with a blank SBF framework to complete. Among the source materials you will find a version of the framework filled out with some possible answers.

[slideshare id=3364939&doc=4sp-practicalstakeholderanalysis-1-100308083910-phpapp02]

Presentation 4

The second part of the exercise, presentation 4, links the needs of different stakeholders with the actual behaviours required of an email. The stakeholders considered in this exercise are the creator, a recipient, and a custodian (e.g. an archivist). Groups performing the exercise are asked to identify 2-5 behaviours for each stakeholder. You will notice some parallels with the first exercise. In the final slide participants are asked to review the correlation between object and stakeholder properties.

A participant’s perspective of these exercises and of this course module has been reported.

[slideshare id=3365063&doc=5sps-in-archive-100308085155-phpapp02]

Presentation 5

By this stage we are becoming aware that it is not feasible to manually maintain a large number of digital objects. Presentation 5 introduces a selection of tools available to perform anaysis of digital objects, including DROID for format identification, JHOVE for format validation, and XCL (eXtensible Characterisation Language) to extract and document object properties.

[slideshare id=3365130&doc=6sp-summary-100308085925-phpapp02]

Presentation 6

Concluding, presentation 6 considers the role of significant characteristics in repository workflows.

Next, we consider how to describe these characteristics, and how to record actions taken on the digital object that might alter them.

]]>
https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-significant-characteristics/feed/ 0
KeepIt course 3: Introducing preservation workflow and format risks https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-introducing-preservation-workflow-and-format-risks/ https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-introducing-preservation-workflow-and-format-risks/#respond Mon, 23 Aug 2010 13:47:27 +0000 http://blog.soton.ac.uk/keepit/?p=1356 KeepIt course module 3, London, 2 March 2010
Principal topic this module: significant properties
Tools this module: PREMIS, Open Provenance Model (OPM)
Tags Find out more about: this module KeepIt course 3, the full KeepIt course
Presentations and tutorial exercises course 3 (source files)

After the simple game designed to highlight significant properties of data, we are ready to begin on preservation workflow and format risks. First the slideshow introduces some terms we will encounter later in the module: representation information, and the Open Archival Information System (OAIS). Unusually, some brief pre-course background reading was recommended to familiarise participants with the role of file formats: Preservation & Storage Formats for Repositories.

[slideshare id=3364088&doc=keepit-course3-workflow-100308062001-phpapp02]

By combining data with representation, e.g. markup (or encoding in machine terms), by analogy this can be seen as a ‘performance’ of the digital object (slide 3). While this is easy to understand for audio-visual content, the analogy holds for text and other forms of content too: an element of machine performance is required to present content to an end user. We can see already there are a number of components required for a performance of a digital object, and our concept of performance introduces expectations, perhaps different, on the parts of creator and user.

Our framework for preservation is described by OAIS (slide 4). From one view this model looks much like our repository model. A strength of OAIS is you can use it to model processes in almost any system, but once you subscribe to its strictures you are bound to its detailed specifications. OAIS is not perfect but is a standard and is universally adopted in formal preservation practice. As you build preservation into your repository you will want to demonstrate that others can trust your procedures and ability to manage data, and to do this you will need to show compliance with OAIS.

Broadly we can reduce OAIS to our three-stage repository model (slide 5), and within our ‘manage content’ activities we can identify the three stages of our format preservation workflow: check, analyse, action. When we stretch these out chronologically (slide 6) we can see there are tools for each stage, but the hard part is connecting format check with action. In other words, when we have identified the format of a digital object, we can migrate that to another format if needed, but connecting this information and the action decision to migrate is harder because we don’t yet know how to evaluate whether or when to convert, and what to convert to. Again we can see there are tools to help us with this. Most of our work in course modules 3 and 4 will concern this ‘analyse’ bit in the middle of our workflow.

Rosenthal has argued against preemptive format migration, arguing for the relaxed rather than aggressive approach, and referring to format obsolescence as the ‘prostate cancer’ of preservation: “It is a serious and ultimately fatal problem. But it is highly likely that something else will kill you first.” (slides 7-9)

Let’s look at what types of file format we might find in a typical institutional repository. We have two example format profiles from the Registry of Open Access Repositories (ROAR), past and present (slides 10, 11). Both show the dominance of PDF, not just the generic format but different versions of PDF. A recent summary of responses to a list mail enquiry about repository file formats partly explains why PDF dominates (slide 12), but is this justified?

We now know there are many factors that affect the risk associated with any file format, and 10 principal risk classifications have been identified by the National Archives (slide 13). We will use this classification as the basis of our first exercise in this module. Without explaining these factors in any more detail than is given in the slide, but after striking out two factors that might require prior knowledge or documentation (slide 14), our groups were asked to select and compare a pair of formats, to score the formats against each other for each risk factor, and produce a final score and recommendation in favour of one or other format. In addition, groups were asked to suggest a reason why the ‘winning’ format might not be the best choice for their repository (slide 15). The results were reported in an earlier blog post. This exercise is described in a handout sheet used on the day, as well as in the course slides.

We used to think that the best formats were open: open standards and open source (slide 16). Well that tended to differentiate formats from Microsoft applications from the most of the rest, and perhaps explains the recent history of PDF preference by repositories, but more recently Microsoft has lobbied successfully for its Office formats to be released as open standards. So now which popular text and document formats are best for preservation? MS Word, which can be converted to Open XML, or PDF, which can’t (refer again to the results of our group exercise)? It shows there is no single answer, and even if there was, it might have to be reconsidered at another time. Rosenthal provides some reassurance about the impact of format risk, even for proprietary formats (slide 17).

Repositories know they must work with their authors and avoid imposing unnecessary constraints. Authors create digital objects with no doubt little regard for format over functionality, so repositories should neither impose format restrictions or require authors to convert formats (slide 18). For now repositories should require source files from authors and make their own decisions, and that is what modules 3 and 4 will help them to do.

Before we end this section, let’s return to the exercise and consider the comparison of image formats (slide 19) – TIFF vs JPEG – rather than document formats. Again, there are cases for each. This might seem unlikely after three major institutions chose JPEG as an archival format over TIFF (slide 20), but an independent evaluation of both formats using the Plato preservation tool, which we will cover comprehensively in module 4, was less decisive, leaving a final decision to a later time (slide 21). Whichever formats we consider, format migration is not an open and shut case, and could use more sophistication than has been available to repositories thus far.

In a sense file formats are nice and well defined, if sometimes highly technical and complex. What is less well defined are other characteristics of a digital object that may affect its performance, or the way it is perceived, used and understood by different users. Our next session in this module will introduce and provide working examples of what is referred to as significant characteristics.

]]>
https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-introducing-preservation-workflow-and-format-risks/feed/ 0
KeepIt course 3: game playing characterisation, transmission, metadata and provenance https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-game-playing-characterisation-transmission-metadata-and-provenance/ https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-game-playing-characterisation-transmission-metadata-and-provenance/#respond Mon, 23 Aug 2010 13:38:06 +0000 http://blog.soton.ac.uk/keepit/?p=1337 KeepIt course module 3, London, 2 March 2010
Principal topic this module: significant properties
Tools this module: PREMIS, Open Provenance Model (OPM)
Tags Find out more about: this module KeepIt course 3, the full KeepIt course
Presentations and tutorial exercises course 3 (source files)

So far in the course we have covered institutions and costs. When do we reach ‘preservation’? We begin that journey into some of the more technical aspects of preservation here, and it will take us into KeepIt course 4 as well, when we will discover tools that can help us with repository preservation workflow from within your repository software.

There is a slightly different structure to this module compared with the first two modules: more tools, more and shorter presentations, and also practicals mixed in, but this module is less about practicals and more a primer for the tools that follow.

[slideshare id=3363963&doc=keepit-course3-intro-100308055942-phpapp02]

The emphasis of the day will be on understanding the significant characteristics of digital objects that might affect preservation decisions. Later we’ll look at tools for recording metadata about these characteristics and preservation actions using PREMIS, an authoritative community-based standard, and the Open Provenance Model, an emerging and perhaps a more concise approach.

First we played a short game. Each group of four people was handed some simple information: three groups received randomly generated numbers on a small piece of paper; one group was handed four playing cards. The groups were asked to transmit the information in various ways, either written or spoken. In each case, the information was to be transferred sequentially between adjacent members and not shared openly with others in the group. The results from two groups, using written and spoken transmission, respectively, are shown below.

Provenance numbers game: written and spoken transmission

We can see that each group were given a PIN-like four digit number that was colour coded. The whispering group (right) transmitted both numbers and colour correctly. They had clearly decided the colouring had significance and retained this information. The writing group chose not to retain the colours. (This evidence was retrieved after an obvious and desperate attempt at disposal!) Had they been handed information identified as a PIN number, for example, this would have been a reasonable omission, because colour has no significance in a PIN. This was not in the instructions, however. In addition, an error was transcoded in the first transmission (top), retained in a second transmission, and somehow corrected in the final result (bottom). This last transcription must have broken the rule about group conferrals, but we can forgive them because it demonstrates how important recording can be in a transmission chain to enable errors to be identified and corrected.

In the case of the group receiving the playing cards, without being told this the information they were expected to transmit was number (or picture) and suit, as these are the two familiar significant characteristics of playing cards, although they may not be commonly referred to in this way.

A simple 10 min game had revealed all the key issues to be covered in module 3:

  • the importance of recognising significant characteristics in deciding which aspects of information are to be retained,
  • the effect of transmission on information, and
  • the need to record data and metadata to identify and correct errors, to record changes and provenance, decisions made and actions taken on data.

Next we will introduce a format preservation workflow and format risks.

]]>
https://blog.soton.ac.uk/keepit/2010/08/23/keepit-course-3-game-playing-characterisation-transmission-metadata-and-provenance/feed/ 0
https://blog.soton.ac.uk/keepit/2010/08/feed/?withoutcomments=1
blog.soton.ac.uk
CC-MAIN-2023-50
40667
application/http; msgtype=response
sha1:M654LJ2P24T27XKJGBZYQ4HRNFRBOR7E
<urn:uuid:e424db7d-f626-4581-b345-86b5342c86fe>
2023-12-03T00:46:55Z
152.78.118.190
application/rss+xml
sha1:A5YRHQHRESPAYKKLHMK5TBJ6TDZH3KKU
<urn:uuid:bc3f2ec8-c0a5-477e-9fbe-11cc2c38268f>
https://blog.soton.ac.uk/keepit/2010/08/feed/?withoutcomments=1
response
<urn:uuid:f4f0ad08-7d2d-4b97-bee2-9c0019b4d90d>
isPartOf: CC-MAIN-2023-50
publisher: Common Crawl
description: Wide crawl of the web for November/December 2023
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-40
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/