Archive for Uncategorized

Bring out your dead!

Yesterday I attended the Digital Preservation Coalition’s ‘Bring out your dead!’ day (subtitled more soberly: ‘Collaborative approaches to managing file formats – a day of action’. Monty Python reference appreciated, though). The point of the day was to discuss the problems that file formats present to digital preservation (and also to bring along our own problem files). These obviously include dealing with very unusual files and files of unknown type. There are tools, such as DROID, which can help with identifying unknown files. The surprise for me was how much of the discussion focussed on older versions of very well-known file types, particularly Microsoft Office files. Chris Rusbridge, in his opening presentation, talked about his experience of trying to convert his old PowerPoint files, created years ago on an old Mac, to a current format (he found a company that could do it) and also his open letter to Microsoft about publishing the specifications for old versions of their file formats. The latter had surprising results (Microsoft willing to help, but they don’t have the specs).

 

Much of the discussion also centred on the importance of collaboration to solving the file format problem. A key part of this is contributing information to file format registry projects such as PRONOM and CRISP, which suffer from lack of detail in their records (though TNA’s David Clipsham explained that the focus for PRONOM had been on populating it with file signatures, rather than other details). David Clipsham presented a session on how to produce file signatures which, to a non-techy like me, was an eye opener. It was especially interesting to see how easy it is. Essentially, a file signature is a string of binary data which always appears within a file of a particular sort and so which can be used by a tool such as DROID to diagnose the file type. To spot these strings, all one has to do is open a number of examples of the file type in a hexadecimal editor and flick through the open tabs quickly (a bit like a flicker book) to see which bits of the file stay the same (usually it’s the beginning bit). Submitting examples of file signatures to PRONOM helps to build up the registry and increase its usefulness.

 

Other ways that we need to collaborate as a sector include better coordination between projects and coordinated lobbying of software companies (and government), of the sort pioneered by Chris Rusbridge. There are several projects which have attempted to create file format registries and the consensus seemed to be that it isn’t necessarily a bad thing to have more than one registry, but that data needs to be shared systematically (preferably automatically) between them. The same body of data shared in several places isn’t bad, but data split between several places is. In the lively final discussion a big topic was the need for someone to take a strong leadership or coordination role in carrying forward the lobbying of software companies to release file specifications. Chris Rusbridge rather theatrically put William Kilbride of the DPC on the spot and William deftly turned this round as a question for the DPC’s members as to whether the DPC was the organisation to take a lead in this.

 

If I have a criticism of the day it was that it was billed (unless I got the wrong end of the stick, which commonly happens) as a workshop for dealing with delegates’ problem files (the ‘dead’ of ‘Bring out your dead’). The discussions and workshops were interesting, presenting a number of tools for charactersing files and creating signatures. However, it was less hands-on than I expected. I came down with a slightly different problem to that of unknown/obsolete file formats. We have some TIF files which create problems when converted to JP2 and streamed and we don’t know why. I didn’t get any answer from the workshop itself, but I was at least able to discuss it with other people during the break and find a possible source of help. Which was useful!

Advertisements

Comments (1)

Digital Library Improvements for the autumn term

Over the summer we have been working with a student on a usability study of York Digital Library. The resulting dissertation has given us valuable data for the ongoing improvement of the Digital Library. As a first step, we assembled a list of priority issues for the start of this academic year and have pushed forward with a ‘usability sprint’. The following list of changes are mostly small, but together we feel they will help students and other users navigate around the Digital Library and get more out of our collections.

All of these changes will be visible early on Sunday 7th October.

Release Notes – Digital Library – Version YODL 2 2012-09

Usability and Display Issues

  • Artist names do not display in the metadata detail page if there is no role specified (eg. sculptor)
  • Attribution field does not display in image metadata detail
  • Reorganise label information for location image detail
  • Add a hyphen between earliestDate and latestDate in image metadata detail
  • Problems with display of rights info for images
  • Display multiple titles in image detail page
  • Make order of fields in ‘home’ detail view consistent
  • Truncate long titles in search results display
  • Make description display for collections in collections detail page
  • Make sort order of browse tree and results alphabetical by default
  • Indexing Issue with DC, some fields are not searching; allow all fields to be searched
  • Change label for location in image detail page, for clarity
  • Correct bug where creator appears twice in home detail page
  • Correct bug where type displays twice on collection detail view
  • Add which collection(s) a resource belongs to for the resource detail page
  • Offer sort options for browse/search
  • Display date with search results on Exam Papers
  • Breadcrumb trail for search results
  • Change sort order in browse to alphabetical
  • Exclude openart results from search, as reported as confusing to users
  • Remove thumbnail from download list on all objects, as reported as confusing to users
  • Make “about the image” open by default in image metadata detail
  • Refresh featured items on homepage
  • Improve introductory information and help on homepage
  • Add a general help page
  • Add a help pop-up by the download button
  • Fix bug in sort by in search
  • Add a link to the help page from ‘need help’ section on homepage
  • Re-order collection metadata display: dd subject before rights in collection
  • Enhance collection metadata

Access and security improvements

  • YODL login / session management improvements, to prevent users from being ‘thrown out’ of their logged in session
  • Make Masters theses metadata public; restrict files to York users
  • Make top level and department exam papers collections metadata public to aid navigation

Leave a Comment

York Digital Library Survey & Focus Groups

Have you ever searched online for University of York resources? If you have, it’s likely that you’ve used the University of York Digital Library (YODL). Named YODL for short, this where past exam papers are stored, as well as a huge number of History of Art images and other resources.

The library is currently looking for feedback from students and staff on YODL, so if you’ve ever used it, we’d love to hear your opinions via our short questionnaire; the link is on the front page of the site.  If you haven’t used YODL before, perhaps you would be interested in taking a look around the site at http://dlib.york.ac.uk/ before completing the questionnaire. All responses will be anonymous, but if you do want to leave an email address, you’ll be entered into a prize draw to win a £25 Amazon voucher.

We are also looking for participants for a focus group to give us more detailed feedback on the site; participants will be paid £10 for approximately 30 – 60 minutes of your time. Again, no previous experience of the site is needed; just have a look around to form some opinions before taking part. Even if you think you don’t have much to say, we want to hear it!

There are two focus groups, one for staff and one for students:

The staff focus group will take place this Thursday, 28th June at 4pm in LFA/205 (top floor of the Harry Fairhurst Building).

The student focus group will take place on Wednesday 4th July at 11.30am in LFA/205 (top floor of the Harry Fairhurst Building).

Please email mgedwards1@sheffield.ac.uk if you’d be interested in taking part.

Leave a Comment

Digital Futures Academy

Last week I attended the Digital Futures Academy, a course on digitisation run by King’s Digital Consultancy Service at the British Library in London. I had won a scholarship to attend this from the Digital Preservation Coalition, to whom many thanks. I thought I‘d jot down a few key thoughts that I took away, before they get swept away in the stream of everyday this-and-that, like when you go on holiday and the memory of sitting on the beach stays with you for about a day and a half and then your brain files the memories away in a box somewhere labelled ‘things that happened ages ago – possibly for deletion’.

A key thought that kept recurring in the week was the importance of thinking in terms of people, rather than data. Data is the trace of human activity and aspiration. Data fulfils human needs and provides opportunities. This came through particularly when we were discussing the planning and pitching of digitisation projects. The questions we need to ask are: who will benefit? what need does the project fulfil? what will be the outcomes, in terms of people using the data? what is your narrative?

William Kilbride of the Digital Preservation Coalition, one of the speakers, presented an interesting idea, that of the ‘attention economy’ rather than the much-vaunted ‘knowledge economy’. Knowledge and data are not what is scarce, but time. The most valuable thing in this economy is the time people take to engage with your content.

Alastair Dunning, of the European Library, another one of the speakers, presented a nifty analogy of the boutique versus the shopping mall. He presented a photograph of a quaint shop, stuffed with wares and full of character, and said that very often the services we create are like this: beautiful presentation of wares, but hidden away and nobody goes there. The next image was of a shopping mall: soulless and impersonal, but everybody goes there. This stands for the aggregators and mass audience sites (such as Flickr and Facebook). The boutique is fine, but it is important to get our content in the shopping mall, where it will be seen. The final image was a cautionary one: a boarded-up and derelict shop!

Overall, I found the course stimulating and thought-provoking. It raised as many questions as it answered – as it is intended to (there is often no universally correct answer with digitisation projects, but it helps to start by asking the right questions).

Matthew Herring

Leave a Comment

York Cause Papers, another Digital Library project

This morning sees the launch of the York Cause Papers images, the result of a JISC-funded rapid digitisation project which ran through to the summer of 2011. The Digital Library has been responsible for taking the scanned images and ingesting them into a Fedora Commons repository, using a configured version of our YODL interface. The HRI in Sheffield host and mange the searchable database of the Cause Papers, and have added links from this database to the image repository, hosted here in York.

The Cause Papers are a fascinating resource, with originals held in the Borthwick Institute for Archives. Further information about the papers and the project can be found in the University’s press release.

To search the York Cause Papers database and find page images visitwww.hrionline.ac.uk/causepapers. Images may also be accessed directly from http://dlibcausepapers.york.ac.uk.

 

Leave a Comment

OpenART Technology Choices

During the course of the OpenART project we’ve come across a number of different technologies and standards that we could use. In this post we aim to go into the ones we’ve used and found useful.

We aim to answer the following series of questions:

What technologies, frameworks, standards are you using in your project? What are your impressions of them? What difficulties have you encountered? What approaches and techniques have worked well? What advice would you give to others engaging with them?

“What technologies, frameworks, standards are you using in your project?”

Data preparation and mapping/triplification/lifting

The source information was supplied as a set of Excel spreadsheets containing semi-structured data. These spreadsheets were provided by the researcher and were the “live” data capture mechanism for the transcribed source data. As such they were subject to structural changes during the lifetime of the project. Various tools and approaches for manipulation of this source data were explored.

  • Relational Database (RDBMS)
    • Early in the project an approach of migrating the data to an RDBMS was explored
    • This resulted in a cleaner version of the source information, but the approach was abandoned as the spreadsheets were a live tool
    • The RDBMS ontology mapping plugin for Protege was explored for transfer of RDBMS data to RDF/OWL
  • Excel Spreadsheet manipulation
    • Google Refine was used for very quickly visualising and manipulating source information
    • RDF Extension for Google Refine was used for exporting cleaned up data held in Refine to test versions of RDF linked data, and for for matching nodes to validate and connect parts of the data (via the reconciliation feature)
    • GNU SED (Stream Editor) is a powerful general purpose text manipulation tool. This hits the cleaning spots the developers can’t reach (quickly) in the visual Refine environment.

Ontology development

  • Protege version 4.1 was used for ontology development. Protege is a powerful and fully featured ontology IDE (integrated development environment)

Linked Data manipulation and production

  • Rapper (part of the Raptor RDF Parser toolkit) was used for translating between various RDF serialisations. Rapper is a parsing and serialising utility for RDF.
  • SPARQL (A standard query language for RDF) for querying and checking dtata
    • Rasqal via the online service at Triplr. Rasqal provides RDF querying capabilities, including SPARQL queries
    • ARQ, a SPARQL query engine for Jena

Hosting and base systems

  • openSUSE Linux, a major Linux distribution, was used as a base operating system
  • SuseStudio was used to build and deploy bespoke virtual machines to Amazon EC2. SuseStudio allows you to choose packages to build the virtual machine and will directly deploy to an Amazon web service account all within a web-based interface
  • Amazon Web Services (AWS) was used for hosting, particularly the Elastic Compute Cloud (EC2) service for virtualisation
  • The Apache web server with mod_rewrite was used for web hosting of ontologies and data, with content negotiation
  • OntologyBrowser was used for for storing, viewing, manipulating and accessing the ontology, using a web front end and REST API

“What difficulties have you encountered?”

Source data manipulation and conversion

  • The source information was being actively worked upon during the lifetime of the project. This presented difficulties as working on the ontology and triplification to RDF concurrently with changes in the source information made decision-making and coordination difficult
  • The source information was provided as Excel spreadsheets. These were used as an organisational environment for capturing data by the researcher, and provided free text search, categorisation and a basic namiing scheme for entities. Although a degree of rigour was applied, the environment could not be described as “structured data” such as that provided by a database, and this provided challenges in interpreting and modelling the information.
  • Originally the project assumed that hosting of the information as part of the Court, Country, City project would provide a structured source for the information; however the requirements from this project were sufficiently divergent from the OpenART project’s requirements for this not to be the case. As a result the project did not consider alternative automated processes to assist in source data manipulation and triplification until late in the project.

Ontology development tooling

  • Bulk changes in the ontology using Protege was difficult to do efficiently
  • Synchronisation between environments was a challenge with new versions of ontologies

Ontology development informational issues

  • Difficulties in extending the combined ontologies – particularly when trying to move from earlier versions expressing “simplifications” to a more complex framework later in the project. It would have been easier to start with a more complex framework initially rather than trying to extend the earlier version.
  • Naming conventions became very cumbersome, for example when trying to create inverse properties that sometimes had no natural or simple English language fit

“What are your impressions of them?”

  • Protege and RDF tools used locally were found to be very mature. Protege is now mature enough to be usable by newcomers; and also provides a route to further development as it is based on an OWL-API. However it was difficult trying to reason with anything other than a small set of data points.
  • Google Refine was surprisingly efficient and workable. Additional functionality such the as reuse of RDF mapping templates was useful.
  • Cloud services (Amazon EC2) were relatively easy to use; and the combination of openSUSE and SuseStudio provided an efficient and repeatable mechanism for deploying virtual machines
  • OntologyBrowser presents a nice iinterfacae for ontology browsing in HTML. Its user-friendly syntax for ontology axioms means that you don’t need to be an OWL or logic expert. However it is very “data-centric” and not an end-user environment.

“What approaches and techniques have worked well?”

  • Google Refine is good for rapid development when a non-trivial ontology requires that mappings and ontology need to be developed in tandem
  • Use of Protege (see above)
  • Use of cloud services (see above)

“What advice would you give to others engaging with them?”

  • Investigate using a collaborative ontology environment, such as knoodl
  • Invest a small amount of time in collaborative environments, this can lead to a big payoff; cloud services are relatively easy to use
  • Start with an ontology framework complex-enough to represent the modelling required; it can be difficult moving from a simple to a more complex version
Author: Martin Dow

Leave a Comment

Exploring different approaches to getting stuff done

I’m probably not alone in having some budget to spend up before July, the University financial year end. Like most Universities, we’re year facing a lean year in 2011/12 , so I have been trying to make best use of the funds I have available, within the timeframe given.

As part of this, I’ve identified some pieces of technical work and put these out in an ITT.  The full ITT is available from:
https://vle.york.ac.uk/bbcswebdav/xid-901764_3

In brief, the three pieces of work are:
1) Implementation of the University of York Archives Hub Spoke and archival stylesheet development

2) Re-usable EAD generation and conversion (from spreadsheets)

3) Implementing page turning and sequencing within a Fedora Commons repository

There are many more things we want to do in the Digital Library but I have chosen these three to help us get a feel for how this approach would fare in the community, whether there are contractors out there looking for small pieces of work like this.

Leave a Comment

Older Posts »
https://yorkdl.wordpress.com/category/uncategorized/
yorkdl.wordpress.com
crawl=CC-MAIN-2017-43
63345
application/http; msgtype=response
sha1:REGXZDLINXGJR3OLKJI7J3DJDJ77YIB7
<urn:uuid:ee9d37ec-ce4b-434c-b3bc-fe0fcd01cb20>
2017-10-22T06:34:08Z
192.0.78.12
application/xhtml+xml
sha1:P6FHBD4IHMSR77DI5AKWJQLXKS7Q7M6F
<urn:uuid:1d5bd6fc-cad2-4d3e-9ec5-d7e34bee32a6>
https://yorkdl.wordpress.com/category/uncategorized/
length
response
<urn:uuid:07e7e55a-cf53-4219-a9e9-79fa8044fca3>
robots: classic
hostname: ip-10-228-0-15.ec2.internal
software: Nutch 1.6 (CC)
isPartOf: CC-MAIN-2017-43
operator: Common Crawl Admin
description: Wide crawl of the web for October 2017
publisher: Common Crawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf