Tracking for the long term

From UA Libraries Digital Services Planning and Documentation
Revision as of 17:17, 18 December 2012 by Jlderidder (talk | contribs) (The InfoTrack database)

Jump to: navigation, search

We're developing tracking databases for management of content over time. Recognizing that this is still software-dependent, and that relational databases are inherently unstable, the plan is to periodically print reports from the database and store these as flat files in the top levels of the storage system, as a form of manifest.

InfoTrack and md5sums are located on libcontent1.lib.ua.edu. The checkscripts database, which documents any errors encountered and also when each script runs (and thus when the MD5 sums are verified) is described in Tracking_automated_scripts.

The InfoTrack database

allColls

This table contains information about each collection sufficient for a PHP script to be able to dynamically deliver an alphabetized list of current online collections and EAD finding aids, descriptions, icons, links to the content (and if available, the finding aid associated, and the link to that online). Most of this information comes from the Collection_Information file created by Digital Services personnel when they begin to digitize a collection, drawing information from the spreadsheets filled out by archivists. This information is uploaded by the collToDbase script described in step 10 of Moving_Content_To_Long-Term_Storage or the moveContent script described in the 3rd section of Most Content. These scripts also add the expected canned link for retrieving content, and specify if the collection is online yet. More information about this table is available here: allColls

inLOCKSS

Once a collection has been released for harvesting into LOCKSS, we must monitor the size and additions to that content, and avoid any changes. As content is communicated over the network, we log here the identifier, manifest number, and date (and for collections which are subcollections of others, such as rare books, the subcollection title and parent identifier).

lookup

This table was created to support persistent identifiers. It's a lookup table to provide redirects. Each item for which we would like to provide this support is assigned a number. Retrieval of that item will be by using a URL of the form: http://purl.lib.ua.edu/3234 where the number following the last forward slash is the number of the item. The actual url is stored in this table (realurl), along with the assigned number (purlnum), an original identifier (id_2009), a datestamp, and a history of any changes (history). This table is accessed by the script redirect.pl which lives in the cgi-bin of libcontent1.lib.ua.edu. A URL rewrite and a virtual host configuration, along with the DNS registration of purl.lib.ua.edu, were the only other support necessary for this to work. Whenever a file must be moved, the database is updated, and the persistent URL continues to work. In this fashion, we may enter URLs into metadata, webpages, online catalogs, etc., and never have to change them.


bornDigital

This table tracks born digital content such as Electronic Theses and Dissertations, which may have embargoes on web delivery which must be tracked and supported. Fields here include the identifier (id_2009), first and last name of the primary author, collection number (collnum), datestamp entered, title, the date the content should be made available (dateAvailable, in form yyyy-mm-dd), and exceptions.


archival_formats

This table is designed to capture the type of format an file is in, a URL to information on that format, the quality of the capture, and the version of the format. For example, a TIFF version 6.0 file captured at 600 dpi would have a URL entered from the Unified Digital Format Registry ([[1]] similar to this one now available via the PRONOM Digital Format Registry: [[2]].

By maintaining a record of the current format, version, and quality of each file, in a database, we can easily identify and locate all files in need of migration or emulation in the face of approaching obsolescence.

There is a good chance that this table will be combined with the md5sums database tables.

dates_tagged

Here we track when content was rotated through the tagging software. For more on this see User_tagging

dates_transcription

Same thing for rotation through transcription software. For more on this see User_transcription

userTagged

Here we track what was tagged, so it need not be rotated back through. For more on this see User_tagging


userTranscribed

Same thing for success in transcription collection. For more on this see User_transcription

tags

This is the list of tags, what they were applied to, and how many times a particular tag was used for a particular item.

numItemPages

To find out how relevant it is to have 3 transcriptions for an item, we need to know how many pages that item has. If it has 600 pages, only 3 transcriptions (one per page) is not very successful. Here we collect info on how many pages items have within u0002 and u0003. For more on this see User_transcription

geocode

Information extracted using Google API is stored here for particular item locations. This enables us to apply the same lat/long for other items with the same location without calling the API again and again and again (there are limits on calls, as well as server overhead.

itemLocation

This works with the geocode table, indicating the item location for each of the items in the database. The locationID here corresponds to the locationID in the geocode table.


Donors

This table is not yet in use, and will likely expand. The purpose is to track which donors have provided support for digitization, processing, or donated content, and be able to link to their website and use a logo for the display of content.

md5sums

itemSums This table contains the identifier, file name, file path, current MD5 checksum, number of times modified, date of first entry, and a notes field.

modified If a file was indeed modified (in which case that would have been indicated in the itemSums table), then there will be an entry here for each modification. The timestamp of the change, the identifier, filename, previous MD5 checksum, the reason modified, and by whom are recorded here.

Dates that the files are verified are stored in the checkscripts database described in Tracking_automated_scripts
http://www.lib.ua.edu/wiki/digcoll/index.php?title=Tracking_for_the_long_term&oldid=4356
www.lib.ua.edu
crawl=CC-MAIN-2019-30
21082
application/http; msgtype=response
sha1:36IXTADQSJVDYALPUZFAHKPZB2JGIMF4
<urn:uuid:c12fd96f-18f7-4526-86cf-c2873acc78f7>
2019-07-16T16:48:44Z
130.160.24.157
text/html
sha1:M2IXROZUYQBKUXRC6GD272OZKEMRXNRP
<urn:uuid:562bb5da-ddfc-4705-a52d-d85d8b1dc62f>
http://www.lib.ua.edu/wiki/digcoll/index.php?title=Tracking_for_the_long_term&oldid=4356
response
<urn:uuid:5aef3b42-d1e3-4258-9110-5748e9ae712b>
isPartOf: CC-MAIN-2019-30
publisher: Common Crawl
description: Wide crawl of the web for July 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-168-228-208.ec2.internal
software: Apache Nutch 1.15 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/