In Depth

The motivation behind our architecture is to follow standards, to utilize open source software, and to leverage internal and external partner resources in effort to support Yale’s open access to collections. The following diagram illustrates our current implementation of online collections. 

How the technology works

Each institution has its own collections management system, database, or spreadsheet. Data from each system is exported into standard XML format and exposed for harvesting into Yale’s Cross Collections Discovery. 

For the Center, this process exposes art collection data. In technical terms this means configuring and utilizing an XML exporter (COBOAT application) to generate standard metadata files. The files are then sent to our OAI-PMH data provider (OAICatMuseum web application) that exposes the data on the web. Flat XML files can be exposed on the web as well; however, in this case a provider that supports standard protocols for regular updates and deletes is more suitable. 

At the end of 2011 we enabled a production version of the OAI-PMH data provider that is based on the Lightweight Information Describing Objects (LIDO) metadata standard. The move away from the previous CDWA Lite to a LIDO 1.0 schema allowed us to expand our metadata records while maintaining ease of use and conforming closer to CIDOC CRM compliant metadata standard.

Collections data is updated nightly and the structure of our LIDO records is expanding several times per year in production. We maintain a list of updates for institutions harvesting our data. The metadata contains references to images, and we do allow unrestricted and CAPCHA-free access to high resolution TIFF images to partner institutions providing tools and resources for our data. Images are served out using a Content Delivery Service

The primary harvester of our data is Yale’s own Office of Digital Assets and Infrastructure (ODAI). Data in various metadata standards from partner providers is normalized and goes into Yale’s Cross Collections Discovery (CCD) system. CCD is based on Apache Lucene and has a Solr Application Programming Interface (API). That API allows us to access the Center’s library and art collections together through VuFind open source software.  

All work to date has been done in partnerships with Yale campus organizations, foreign institutions, and private corporations. Due to the size of our combined campus collections and web performance needs, a combination of open source, cloud based, and commercial systems are being used. 

Metadata Systems: 

Media Systems:
https://britishart.yale.edu/collections/using-collections/technology/in-depth
britishart.yale.edu
crawl=CC-MAIN-2018-09
45759
application/http; msgtype=response
sha1:6RDJLPM45FIM2GWYZFSYSPXNJUDSD3EU
<urn:uuid:130b0b14-e949-43c8-b2aa-d03fc28f2339>
2018-02-18T08:58:42Z
52.21.112.188
text/html
sha1:6CCZ5Q36GFWFGDWPBZFVTJ7G4ASAY7ML
<urn:uuid:ed09c700-3676-473d-b668-90366feb939e>
https://britishart.yale.edu/collections/using-collections/technology/in-depth
response
<urn:uuid:9c88c2a3-8d08-40f2-9fe8-cfda82ef437c>
robots: classic
hostname: ip-10-185-198-243.ec2.internal
software: Nutch 1.6 (CC)
isPartOf: CC-MAIN-2018-09
operator: Common Crawl Admin
description: Wide crawl of the web for February 2018
publisher: Common Crawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf