Integrating the Fedora based DOMS repository with Hadoop, SCAPE Information Day, 25 June 2014

  • 53 views
Uploaded on

The State and University Library, Denmark, hosted an information and demonstration day on 25 June 2014 for delegates from other large cultural heritage institutions in Denmark. The information day …

The State and University Library, Denmark, hosted an information and demonstration day on 25 June 2014 for delegates from other large cultural heritage institutions in Denmark. The information day introduced the EU-funded project SCAPE (Scalable Preservation Environments) and its tools and services to the participants. Read more about the event in this blog post, http://bit.ly/SCAPE_SB_Demo.
One of the presentations was given by Asger Askov Blekinge who showed how the library has worked on integrating its digital object management system with Hadoop. The library is currently digitizing 32 million newspaper pages and is using Hadoop map/reduce jobs to do quality assurance on the digitized files with the help of the SCAPE Stager/Loader so updated QA’ed files are stored in the repository.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
53
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Asger Askov Blekinge State and University Library, Denmark SCAPE Information Day State and University Library, Denmark, June 25th 2014 Integrating the Fedora based DOMS repository with Hadoop
  • 2. • Each File is stored in Bit Magasinet, our bit preservation storage system. • Each Record is stored in DOMS and have have reference to the File in Bit Magasinet • Can Hadoop be added to this setup? 2 Our Repositories This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 3. • Rule 1: The size of the Hadoop cluster should be independent of the size of the data storage • The reading of data should happen from local disks. This prevents a central storage system from limiting the speed of the cluster • With this restriction, the number of nodes in the cluster can keep growing • Without, the cluster will reach a point where it will overload the central storage system. 3 Hadoop Data Locality This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 4. • Repositories, especially Fedora 3.x, are single headed. You cannot add more machines to the repository to increase the performance. • If Hadoop accesses the repository directly, it will be limited to the speed of the repository. 4 Repositories (DOMS) and Hadoop This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 5. • Hadoop provides it's own bit archive system in the form of HDFS, which is integrated with the cluster • We do not use this. We have built our own system instead, Bit Magasinet • We can handle many more files because we use magnetic tapes, rather than disks. • But: it require us to request a number of files, which will then be made available for Hadoop. 5 Bit archive systems and Hadoop This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 6. • Hadoop does not play nice with DOMS or Bit magasinet • This state of affairs is not acceptable to us. • Besides, it is a nice challenge ;) 6 State and University Library This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 7. 7 How we do it in the Newspaper digitisation project This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐ • Files are stored in Bit Magasinet • One Batch Object – Batch object have list of files • One Record per File
  • 8. • A Hadoop map/reduce job is split into two steps – Map, where the work on each “record” is performed. – Reduce, where the results are collated • In the Map step, we run the tool on the file. – We have a lot of Map workers. • In the Reduce step, we store the results in the repository. – We have only a few Reduce workers. 8 How we do it in the Newspaper digitisation project This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 9. ● Retrieve the list of files from DOMS ● Request these files from Bit Magasinet ● Start Hadoop job on files – Map: Run Jpylyzer on each file (Many worker nodes) – Reduce: Store the results back in DOMS (Few worker nodes) ● This way, the actual work on the records is not connected to DOMS, and we can scale the cluster 9 How we do it in the Newspaper digitisation project This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 10. ● Staging: Retrieve the records from DOMS to an archive file ● Hadooping: Hadoop reads the records, work and writes new, updated records to the archive file ● Loading: Store the updated records in DOMS 10 How we do it in SCAPE This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 11. ● SCAPE has devised a repository agnostic object format based on METS – github.com/openplanets/scape-platform-datamodel ● SCAPE has designed a generic repository REST interface – github.com/openplanets/scape-apis ● SB has implemented the SCAPE Repo API for DOMS – github.com/statsbiblioteket/scape-doms-data-connector ● We have implemented a client for the SCAPE Repo API – github.com/statsbiblioteket/scape-stager-loader 11 Step 1 – Retrieve records This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 12. 12 SCAPE Datamodel mapping This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 13. <mets:mets ID="scape-entity:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af" OBJID="scape-entity:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af" PROFILE="scape"> <mets:metsHdr RECORDSTATUS="NEW"/> <mets:dmdSec ID="DMD-8c72c14d-475a-49a2-9f43-321732c4e7a2"> <mets:mdWrap MDTYPE="OTHER"> <mets:xmlData/> </mets:mdWrap> </mets:dmdSec> <mets:dmdSec ID="DMD-747421f1-fc0d-4c1d-896c-9087d43b5e10"> <mets:mdWrap MDTYPE="OTHER"> <mets:xmlData> <scape:versionMD version-number="1"/> </mets:xmlData> </mets:mdWrap> </mets:dmdSec> <mets:amdSec> <mets:techMD ID="TMD-scape-representation:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af-SCAPE_REPRESENTATION_TECHNICAL"> <mets:mdWrap MDTYPE="OTHER"> <mets:xmlData/> </mets:mdWrap> </mets:techMD> <mets:techMD ID="TMD-scape-representation:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af-scape-file:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af-JPYLYZER"> <mets:mdWrap MDTYPE="OTHER"> </mets:xmlData> </mets:mdWrap> </mets:techMD> </mets:amdSec> <mets:fileSec> <mets:fileGrp> <mets:file ID="scape-file:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af" SEQ="0" ADMID="TMD-scape-representation:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af-scape-file:uuid:1c0194a3-c5af- 4b40-b140-5ac64cfa43af-JPYLYZER" MIMETYPE="image/jp2"> <mets:FLocat xlink:href="http://bitfinder.statsbiblioteket.dk/newspapers/B400022028241-RT1_400022028241-1_1795-06-01_adresseavisen1759-1795-06-01-0006.jp2" xlink:title="B400022028241-RT1_400022028241-1_1795-06-01_adresseavisen1759-1795-06-01-0006.jp2" LOCTYPE="URL"/> </mets:file> </mets:fileGrp> </mets:fileSec> <mets:structMap> <mets:div TYPE="Intellectual entity"> <mets:div ID="scape-representation:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af" ADMID="TMD-scape-representation:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af- SCAPE_REPRESENTATION_TECHNICAL" TYPE="Representation" xlink:label="page-image-adresseavisen1759-1795-06-01-0007A"> <mets:fptr FILEID="scape-file:uuid:1c0194a3-c5af-4b40-b140-5ac64cfa43af"/> </mets:div> </mets:div> </mets:structMap> </mets:mets> 13 SCAPE Repository API This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 14. ● Get Entity – GET /entity/<entityID> ● Update Entity – PUT /entity/<entityID> ● Create Entity – POST /entity/<entityID> ● And many more 14 SCAPE Repository API This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 15. ● Checkout java -jar scape-stager-loader.jar --id_file=identifierFile.txt --checkoutSequenceFile="test.seqfile" checkout ● Commit java -jar scape-stager-loader.jar --commitSequenceFile="test.seqfile" commit 15 SCAPE Stager/Loader This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 16. ● The Hadoop job is started with the sequence file as input ● For each record in the sequence file – Read the record – Do work – Update the record in the sequence file with the result of the work 16 Step 2: Hadoop reads and updates records This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
  • 17. ● The hadoop job produces a sequence file ● For each record in the sequence file: – Read the record into memory – Any changed fields are updated in the corresponding DOMS objects This way, the actual work on the records is not connected to DOMS, and we can scale the cluster independently from the repository 17 Step 3: Store the updated records in DOMS This work was partially supported by the SCAPE Project. The SCAPE project is co funded by the European Union under FP7 ICT 2009.4.1 (Grant Agreement number 270137).‐ ‐
http://www.slideshare.net/SCAPEproject/scape-information-day-sb-integrating-doms-with-hadoop
www.slideshare.net
crawl=CC-MAIN-2014-52
222168
application/http; msgtype=response
sha1:2VTLCPPL4NBLQ4A554OMGK2BNAULEDS4
<urn:uuid:f1861cf3-f5f3-4f86-b438-5e98945e3b28>
2014-12-19T16:33:26Z
108.174.2.100
sha1:5AJA4KI2D2TCEF2MPCY6DJW3FR6CG6EZ
<urn:uuid:4902f524-dd37-44a8-bfd9-b36d2508af2d>
http://www.slideshare.net/SCAPEproject/scape-information-day-sb-integrating-doms-with-hadoop
length
response
<urn:uuid:0e083cb4-a7da-49eb-b6e0-70e6621c4a19>
robots: classic
hostname: ip-10-231-17-201.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2014-52
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for December 2014
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf