Progress, progress, progress

We are definitely getting there.

If you are interested you can take a gander at this presentation that I put together for the Steering Committee.

The real value in this project lies in the upload (and download) tool that I’ve written about previously.

It actually does a lot:

  • takes a collection of related files and assembles them into a bag (using the bagit specification from the library of congress)
  • it can function as command line tool in a script
  • it can function as a java service for automated machine to machine upload
  • at the backend it checks the files against Pronom using KNB Fido
  • and does some antivirus checking
  • and deposits the files with the metadata provided

and it gets round various upload limits ….

About dgm

Former IT professional, previously a digital archiving and repository person, ex research psychologist, blogger, twitterer, and amateur classical medieval and nineteenth century historian ...
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s
https://nennius.wordpress.com/2012/09/13/progress-progress-progress/
nennius.wordpress.com
crawl=CC-MAIN-2022-33
91848
application/http; msgtype=response
sha1:YH4F34B5X47XHG53AQOOARGM5LIGD246
<urn:uuid:43db54c0-fddc-44a3-b456-a30bac3c649b>
2022-08-19T15:02:06Z
192.0.78.13
text/html
sha1:CMCVCK5PKOYUHZKHL57R35C333B57E3B
<urn:uuid:ae37ceee-4407-4223-a5aa-15bbc13bf280>
https://nennius.wordpress.com/2012/09/13/progress-progress-progress/
response
<urn:uuid:39d00035-2cb7-435b-92f2-4ae6c4e3e8f9>
isPartOf: CC-MAIN-2022-33
publisher: Common Crawl
description: Wide crawl of the web for August 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-209
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.4-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/