 

Are Repositories Impeding Big Data Reuse?

TR Number
Date
2016-06-14
Journal Title
Journal ISSN
Volume Title
Publisher
Virginia Tech
Abstract

In this intentionally provocative presentation, we question the scalability of popular digital repositories and whether they are suitable for big data reuse. Are the layers of API these repositories have painted over file system primitives necessary? How essential is it for the repository to insist on being the sole manager of the content, and arranging files in ways to prevent access other than from their own APIs? We explore these questions from the perspective of big data reuse, and describe controlled reuse experiments against Fedora 4 to evaluate the cost of these practices.

Description
Keywords
Institutional repository, Data management, Big data, Scalability, Throughput
Citation
https://vtechworks.lib.vt.edu/items/f3a6c965-53b7-460e-a679-9b8e200109e5
vtechworks.lib.vt.edu
CC-MAIN-2024-18
494798
application/http; msgtype=response
sha1:RRXRRZAZKIBRYAG2MQDORFJY762R6QS7
<urn:uuid:fc9598b8-97b2-413e-ba0e-809ae463b172>
2024-04-15T20:25:47Z
198.82.230.96
text/html
sha1:ZJKNRQYBX2W5MB4RFQYHJL6RSDHDTXIS
<urn:uuid:4fef359f-1308-4e6f-aae9-7d5b75b6c757>
https://vtechworks.lib.vt.edu/items/f3a6c965-53b7-460e-a679-9b8e200109e5
response
<urn:uuid:3a2cb018-83de-4a0f-b5f4-bde4f25c3d34>
isPartOf: CC-MAIN-2024-18
publisher: Common Crawl
description: Wide crawl of the web for April 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-51
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/