Systems

WorldCat in your pocket

Lorcan 1 min read

We have been looking at things recently which have made my colleagues interested in hardware again, and which also show how far we have come in being able to manipulate and move large amounts of data.
WorldCat is our union catalogue of about 56 million bibliographic records, which represent approximately a billion holdings. It is about 50 gigabytes in MARC Communications (100+ gigabytes in XML) format and about 23 gigabytes compressed.
OCLC Research recently acquired a 24-node (48-cpu) Beowulf cluster with 96 Gigabytes of memory. According to my colleague Thom Hickey, whose team has been working on the machine, the cluster speeds up most bibliographic processing by about a factor of 30. This means that what might have taken a minute now takes two seconds, what might have taken an hour takes two minutes, what might have taken a month takes a day. For jobs that will fit entirely in memory (e.g. a `grep’ of WorldCat) avoiding disk i/o gives another factor of about 20, reducing 1-hour jobs down to 6 seconds. We can ‘frbrize‘ WorldCat on the cluster in about an hour.
WorldCat is also now more mobile. Thom has a 40 gig iPod which can accommodate WorldCat on its disk with room left for 5,000 song tracks. Now, you can’t do much with the data on the iPod, but you can certainly carry it around. Again, it takes about an hour to get it on and off the iPod.
Thom adds “much as ‘quantity has a quality all its own’, such a speed-up changes both the approaches that can be taken to solving problems and the type of problems that can be tackled at all. Google typically works with cluster of 3,600 CPUs, raising performance to yet another level.”

Share
More from LorcanDempsey.net
Two metadata directions
Systems

Two metadata directions

Metadata practice continues to evolve as research and cultural practices diversify. After a brief environmental view, I discuss two important metadata trends here: entification and pluralization.
Lorcan 16 min read
icon

Follow along

Deep dives and quick takes. Libraries, society, culture, technology, ...

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to LorcanDempsey.net.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.
https://www.lorcandempsey.net/worldcat-in-your-pocket/
www.lorcandempsey.net
crawl=CC-MAIN-2022-49
50643
application/http; msgtype=response
sha1:O43WHSOICEY6JQVSYF2YUHXXEBRJKB54
<urn:uuid:ef2ae769-2e38-4eb2-8cb5-e730e91f0db0>
2022-12-06T03:28:39Z
151.101.147.7
text/html
sha1:GGNKGIZ3UMCZWW4ASZHMKSPZ5WYAGCWE
<urn:uuid:6a6294f6-0c00-4c9d-825e-b05ee72887b9>
https://www.lorcandempsey.net/worldcat-in-your-pocket/
response
<urn:uuid:bc8360d6-b924-4034-943f-b64f70a6d515>
isPartOf: CC-MAIN-2022-49
publisher: Common Crawl
description: Wide crawl of the web for November/December 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-92
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.4-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/