What Research Libraries And Web Archives Could Learn From The Commercial Cloud

This article is more than 2 years old.

In 2014 I optimistically wrote for the Knight Foundation blog that libraries could reinvent themselves in the digital era, tracing my own collaborations with the Internet Archive over the prior year and drawing from my opening keynote address to the 2012 IIPC General Assembly at the Library of Congress. Yet, reflecting back three years later, looking at just how adrift and leaderless so many research libraries have become in the digital era, unsure of how to reinvent themselves and often too arrogant and insular to reach out beyond the communities they have worked with for centuries, I am no longer so certain that research libraries and the academic communities that work most closely with them can genuinely reimagine themselves on their own. Community libraries have found great success reinventing themselves to better fit into modern lifestyles, from collaborative spaces to free wifi to ebooks to and even 3D printers and virtual reality systems, but research libraries as a whole seem to be struggling to find their footing in the digital era. What might they learn from the world of the commercial cloud and indeed the broader technological future of Silicon Valley?

The commercial cloud has truly transformed how we think about computing in the modern era, from the shift from hardware to services and experts, the rise of seamless security and unimaginable deep learning systems accessible by a single API call. Moreover, the scale that companies like Google operate at dwarfs even the largest collections in the library world. When I talk with many in the library community, they point to numbers like the Internet Archive’s 20 petabytes of holdings as evidence that the library world has some of the largest datasets in existence. Yet, in Google’s world, a couple of engineers could borrow a 50 petabyte cluster to run a quick sorting experiment half a decade ago and today anyone can table scan an entire petabyte in just 3.7 minutes. With one line of SQL you could analyze the entire holdings of the Internet Archive in just under an hour and 14 minutes without having to write a line of programming code or manage a single file.

This immense technical capacity is increasingly being delivered as user-friendly API building blocks that can be rapidly plugged together to build global-scale applications of incredible capability and complexity with just a few lines of code. In my own open data GDELT Project, I’ve long looked for a way that my web crawlers could live stream the unusual files they encounter each day (such as images that crash a major image library or which work with one library but fail in another) back to a central analysis server for automated diagnosis. I didn’t want to add a lot of complexity to my crawlers or stand up and manage a large cluster that could absorb a sustained stream of gigabytes per second of data. Building any kind of live stream absorption system requires a lot of architectural design to ensure it is robust and to deal with issues like automatic scaling and capacity management.

That’s when I came across Google’s Cloud Storage Streaming Transfers service, which allows one to live stream effectively infinitely into Google’s storage fabric and access the results as standard files. With the addition of just three lines of code to my crawlers (one to open the stream, one to write to the stream and one to close the stream) suddenly I was live streaming all of the data I needed with all of the scaling and capacity planning being handled by Google itself, rather than being something I needed to manage. As the number of crawlers increased, the system simply scaled linearly and transparently.

In my case I’m using this capability as essentially a managed infinitely-sized working buffer, but since Streaming Transfers are handled the same as any other file on GCS, the resulting streamed files can be configured as Multi Regional files and made as robust and durable as Google is capable of, mirroring those files across multiple physical data centers throughout the world.

Why is this relevant to web archiving? In short, because those three lines of code effectively transformed those crawlers into a miniature Internet Archive. Instead of discarding all of the files my crawlers encounter, the addition of those three lines of code allow those crawlers to live stream files of interest back to Google’s global storage infrastructure. To make my own Internet Archive all I’d have to do is tell those crawlers to stream all files back instead of just some files and to configure my GCS storage to retain the files infinitely and copy across regions instead of using it as a scalable cloud scratch disk.

In fact, if all you wanted to do was crawl the open web and archive everything you encountered (literally making your own personal Internet Archive), you could pretty much build a web-scale crawling and archival infrastructure by just connecting Google’s existing cloud building blocks like Cloud Pub/Sub, Cloud Datastore and Stackdriver Monitoring, data mine the ingested content with Cloud Vision and Cloud Natural Language and store all of the results to Cloud Storage, all with the knowledge that your system can transparently scale to the tens or even hundreds of petabytes without having to change a line of code.

In short, with very little code, you can build a system capable of open web archiving just by connecting these prebuilt cloud building blocks together. Compare this with hiring a huge development team, building and managing your own massive hardware and software cluster infrastructure and trying to build tools that can scale to even a small fraction of the web. After all, it’s hard to beat a cloud environment that cost more than $30 billion to build.

Even for those web archives that have massive legacy data centers they want to continue to maintain for the time being, there are still so many lessons they can learn from the building block and services mentality of the commercial cloud, especially around the areas of automatic scaling, reliability, robustness, security and developer workflows.

Of course, not every research library conducts web archiving and the lessons here apply equally well to any kind of research workflows.

Want to OCR scanned books and other materials in 56 languages for as low as 15 to 30 cents a book with accuracy on languages like Arabic that approaches human quality in many cases? There’s no hardware to maintain, no massive OCR clusters to build and no worrying about scaling and capacity – just plow the images through a cloud API as fast as you generate them and the get back searchable OCR’d text. In short, the modern cloud has moved beyond renting hardware to providing a myriad prebuilt service building blocks that you can just plug together to instantly build any application imaginable and leave the management to the cloud vendor.

Of course, once libraries and archives acquire all of this content, what can they do with it?

As I pointed out in my 2015 NFAIS keynote address, it has been just over half a century since Dialog introduced us to keyword searching online text archives and 50 years later we still access the modern world through the keyword. It is astounding to think that with the computing revolution and information explosion over the past half century, from landing on the moon to the birth of the web, the mechanism through which we access all of this information has never moved beyond the humble keyword.

This is where the rise of cloud-based deep learning APIs are poised to have an immense impact on research libraries. You can catalog hundreds of millions of photographs by their contents, transform a multi-petabyte video archive into a fully searchable database, generate transcripts of speeches and even translate them into multiple languages or interrogate vast text archives in complex ways. All of this possible through simple remote cloud APIs and requiring no knowledge of deep learning.

With all of this content, how do libraries secure the vast archives they are assembling? Many of the research libraries I’ve interacted with still use primitive IP whitelist filters and VPNs for remote access. Few have implemented any kind of true behavioral monitoring of their systems. Most have no idea that a faculty member just paid a room full of undergraduates to bulk download tens of millions of documents from a licensed resource in strict violation of the university’s legal agreement and then redistributed that content to collaborators at multiple institutions, until they receive an irate phone call from the vendor notifying of forthcoming legal action. This is another area where the cloud can help, offering capabilities like sophisticated identity management, behavioral monitoring and transparent 2-factor authentication.

To date few research libraries or web archives have been priority targets for nation state attackers, largely because the content they hold tends not to be as economically or politically valuable compared with other targets. However, the dearth of major documented attacks is likely due to the relative obscurity of web archives today in which much of the general public are still unaware that there are sites that preserve past copies of web pages. If a major browser like Chrome were to add transparent integration with major web archives, bringing the notion of “undeleting” embarrassing content into the mainstream, it is likely that cyberattacks against those archives would ramp up accordingly. Today a repressive government can simply order removed any web content that poises an embarrassment or order that a news outlet make changes to an article to change its stance towards a topic or political figure. Imagine if as you browsed the web your browser alerted you that the page you’re currently reading looked much different a few hours ago and that several paragraphs about a corruption scandal involving the finance minister of your country recently vanished. Or if you received an alert that a number of other pages on the site on the same topic have disappeared in the past week, all while letting you view the deleted content. In such a world nation states are much more likely to view web archives as a significant target. Once again, this is where the commercial cloud can help with its vastly superior security posture and investments.

Putting this all together, today’s commercial cloud has much to teach the research library and web archiving communities, from how to build and secure robust global-scale infrastructure to how to make all of their content searchable to how to bring innovative technologies like deep learning to bear to move beyond the half-century-hold keyword search. In the end if research libraries and web archives want to maintain their relevance into the coming years they must reinvent themselves for the digital era and to do that they should take a closer look at the commercial cloud that has reinvented the business world.

Based in Washington, DC, I founded my first internet startup the year after the Mosaic web browser debuted, while still in eighth grade, and have spent the last 20 year

...
https://www.forbes.com/sites/kalevleetaru/2017/03/27/what-research-libraries-and-web-archives-could-learn-from-the-commercial-cloud/
www.forbes.com
crawl=CC-MAIN-2019-51
450253
application/http; msgtype=response
sha1:LCOGSAEP2IAYS7FJ4GGHZH6VD2AFETX6
<urn:uuid:04af2839-6141-4f35-97b4-58eb148f8b5a>
2019-12-14T04:47:51Z
151.101.250.49
text/html
sha1:OZQWG6LDULG56CBKQCBBJO2IW4UITOGC
<urn:uuid:f8ffc474-5a25-491e-b2f0-8c9bae0fd0e9>
https://www.forbes.com/sites/kalevleetaru/2017/03/27/what-research-libraries-and-web-archives-could-learn-from-the-commercial-cloud/
response
<urn:uuid:de1527f9-e665-4eb9-8245-311069be4254>
isPartOf: CC-MAIN-2019-51
publisher: Common Crawl
description: Wide crawl of the web for December 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-27.ec2.internal
software: Apache Nutch 1.16 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/