Datasets

If you are interested in mirroring these datasets for archival or LLM training purposes, please contact us.

Our mission is to archive all the books in the world (as well as papers, magazines, etc), and make them widely accessible. We believe that all books should be mirrored far and wide, to ensure redundancy and resiliency. This is why we’re pooling together files from a variety of sources. Some sources are completely open and can be mirrored in bulk (such as Sci-Hub). Others are closed and protective, so we try to scrape them in order to “liberate” their books. Yet others fall somewhere in between.

Updates

Overview

Below is a quick overview of the sources of the files on Anna’s Archive.

Source Size Mirrored by AA / torrents available
Percentages of number of files
Last updated
Libgen.rs
Non-Fiction and Fiction
7,217,242 files
79.9 TB
44.324% / 99.81%
AA is catching up with mirroring by leeching torrents.
2024-04-19
Sci-Hub
Via Libgen.li “scimag”
97,846,840 files
87.1 TB
0.006% / 88.481%
Sci-Hub: frozen since 2021; most available through torrents
Libgen.li: minor additions since then
Libgen.li
Excluding “scimag”
17,472,558 files
258.7 TB
68.681% / 85.694%
Fiction torrents are behind (though IDs ~4-6M not torrented since they overlap with our Zlib torrents). Magazine torrents are being remade.
2024-01-26
Z-Library 16,668,714 files
125.2 TB
76.641% / 99.82%
Torrents partly under embargo (still counted).
2024-01-24
Internet Archive Controlled Digital Lending 8,874,212 files
268.3 TB
65.022% / 65.022%
98%+ of files are searchable. Torrents partly under embargo (still counted).
2024-01-26
DuXiu 读秀 5,069,579 files
270.9 TB
100% / 0.001%
No torrents released yet; will be under embargo for LLM access.
~2023
Total
Excluding duplicates
134,660,826 files
922.2 TB
20.915% / 83.646%

Since the shadow libraries often sync data from each other, there is considerable overlap between the libraries. That’s why the numbers don’t add up to the total.

The “mirrored and seeded by Anna’s Archive” percentage shows how many files we mirror ourselves. We seed those files in bulk through torrents, and make them available for direct download through partner websites.

Source libraries

Some source libraries promote the bulk sharing of their data through torrents, while others do not readily share their collection. In the latter case, Anna’s Archive tries to scrape their collections, and make them available (see our Torrents page). There are also in-between situations, for example, where source libraries are willing to share, but don’t have the resources to do so. In those cases, we also try to help out.

Below is an overview of how we interface with the different source libraries.

Source Metadata Files
Libgen.rs
✅ Automated torrents for Non-Fiction and Fiction
👩‍💻 Anna’s Archive manages a collection of book cover torrents.
Sci-Hub / Libgen “scimag”
❌ Sci-Hub has frozen new files since 2021.
✅ Metadata dumps available here and here, as well as as part of the Libgen.li database (which we use).
✅ Data torrents available here, here, and here.
❌ Some new files are being added to Libgen’s “scimag”, but not enough to warrant new torrents.
Libgen.li
✅ Quarterly HTTP database dumps.
✅ Non-Fiction torrents are shared with Libgen.rs (and mirrored here).
🙃 Fiction collection has diverged but still has torrents, though not updated since 2022 (we do have direct downloads).
👩‍💻 Anna’s Archive manages a collection of comic books and magazines.
❌ No torrents for Russian fiction and standard documents collections.
Z-Library
❌ No metadata available in bulk from Z-Library.
👩‍💻 Anna’s Archive manages a collection of Z-Library metadata.
❌ No files available in bulk from Z-Library.
👩‍💻 Anna’s Archive manages a collection of Z-Library files.
Internet Archive Controlled Digital Lending
✅ Some metadata available through Open Library database dumps, but those don’t cover the entire Internet Archive collection.
❌ No easily accessible metadata dumps available for their entire collection.
👩‍💻 Anna’s Archive manages a collection of Internet Archive metadata.
❌ Files only available for borrowing on a limited basis, with various access restrictions.
👩‍💻 Anna’s Archive manages a collection of Internet Archive files.
DuXiu 读秀
✅ Various metadata databases scattered around the Chinese internet; though often paid databases.
❌ No easily accessible metadata dumps available for their entire collection.
👩‍💻 Anna’s Archive manages a collection of DuXiu metadata.
✅ Various file databases scattered around the Chinese internet; though often paid databases.
❌ Most files only accessible using premium BaiduYun accounts; slow downloading speeds.
👩‍💻 Anna’s Archive manages a collection of DuXiu files.

Metadata-only sources

We also enrich our collection with metadata-only sources, which we can match to files, e.g. using ISBN numbers or other fields. Below is an overview of those. Again, some of these sources are completely open, while for others we have to scrape them.

Note that in metadata search, we show the original records. We don’t do any merging of records.

Source Metadata Last updated
Open Library
✅ Monthly database dumps.
2024-03-26
ISBNdb
❌ Not available directly in bulk, only in semi-bulk behind a paywall.
👩‍💻 Anna’s Archive manages a collection of ISBNdb metadata.
2022-09-01
OCLC (WorldCat)
❌ Not available directly in bulk, protected against scraping.
👩‍💻 Anna’s Archive manages a collection of OCLC (WorldCat) metadata.
2023-10-01

Unified database

We combine all the above sources into one unified database that we use to serve this website. This unified database is not available directly, but since Anna’s Archive is fully open source, it can be fairly easily reconstructed. The scripts on that page will automatically download all the requisite metadata from the sources mentioned above.

If you’d like to explore our data before running those scripts locally, you can look at our JSON files, which link further to other JSON files. This file is a good starting point.
https://annas-archive.gs/datasets
annas-archive.gs
CC-MAIN-2024-18
57758
application/http; msgtype=response
sha1:I7H5ZURZKWGA3CXV25LUYNH4DSQJV7WI
<urn:uuid:1e274c6a-76a3-4d82-a09b-4d5104cc3a38>
2024-04-24T13:51:15Z
172.67.169.27
text/html
sha1:7NX5VMX4LTEKQEQH7PXY6HXSLEFOYJ76
<urn:uuid:96639a0d-4fbd-43e3-b161-89728c67c5ad>
https://annas-archive.gs/datasets
response
<urn:uuid:34bf48c9-55cb-4b4f-85b0-05b65d2c62b5>
isPartOf: CC-MAIN-2024-18
publisher: Common Crawl
description: Wide crawl of the web for April 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-23
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/