OpenLibrary

We also source book data from OpenLibrary, as downloaded from their developer dumps.

The DVC control files automatically download the appropriate version. The version can be updated by modifying the data/ol_dump_*.txt.gz.dvc files.

Imported data lives under the openlibrary directory.

erDiagram
    editions ||--o{ edition-isbn-ids : ""
    edition-isbn-ids }o--|| all-isbns : ""
    editions {
        Int32 id PK
        Utf8 key
        Utf8 title
    }
    editions }o--o{ works : "edition-works"
    editions |o--o{ edition-subjects : ""
    edition-subjects {
        Int32 id
        Utf8 subject
    }
    works {
        Int32 id PK
        Utf8 key
        Utf8 title
    }
    works |o--o{ work-subjects : ""
    work-subjects {
        Int32 id
        Utf8 subject
    }
    authors {
        Int32 id PK
        Utf8 key
        Utf8 name
    }
    authors ||--o{ author-names : ""
    editions }o--o{ authors : "edition-authors"
    works }o--o{ authors : "work-authors"

Import Steps

The import is controlled by the following DVC steps:

scan-*
The various scan-* stages (e.g. scan-authors) scan an OpenLibrary JSON file into the resulting Parquet files. There are dependencies, to resolve OpenLibrary keys to numeric identifiers for cross-referencing. These scan stages do not currently extract all available data from the OpenLibrary JSON; they only extract the fields we currently use, and need to be extended to extract and save additional fields.
edition-isbn-ids
Convert edition ISBNs into ISBN IDs, producing edition-isbn-ids.parquet.

Raw Data

The raw data lives in the data/openlib directory, as compressed JSON files. Right now we do not extract very many fields from OpenLibrary; additional fields can be extracted by extending the import scripts.

Extracted Edition Tables

We extract the following tables from OpenLibrary editions:

openlibrary/editions.parquet
This file contains a primary record for each edition, with the numeric edition ID, OpenLibrary key, and edition data.
openlibrary/edition-authors.parquet
This file contains mappings between editions and their authors.
openlibrary/edition-works.parquet
This maps editions to their works.
openlibrary/edition-isbns.parquet
This contains the ISBN fields extracted from each OpenLibrary edition. This is primarily for internal purposes and most people won’t need to use it. ISBNs are cleaned (with clean_isbn_chars or clean_asin_chars prior to being stored in this file.
openlibrary/edition-subjects.parquet
This table contains the subjects for OpenLibrary editions. Each row contains an edition ID and one subject. Its schema is in EditionSubjectRec.
openlibrary/edition-isbn-ids.parquet
This file maps editions to numeric ISBN identifiers. It is derived from openlibrary/edition-isbns.parquet.

Extracted Work Tables

We extract the following tables from OpenLibrary works:

openlibrary/works.parquet
This file contains the primary record for each work, mapping a numeric ID to its OpenLibrary key and containing other per-work fields.
openlibrary/work-authors.parquet
This file links work records to the work’s author list (works may have separate author lists from their editions).
openlibrary/work-subjects.parquet
This table contains the subjects for OpenLibrary editions. Each row contains an edition ID and one subject. Its schema is in WorkSubjectRec.

Extracted Author Tables

openlibrary/authors.parquet
This file contains basic information about OpenLibrary authors.
openlibrary/author-names.parquet
This file contains the names associated with each author in openlibrary/authors.parquet.

Utility Tables

openlibrary/work-clusters.parquet
This file is a helper table to make it easier to connect OpenLibrary data to clusters by mapping OpenLibrary work IDs to book data cluster IDs.
https://bookdata.piret.info/data/openlib.html
bookdata.piret.info
CC-MAIN-2023-50
34876
application/http; msgtype=response
sha1:CZ3LR7DG6W3XVW2UU5DYMUG7VJQXCSMD
<urn:uuid:3d44cc83-91f3-4e2c-9161-c6ccd0c04308>
2023-12-08T13:11:09Z
185.199.109.153
application/xhtml+xml
sha1:5TJGZR64D5YNQJFPH2WJMA3L5HCJWA3N
<urn:uuid:b8de86cf-b4af-4952-b218-f4c7fe33d2ce>
https://bookdata.piret.info/data/openlib.html
response
<urn:uuid:55c1345c-11dd-4993-9705-891592722e02>
isPartOf: CC-MAIN-2023-50
publisher: Common Crawl
description: Wide crawl of the web for November/December 2023
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-66
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/