Date: Mon, 16 May 2022 13:57:54 -0500 (CDT) Message-ID: <1686262315.7029.1652727474624@06f670804a60> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_7028_768682436.1652727474623" ------=_Part_7028_768682436.1652727474623 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html Extracted Features [v.2.0]

Extracted Features [v.2.0]

=20
=20
=20
=20

Read about HTRC's most recent version of the Extracted Features derived = dataset.

Page-level features from 17.1 million volumes

=20
=20
=20
=20
=20
=20

Short URL for this page: https://go.illinois.edu/EF20_documentation

=20
=20
=20
=20
=20
=20

HTRC Extracted Features 2.0 is the most current version of a deriv= ed dataset consisting of metadata and data elements extracted from volumes = in the HathiTrust Digital Library. The dataset is composed of 17+ million J= SON files representing a snapshot of the HathiTrust corpus from February 20= 20. 

This documentation describes the structure and data in the HTRC Ex= tracted Features 2.0 files for users of those files. The specific features = extracted are described in more detail below.

You can also refer to technical documentation of the Extracted Fea= tures 3.0 JSON-LD schema here<= /a>. The schema was developed collaboratively with JSTOR-Portico, and= it could be applied to data from non-HathiTrust sources to create compatib= le datasets. This version of the extracted features vocabulary is designed = as a linked data st= andard (JSON-LD). 

Downloading t= he files

See directions for Extracted Features v.2.0

Attribution

Jacob Jett, Boris Capitanu, Deren = Kudeki, Timothy Cole, Yuerong Hu, Peter Organisciak, Ted Underwood, Eleanor= Dickson Koehl, Ryan Dubnicek, J. Stephen Downie (2020). = The HathiTrust Research Center Extra= cted Features Dataset (2.0). Hat= hiTrust Research Center. https://doi.org/10.13012/R2TE-C227

This feature dataset is released u= nder a C= reative Commons Attribution 4.0 International License

=20
=20
=20
=20

= Data Stats

# of volumes represented

17,123,746=

# of pages represented
6,221,631,336=
# of tokens represented
2,906,819,723= ,689
# files derived from in-copyright volumes<= /span> 10,550,952
# pages derived from in-copyright volumes<= br> 3,743,561,467=
# tokens derived from in-copyright volumes=
1,709,172,692= ,891
# files derived from public domain & C= reative Commons volumes
6,572,794
# pages derived from public domain & C= reative Commons volumes
2,478,069,869=
# tokens derived from public= domain & Creative Commons volumes
1,197,647,030,798
=20
=20
=20
=20
=20
=20

Files and file= format

1 Extracted Features file per volume

  • There is one Extracted Features 2.0 file per volume from HathiTru= st that has been processed into the dataset. (17.1 million volumes processe= d =3D 17.1 million Extracted Features files in the entire dataset.)<= /li>
  • The dataset is not continuously updated, and files are not create= d on request. There are volumes in HathiTrust for which an associated Extra= cted Features file does not exist.

Size of the entire dataset

  • Each Extracted Features file is BZIP2 compressed.
  • The entire compressed dataset is 4TB.
  • For that reason, we advise against downloading the entire dataset= unless you are prepared to store and work with that amount of data. <= /span>

File sections

The main sections are the metadata block and the features block. For a basic overview, follow this intr= oductory guide to the HTRC Extracted Features 2.0 format.

JSON-LD and the linked data format

Extracted Features 2.0 files are in JSON-LD format. The linked dat= a integration is new to this version of HTRC Extracted Features, and incorp= orates semantics from the Schema.org web ontolog= y with extensions of our devising. 

Linked data is found primarily in the metadata section of the file= , and it connects to ontologies and vocabularies like Schema.org, or to authorities databases such as Virtual Intern= ational Authority File (VIAF) or Library of Con= gress linked data resourcesWhen no entry was found in a 3rd-party database, a non-authoritative placeholder entit= y was created for the name in a named entity database maintained by HTRC.

Terminology

These terms found in the documentation may be unfamiliar. 

Volume: A digitized item in the HathiTrust Digita= l Library, may be a book, an issue of a periodical, or several items bound = together. Represents a physical item that would be pulled off a library she= lf and digitized as one object.

Object: A key-value pair in the JSON file. Denote= d by curly braces. E.g. {type:=E2=80=9DBook=E2=80=9D}

Node: A linked data concept, they are the individ= ual elements that make up a JSON-LD document=E2=80=99s graph (the subject-object-predicate= of a linked data triple). Read more.= They can be:

  • A number or string
  • Arrays (a list)
  • Node objects (0 to more node properties)
  • Arrays that list node objects

Term: A label for a particular node

=20
=20
=20
=20
=20
=20

Feature = File Documentation

=20
=20
=20
=20
=20
=20

Repeating terms

Some terms in the Extracted Features files repeat, and their value= s depend on their context within the file. The repeating terms are describe= d below.


id

Description: Used to uniquely identify the nodes = composed by the Extracted Features file. Each id is a URI that allows a nod= e to be externally referenced. 

type

Description: Part of linked data standards. Wherever possible, we have used types = that have already been defined in the Schema.org vocabulary. In places where this was not possible, we have either de= fined new types or extended Schema.org types through development of a sub-c= lass. 

name

Description: Used wherever Perso= n and Organization= object types are found in the metadata block. It identifies a Person or Or= ganization in a human readable manner. 

=20
=20
=20
=20
=20
=20

File descr= iption section

Each Extracted Features file opens with metadata describing the fi= le itself.


@context

Description: Links to a con= text document written by HTRC, which maps terms in the Extracted Featur= es 2.0 file. The context is written by HTRC.

Value: The URL identifying the context document. Always= =E2=80=9Chttps://worksets.htrc.illi= nois.edu/context/ef_context.jsonld=E2=80=9D 

schemaVersion

Description: The version of the extracted features sche= ma.

Value: =E2=80=9C3.0=E2=80=9D

id

Description: See id

Value: The URL identifying the Extracted Features file.= E.g., "h= ttps://data.analytics.hathitrust.org/extracted-features/20200210/ufl2.uf001= 03078_00001"

htid

Description: Every volume in HathiTrust is assigned a u= nique HathiTrust identifier (htid). 

Value: The HathiTrust volume identifier. E.g. =E2=80=9C= ufl2.uf00103078_0000=E2=80=9D

type

Description: See type

Value: =E2=80=9CDataFeed=E2=80=9D [Learn more= ]

publisher

Description: Describes HTRC, the publisher of the= Extracted Features files. Consists of three parts: id, type, and name.

id

Description: See <= /span>id

Value: =E2=80=9Chttps://analytics.hathitrust.org=E2= =80=9D

Type

Description: See <= /span>type

Value: =E2=80=9COr= ganization=E2=80=9D

Name

Description: See <= /span>name

Value: =E2=80=9CHathiTrust Research Center=E2=80=9D

datePublished

Description: The date the Extracted Features file= was created, in YYYYMMDD format [<= span>ISO 8601]

Value: E.g. =E2=80=9C20200221=E2=80=9D

=20
=20
=20
=20
=20
=20


Metadata section

Metadata primarily about the volume described by the Extracted Fea= tures file. 

The metadata is received from HathiTrust in M= ARC (library catalog metadata) format. (Learn more about HathiTrust metadata.) It is then t= ransformed to the Bibframe standard, = which is a linked data standard. By shifting the metadata from MARC to Bibf= rame, we are able to improve the overall quality of the metadata by providi= ng links to more contemporary authorities sources for metadata (such as = VIAF, etc.). As a final step, the metadata is conver= ted into the Extracted Features metadata section format.

If there would be no value for a field, then it is left out of the= file. Therefore, you may not find all of the fields below in a given Extra= cted Features 2.0 file.


metadata

Description: Denotes the start of the metadata pa= rt of the Extracted Features file. The metadata describes the volume from H= athiTrust represented by the Extracted Features file. 

schemaVersion

Description: The schema version for the metadata = block of the Extracted Features file.

Value: A URL linking to the schema. "https://schemas.hathitrust.org/= Extracted Features_Schema_MetadataSubSchema_v_3.0"<= /p>

Type: Single value

id

Description: See id

Value: The Handle URL for the volume, which will = point to the volume in the HathiTrust Digital Library. E.g. "http://hdl.handle.net/2027/mdp.39015062779023= "

Type: Single Value

type

Description: See type

Value: Either =E2=80=9CBook=E2=80=9D, =E2=80=9CPu= blicationVolume=E2=80=9D, or =E2=80=9CCreativeWork=E2=80=9D depending on th= e value found in the Issuance<= /span> field of the corresponding Bibframe record. When the Bibfr= ame Issuance value is =E2=80=98mono=E2=80=99, the Book type is assigned. When the value is =E2=80=98serl=E2=80=99, the= PublicationVolume type is ass= igned. In all other cases the CreativeWork type is assigned. The Bibframe Issuance is derived from a variety o= f fields in the MARC record, including <= span>Control Fields and Physical Description Fields. Additionally, "DataFie= ldElement" will always be listed as a type to reflect the item's relationsh= ip with the Extracted Features file.

Type: List of values

dateCreated

Description: The date on which the metadata porti= on of the Extracted Features file is generated, in YYYYMMDD format [= ISO 8601]

Value: E.g., =E2=80=9C20200209=E2=80=9D

Type: Single value

title

Description: The title of the volume when the type (above) is =E2=80=9CBoo= k=E2=80=9D or =E2=80=9CCreativeWork= =E2=80=9D.

Value: Derived from the Bibframe title, which originates from th= e Title & Title-Related Fie= lds in the MARC record.

Type: Single value

alternateTitle

Description: An alternate title for a bibliograph= ic entity described by the Extracted Features file.

Value: Derived from the Bibframe title where the rdf:type is =E2=80=9C<= a class=3D"external-link" href=3D"http://id.loc.gov/ontologies/bibframe.htm= l#c_VariantTitle" rel=3D"nofollow">VariantTitle=E2= =80=9D. Originates from a variety of fields in the MARC record, including <= /span>Main Entry Fields = and Title & Title-Related F= ields. Additionally, if there are multiple valid values fo= r title, one value will be used and all additional va= lues will be listed here.

Type: Single value or list of values

enumeration= Chronology

Description: Information regarding which volume, = issue, and/or year the HathiTrust volume was published.

Value: Taken from the Bibframe enumerationAndChronology<= /a> node, which is derived from a variety of fields in the MARC recor= d, including Holdings Fields and Locally Added Fields<= /span>.

publisher

Description: Information about the publisher of t= he volume described by the Extracted Features file. Includes type, name, and id. [Learn more<= /a>

<= span>type

Description: See type

Value: Either =E2= =80=9COrganization=E2=80=9D or =E2= =80=9CPerson=E2=80=9D. Taken from the typ= e node for the Bibframe provisionActivity=E2=80=99s agent

id

Description: See id=

Value: A URL ident= ifying the publisher, such as a Handle URL, ORCID, etc.

<= span>Name

Description: See name

Value: Taken from = the label node for the Bibframe provisionActivity=E2=80=99s agent. Derived from the Imprint field in the MARC record.<= /p>

pubPlace

Description: Information about where the volume w= as first published. Includes id and type.[Learn more

id

Description: See <= /span>id

Value: Taken from = the Bibframe Instance=E2=80=99= s provisionActivi= ty=E2=80=99s place= rdf:about node, which are derived from the country codes in the MARC 008 field.


type

Description: See <= /span>type

Value: =E2=80=9CPl= ace=E2=80=9D [Learn more]


name

Description: See <= /span>name

Value: Corresponds= to information in the Imprint fie= ld in the MARC record.

pubDate

Description: The year in which that edition of th= e volume was first published.[Learn mo= re]

Value: Taken from the Bibframe provisionActivity date, which is derived from the = Imprint field in = the MARC record.

genre

Description: Information about the volume=E2=80= =99s genre, as determined by the cataloger of the work. [Learn more]

Value: Taken from the Bibframe Work=E2=80=99s genreForm node and corresponds t= o a controlled vocabulary, e.g., the MARC G= enre Terms List. Values are derived from the Genre/Form Field in the MARC r= ecord.

category

Description: The volume=E2=80=99s topic or topics= . [Learn more

Value: Derived from the Bibframe Work=E2=80=99s ClassificationLcc node. Represents= the natural language label for the Library of Congress Classification (LCC= ) value based upon the Lib= rary of Congress=E2=80=99s LCC standard documentation.

language

Description: The cataloger-determined language or= languages of the volume.

Value: Taken from the Bibframe Work=E2=80=99s language=E2=80=99s identifiedBy=E2=80=99s value node, which is der= ived from the Language Code field<= /span> in the MARC record. 

accessRights<= /h3>

Description: The copyright status of the volume.&= nbsp;

Value: Corresponds to attributes in HathiTrust=E2=80=99s accessRights datab= ase. Derived from a HathiTrust-local MARC field (974r) tha= t is added to bibliographic records as they are processed at HathiTrust.

lastRightsUp= dateDate

Description: The most recent date the volume's co= pyright status was updated.

Value: Derived from a HathiTrust-local MARC field= (974d) that is added to bibliographic records as they are processed at Hat= hiTrust.

Contributor

Description: Contains information regarding the a= uthor(s), editor(s), or other agents involved in creating the volume. Consi= sts of id, type<= /em>, and name. [Learn more

Type: Single object or list of objects

id

Description: See id

Value: A URL taken= from the Bibframe agent that links to an authorities database (e.g., VIAF<= /a>). 

Type: Single value<= /p>

<= span>type

Description: See <= /span>type

Value: Either the = Person or Organiza= tion. Taken from the Bibframe agent ty= pe

Type: Single value<= /p>

name

Description: The n= ame of the person or organization who created the volume. See also <= a href=3D"#ExtractedFeatures[v.2.0]-name">name

Value: Taken from = the Bibframe agent=E2=80=99s label. Derived from a variety of fields in the MARC record, in= cluding Main Entry Fields, Title and Title-Relate= d Fields, and = Added Entry Fields.

Type: Single value<= /p>

typeOfResource

Description: The cataloger-determined resource ty= pe of the volume (e.g., text, image, etc.).

Value: The data value of this node is taken from = the Bibframe Work=E2=80=99s= type node,<= /span> which is derived from the MARC record=E2=80=99s 008 Fixed-Length Data Elements Field<= span>.

sourceInstituti= on

Description: An array containing inf= ormation about the institution that contributed the volume to = HathiTrust. Always has a type node and a name node. [Learn more]

id

Description: See id

Value: A URL identifying the source institution

<= span>type

Description: See <= /span>type

Value: Always =E2=80=9COrganization=E2= =80=9D

<= span>name

Description: The name of the source institution. See also name

Value: Derived fro= m a HathiTrust-local MARC field (974c) that is added to bibliographic recor= ds as they are processed at HathiTrust.

mainEntityOfPage=

Description: An array of URLs linking to various = metadata records describing the volume represented by the Extracted Feature= s file. [Learn more]

Value: The array typically contains 3 URLs that p= oint to the HathiTrust Bibliographic API: HathiTrust brief bibliographic record, HathiTrust full bibliograp= hic record, and the HathiTrust catalog record.

oclc

Description: The OCLC number for the volume. An O= CLC number is an identifier assigned to items as they are cataloged in a li= brary. 

Value: Derived from the MARC record=E2=80=99s System Control Number field<= span> (035).

lcc

Description: The Library of Congress Classificati= on number for the volume. An LCC number is a type of call number that would= be used to locate an item on a library shelf. 

Value: Derived from the MARC record=E2=80=99s LC Classification Number field= (053). E.g. PG7158.M76 E23

lccn

Description: The Library of Congress Control Numb= er for the volume. An LCCN is a unique number that is assigned during catal= oging.

Value: Derived from the MARC record=E2=80=99s Library of Congress Control Number field<= /span> (010). 

issn

Description: The ISSN of the volume (when a journ= al). [Learn more]

Value: Taken from the Bibframe Instance=E2=80=99s identifiedBy=E2=80=99s Issn node, which is derived from the MARC= record=E2=80=99s ISSN field

isbn

Description: The ISBN of the volume (when a book)= . [Learn more]

Value: Taken from the Bibframe Instance=E2=80=99s identifiedBy=E2=80=99s Isbn node, which is derived from the MARC= record=E2=80=99s ISBN field.

=20
=20
=20
=20
=20
=20


Features section

This portion of the Extracted Features file contains all of the un= igram tokens (i.e. words), token counts, and other calculated or algorithmi= cally-derived data from the HathiTrust volume. 

The full text of the volume is tokenized using Stanford NLP, which includes par= t-of-speech detection. The tokens are tagged with part-of-speech tags using the tagsets from Stanf= ord NLP.

After an initial section containing volume- and EF file-level meta= data, this section is divided into JSON arrays for each page in the volume.= Nested within those page-level arrays are arrays for the header, body, and= footer of the page.

Fields that would be empty (i.e. if the algorithm does not detect = content on a page) are given NULL values.

features

Description: Denotes the start of the features pa= rt of the Extracted Features file. 

id

Description: See id

Value: The Handle URL identifying the HathiTrust = volume

type

Description: See type

Value: =E2=80=9CDataFeedItem=E2=80=9D

schemaVersion

Description: The version of the features part of = the Extracted Features document model.

Value: A URL linking to the schema. "https://schemas.hathitru=E2=80= =A6_FeaturesSubSchema_v_3.0"

dateCreated<= /h3>

Description: The date the data features were gene= rated in in YYYYMMDD format. [ISO 8601] Note that this will be different from the dateC= reated for both the metadata block and the overall Extracted Features file.=

Value: E.g. =E2=80=9C20200209=E2=80=9D

pageCount <= /h3>

Description: The number of page scans in the volu= me, which is equivalent to the number of scanned images making up the digit= al object in the HathiTrust Digital Library.

Value: A number corresponding to the number of pa= ges.

pages

Description: An array of individual page objects = (one per page of the volume).

seq 

Description: The s= equence number of the page in the volume. Corresponds to the digital object= , so that the first scan in the volume is =E2=80=9C00000001=E2=80=9D, which= may be the cover, a title page, or something else.

Value: E.g. =E2=80= =9C"00000001" or =E2=80=9C00000055=E2=80=9D

= version 

Description: A has= h of the page content used to compute the features for the page. Volumes in= HathiTrust may be updated to improve scan or OCR quality or correct an iss= ue, which would cause the text data to change, and, if features are reproce= ssed, a new hash would result.

Value: The alphanumeric hash. 

calculatedLanguage

Description: The m= ost probable language of the text on the page. Det= ermined algorithmically, and specified by language codes.

Value: A code for = the language. Will be null if no language detected, or if the language was = not recognized by the algorithm.

tokenCount

Description: The t= otal number of tokens detected on the page.

Value: A number re= presenting the number of tokens.

lineCount

Description: The total number of lines of text detected on the page.<= /p>

Value: = A number representing the lines. 

emptyLineCount

Description: The total number of empty lines on the page.

Value: = A number representing the empty lines. 

sentenceCount

Description: The total number of sentences detected on the page.

Value: A number representing the sentences.

<= span style=3D"color: rgb(51,51,51);">header

Description: The h= eader portion of the page.

tokenCount

Description: The t= otal number of tokens detected in the header of the page.

Value: A number re= presenting the tokens.

lineCount

Description: The t= otal number of lines detected in the header of the page.

Value: A number re= presenting the lines.

emptyLineCount

Description: The t= otal number of empty lines detected in the header of the page.

Value: A number re= presenting the empty lines.

sentenceCount

Description: The t= otal number of sentences detected in the header of the page.

Value: A number re= presenting the sentence count.

beginCharCount

Description: An ar= ray of the first non-White Space characters detected on lines in the header= and their occurrance counts.

Value: The charact= er and the number of times it occurs. E.g. =E2=80=9CR:1=E2=80=9D

endCharCount

Description: An ar= ray of the last non-White Space characters on detected lines in the header = and their occurrance counts.

Value: The charact= er (letter, number, or punctuation) and the number of times it occurs. E.g.= =E2=80=9Cg:5=E2=80=9D

tokenPosCount

Description: An ar= ray of individual tokens with their corresponding part of speech and occurr= ence counts in the header of the page. If the language is one that is recog= nized by the Stanford NLP parser= ("ar", "zh", "en", "fr", "de", "es")= StanfordNLP models for the language are used to tokenize the text and do p= art-of-speech tagging. Otherwise, a whitespace tokenizer is used.

Value: The token, = its part of speech (represented by the Penn Tree Bank= for English), and a number representing the number of times that token app= eared in the header.

body

Description: The b= ody portion of the page.

tokenCount

Description: The t= otal number of tokens detected in the body of the page.

Value: A number re= presenting the tokens.

lineCount

Description: The t= otal number of lines detected in the body of the page.

Value: A number re= presenting the lines.

emptyLineCount

Description: The t= otal number of empty lines detected in the body of the page.

Value: A number re= presenting the empty lines.

sentenceCount

Description: The t= otal number of sentences detected in the body of the page.

Value: A number re= presenting the sentence count.

beginCharCount

Description: An ar= ray of the first non-White Space characters detected on lines in the body a= nd their occurrence counts.

Value: The charact= er and the number of times it occurs. E.g. =E2=80=9CR:1=E2=80=9D

endCharCount

Description: An ar= ray of the last non-White Space characters detected on lines in the body an= d their occurrence counts.

Value: The charact= er (letter, number, or punctuation) and the number of times it occurs. E.g.= =E2=80=9Cg:5=E2=80=9D

tokenPosCount

Description: An ar= ray of individual tokens with their corresponding part of speech and occurr= ence counts in the body of the page. If the language is one that is recogni= zed by the Stanford NLP parser (= "ar", "zh", "en", "fr", "de", "es") S= tanfordNLP models for the language are used to tokenize the text and do par= t-of-speech tagging. Otherwise, a whitespace tokenizer is used.
=

Value: The token, = its part of speech (represented by the Penn Tree Bank for English), and a number representin= g the number of times that token appeared in the header.

capAlphaSeq

Description: The l= ongest length of the alphabetical sequence of capital characters starting a= line.

Value: A number re= presenting the length. 

<= span style=3D"color: rgb(51,51,51);">footer

Description: The f= ooter portion of the page.

tokenCount

Description: The t= otal number of tokens detected in the footer of the page.

Value: A number re= presenting the tokens.

lineCount

Description: The t= otal number of lines detected in the footer of the page.

Value: A number re= presenting the lines.

emptyLineCount

Description: The t= otal number of empty lines detected in the footer of the page.

Value: A number re= presenting the empty lines.

sentenceCount

Description: The t= otal number of sentences detected in the footer of the page.

Value: A number re= presenting the sentence count.

beginCharCount

Description: An ar= ray of the first non-White Space characters detected on lines in the footer= and their occurrence counts.

Value: The charact= er and the number of times it occurs. E.g. =E2=80=9CR:1=E2=80=9D

endCharCount

Description: An ar= ray of the last non-White Space characters on detected lines in the footer = and their occurrence counts.

Value: The charact= er (letter, number, or punctuation) and the number of times it occurs. E.g.= =E2=80=9Cg:5=E2=80=9D

tokenPosCount

Description: An ar= ray of individual tokens with their corresponding part of speech and occurr= ence counts in the footer of the page. If the language is one that is recog= nized by the Stanford NLP parser= ("ar", "zh", "en", "fr", "de", "es")= StanfordNLP models for the language are used to tokenize the text and do p= art-of-speech tagging. Otherwise, a whitespace tokenizer is used.

Value: The to= ken, its part of speech (represented by the Penn Tree Bank for English), and a number repres= enting the number of times that token appeared in the header.

=20
=20
=20
------=_Part_7028_768682436.1652727474623--
https://wiki.htrc.illinois.edu/exportword?pageId=79069329
wiki.htrc.illinois.edu
crawl=CC-MAIN-2022-21
73906
application/http; msgtype=response
sha1:Q4F4LSBEOZPZZL7G3MGD57UU65HKRWYW
<urn:uuid:18aec829-d208-4f2f-a6ce-f089960217b0>
2022-05-16T18:57:54Z
128.174.94.214
message/rfc822
sha1:ZD5FESJYYUOLV7RIJEPCIRXT6FZDT266
<urn:uuid:f955db56-8c93-41b1-a94f-1cd9ae76fb63>
https://wiki.htrc.illinois.edu/exportword?pageId=79069329
response
<urn:uuid:831d6e05-5384-4ca0-b076-3862b7df52db>
isPartOf: CC-MAIN-2022-21
publisher: Common Crawl
description: Wide crawl of the web for May 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-28
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/