Dataset originally created 11/6/2018 - UPDATE 11/19/19 (Corrected digest definition) - UPDATE 1/3/19 (Renamed image_width, image_height, & content_length to page_width, page_height, & file_size) - UPDATE 12/27/18 (Added SHA256 and SHA512 checksum fields) - UPDATE 12/3/18 (Removed offset and filename fields) - UPDATE 3/26/19 (Added MediaInfo metadata fields) I. About This Dataset This dataset is based on exploratory work begun by the Library of Congress's Web Archiving Team in 2018. The goal of the work is to explore the contents of the Library's web archives through analysis of the indexes containing metadata from the harvested web content, as stored in CDX files (https://web.archive.org/web/20171123000432/https://iipc.github.io/warc-specifications/specifications/cdx-format/cdx-2006/). The metadata contained in the indexes was used for initial analysis, rather than the archived content stored in WARC (https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml) and ARC (https://www.loc.gov/preservation/digital/formats/fdd/fdd000235.shtml) container files, since W/ARC files present significant challenges due to large size and high processing requirements. The CDX indexes used in this initial analysis were six terabytes (TB) in size, which is a fraction of the web archive content in W/ARC files constituting nearly 1.5 petabytes (PB) at the time of analysis (November 2018). The majority of the information that is provided alongside the files was extracted by the Library's Web Archiving Team, through a process called CDX Line Extraction, which focused on the mimetype field pulled from the CDX indexes. Except when referencing this specific field, we have referred to this designation by its more current name, "media type" (http://webarchive.loc.gov/all/20171105042213/http://www.iana.org/assignments/media-types/media-types.xhtml), and therefore, all references to "media type" below may be considered to be derived from the mimetype field from the CDX indexes. See the “How Was It Created” section below for more information on this process. II. What's Included? This dataset includes: - lcwa_gov_audio_data.zip - compressed bag containing the 1,000 randomly selected audio files from the archive, as well as manifest files with SHA-256 and SHA-512 checksums. The structure of the content follows the structure of the BagIt specification (see http://webarchive.loc.gov/all/20160830141859/https://tools.ietf.org/html/draft-kunze-bagit-08#section-2). - lcwa_gov_audio_metadata.csv - a CSV containing metadata derived from the CDX line entry for each audio file. The fields and their contents are described in the "Dataset Field Descriptions" section. III. How Was It Created? As mentioned above in the “About This Dataset” section, the bulk of this dataset was created using CDX Line Extraction. The extraction process used an Elastic MapReduce (EMR) cluster on AWS cloud services to run a series of MapReduce jobs (see https://en.wikipedia.org/wiki/MapReduce). The jobs filtered and sorted the CDX lines based on the following fields from the CDX line entries: - digest: a unique cryptographic hash of the web object’s payload at the time of the crawl, which provides a distinct fingerprint for that object; it is a Base32 encoded SHA-1 hash. - mimetype: two-part designation (type/subtype) that describes the nature and format of the web object, as reported by the server at the time of the crawl. - statuscode: represents the HTTP response code from the server at the time of the crawl, e.g. 200, 404, etc. - original: the original URL that was captured during the web harvesting process. See the CDX specification for more information about the fields: https://web.archive.org/web/20171123000432/https://iipc.github.io/warc-specifications/specifications/cdx-format/cdx-2006/. The CDX Line Extraction process involved multiple phases. First, a MapReduce job filtered out lines from the six terabyte corpus that were: 1) of the media type requested, in this case, media types that contained the string "audio" 2) had a status code of 200 3) and whose top level domain from the original URL was ".gov" The query results wrote matching lines to new CDX files, which were stored in an S3 bucket to be used in the next step. In this step, another MapReduce job pulled out all the digests from the filtered CDX files created by the first job and wrote them to a list that was also stored in an S3 bucket. Then, a Python script was used to randomly select one thousand digests from the digest list that was created by the second MapReduce job. Finally, a third MapReduce job took this subsection of one thousand digests as an input and extracted the CDX Line(s) each digest was referenced in, and wrote them to a CDX file that was stored in an S3 bucket. The CDX file from the final MapReduce Job was downloaded, converted to a CSV using a Python script, then used as the basis for any additional metadata extracted from the files or other computational methods of exploration. Additional information not extracted from the CDX index was also created for use in tracking file integrity and basic metadata. This included the use of the Apache Tika and MediaInfo tools to extract metadata from each of the files (see https://tika.apache.org/ and https://mediaarea.net/en/MediaInfo), which was then recorded in the accompanying metadata CSV. All information derived through Tika and Mediainfo is noted below. Response headers information stored with the web archives (content length) furnished the data for the file_size field. If the header did not include the content length, the file size (in bytes) was obtained after the file was retrieved. IV. Dataset Field Descriptions This section lists and describes each of the fields included in lcwa_gov_audio_metadata.csv. The CSV contains 26 fields (listed in the first line), and 1,000 lines with the corresponding information for each field as follows: - urlkey: the URL of the captured web object, without the protocol (http://) or the leading www. This information was extracted from the CDX index file. - timestamp: timestamp in the form YYYYMMDDhhmmss. The time represents the point at which the web object was captured, measured in GMT, as recorded in the CDX index file. - original: the URL of the captured web object, including the protocol (http://) and the leading www, if applicable, extracted from the CDX index file. - mimetype: the media type as recorded in the CDX. - statuscode: the HTTP response code received from the server at the time of capture, e.g., 200, 404. In this case, only codes that matched "200" were selected. - digest: a unique, cryptographic hash of the web object’s payload at the time of the crawl. This provides a distinct fingerprint for the object; it is a Base32 encoded SHA-1 hash, derived from the CDX index file. - mp3_version: the mp3 format standard version used to create the file, according to information extracted with Apache Tika. In the original Tika extract, this field was called "version". In cases where no information was found, a value of "-" was recorded (as is the case with every file that is not MPEG format). - album: the name of the "album" associated with the recording, according to information extracted with Apache Tika. In cases where no information was found, a value of "-" was recorded. - artist: the name of the "artist" associated with the recording, according to information extracted with Apache Tika. In cases where no information was found, a value of "-" was recorded. - composer: the name of the "composer" associated with the recording, according to information extracted with Apache Tika. In cases where no information was found, a value of "-" was recorded. - genre: the name of the "genre" (subject or style) of the recording, according to information extracted with Apache Tika. In cases where no information was found, a value of "-" was recorded. - release_date: the time the recording was purportedly originally published or otherwise made available to the public, according to information extracted with Apache Tika. In cases where no information was found, a value of "-" was recorded. - file_size: the size of the web object, in bytes, derived from additional processing methods used in conjunction with Apache Tika. - audio_encoding: the audio stream's format, according to information extracted with MediaInfo. In cases where no information was found, a value of "-" was recorded. - file_format: the name of the file's format, which encapsulates the audio bitstreams and includes metadata that describes the content within, according to information extracted with MediaInfo. In cases where no information was found, a value of "-" was recorded. - file_extension: the file's format extension, according to information extracted with MediaInfo. Each file's extension was assigned when the file was extracted from the web archives. The file's original URL was first checked to see if extension was present, and if so, the extension was assigned. If one was not present, the media type from the response header was tested against this Python library: https://docs.python.org/2/library/mimetypes.html (which was loaded with the complete mime type list from https://www.iana.org/assignments/media-types/media-types.xhtml to cover any gaps). The response from the Python library was used to interpret objects' media types and estimate the correct file extensions. An object's file extension might not be the same as it was when the file was created or harvested. - duration: the length of the audio recording measured in milliseconds, according to information extracted with MediaInfo. In cases where no information was found, a value of "-" was recorded. - format_settings: The type of information from MediaInfo in this field varies depending on the file format of the item. This field can contain information about endianness and data signatures, or stereo configuration. In cases where no information was found, a value of "-" was recorded. - bit_rate_mode: information about the speed of the audio stream's bit rate throughout the duration of the file, according to information extracted with MediaInfo. The values are either constant bit rate (CBR) or variable bit rate (VBR). In cases where no information was found, a value of "-" was recorded. - bit_rate: the speed of the audio stream, measured in bits per second, according to information extracted with MediaInfo. In cases where no information was found, a value of "-" was recorded. - channel_s: the count of channels in the audio stream, according to information extracted with MediaInfo. The values are either 1 or 2. In cases where no information was found, a value of "-" was recorded. - sampling_rate: the number of samples of audio carried per second, measured in Hz, according to information extracted with MediaInfo. In cases where no information was found, a value of "-" was recorded. - bit_depth: the number of bits of information in each of the file's audio samples, according to information extracted with MediaInfo. In cases where no information was found, a value of "-" was recorded. - writing_library: the tool used to encode the audio stream, according to information extracted with MediaInfo. In cases where no information was found, a value of "-" was recorded. - sha256: a unique cryptographic hash of the downloaded web object, computed using the SHA-256 function from the SHA-2 algorithm. It serves as a checksum for the downloaded web object and was created during the bagit process. - sha512: a unique cryptographic hash of the downloaded web object, computed using the SHA-512 function from the SHA-2 algorithm. It serves as a checksum for the downloaded web object and was created during the bagit process. V. Rights Statement This dataset was derived from content in the Library’s web archives. The Library follows a notification and permission process in the acquisition of content for the web archives, and to allow researcher access to the archived content, as described on the web archiving program page, https://www.loc.gov/programs/web-archiving/about-this-program/. Files were extracted from a variety of archived United States government websites collected in a number of event and thematic archives. See a representative Rights & Access statement for a sample collection which applies to all of the content in this dataset: https://www.loc.gov/collections/legislative-branch-web-archive/about-this-collection/rights-and-access/. VI. Creator and Contributor Information Creator: Chase Dooley Contributors: Aly DesRochers, Abbie Grotke, Charles Hosale, Kate Murray, Grace Thomas VII. Contact Information Please direct all questions and comments to webcapture@loc.gov.
https://s3.us-east-2.amazonaws.com/lclabspublicdata/lcwa_gov_audio_README.txt
s3.us-east-2.amazonaws.com
crawl=CC-MAIN-2020-34
13166
application/http; msgtype=response
sha1:OFUGTTTMK2ASZODZC3I4B4IR7VYS4VZC
<urn:uuid:113353b0-a680-49c2-ab02-b7e46da69c73>
2020-08-15T05:49:07Z
52.219.84.194
text/plain
sha1:R3K4YXPZAH6FXADB2YOS4RTH22Z6YMBQ
<urn:uuid:50713bc7-4503-4e25-9907-ac44e94388b5>
https://s3.us-east-2.amazonaws.com/lclabspublicdata/lcwa_gov_audio_README.txt
response
<urn:uuid:4b24a16b-5877-453f-935b-48b3e630ab46>
isPartOf: CC-MAIN-2020-34
publisher: Common Crawl
description: Wide crawl of the web for August 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-230.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/