Home » post » THOMAS bulk download!

Our mission

Free Government Information (FGI) is a place for initiating dialogue and building consensus among the various players (libraries, government agencies, non-profit organizations, researchers, journalists, etc.) who have a stake in the preservation of and perpetual free access to government information. FGI promotes free government information through collaboration, education, advocacy and research.

THOMAS bulk download!

Eric Mill announced today on the openhouseproject mailing list that he and Josh Tauberer (of GovTrack.us) and Derek Willis have completed a milestone in their project to produce a public domain scraper and dataset from THOMAS.gov. Here is the text of his message with links:

Hi all,

I’ve been working for the last month or two with Josh Tauberer (of GovTrack.us http://govtrack.us/) and Derek Willis on a project to produce a public domain scraper and dataset from THOMAS.gov http://thomas.gov/, the official source for legislative information for the US Congress.

It’s a reasonably well documented set of Python scripts, which you can find here: https://github.com/unitedstates/congress

We just hit a great milestone – it gets everything important that THOMAS has on bills, back to the year THOMAS starts (1973). We’ve published and documented https://github.com/unitedstates/congress/wiki all of this data in bulk, and I’ve worked it into Sunlight’s pipeline, so that searches for bills in Scout https://scout.sunlightfoundation.com/search/federal_bills/freedom%20of%20information use data collected directly from this effort.

The data and code are all hosted on Github on a “unitedstates https://github.com/unitedstates/” organization, which is right now co-owned by me, Josh, and Derek – the intent is to have this all exist in a common space. To the extent that the code needs a license at all, I’m using a public domain “unlicense https://github.com/unitedstates/congress/blob/master/LICENSE” that should at least be sufficient for the US (other suggestions welcome).

There’s other great stuff in this organization, too – Josh made an amazing donation of his legislator dataset https://github.com/unitedstates/congress-legislators, and converted it to YAML for easy reuse. I’ve worked that dataset into Sunlight’s products already as well. I’ve also moved my legal citation extractor https://github.com/unitedstates/citation into this organization — and my colleague Thom Neale has an in-progress parser for the US Code https://github.com/unitedstates/uscode, to convert it from binary typesetting codes into JSON.

Github’s organization structure actually makes possible a very neat commons. I’m hoping this model proves useful, both for us and for the public.

— Eric

— Developer | sunlightfoundation.com

CC BY-NC-SA 4.0 This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.


Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Archives
https://freegovinfo.info/node/3791/
freegovinfo.info
crawl=CC-MAIN-2021-49
81843
application/http; msgtype=response
sha1:ULJ66IVW33PWTNRBREN6N6USFLLJXD3W
<urn:uuid:e4432ca4-13ea-40f2-bd22-2ff02031f303>
2021-12-02T16:15:11Z
173.236.29.82
text/html
sha1:VJGX24YQMNUI46G4RWDQYL23SDEQZ3TR
<urn:uuid:30809759-d1d3-4117-8c97-cfb41e3038dc>
https://freegovinfo.info/node/3791/
response
<urn:uuid:e2b5eca2-229c-489d-a244-917cdab40e8b>
isPartOf: CC-MAIN-2021-49
publisher: Common Crawl
description: Wide crawl of the web for November/December 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-99
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/