ProPublica opened a data store

Posted to Data Sources  |  Tags:  |  Nathan Yau

One of the main challenges of any data project is getting the data. It seems obvious, but the effort to get the right data to answer a question seems to catch people off guard. Even data that’s “free” to download can be a huge pain that ends up completely useless. ProPublica, the non-profit newsroom, deals with this stuff on a regular basis and hopes that some of their efforts can turn into a source of funding through the Data Store.

Like most newsrooms, we make extensive use of government data — some downloaded from “open data” sites and some obtained through Freedom of Information Act requests. But much of our data comes from our developers spending months scraping and assembling material from web sites and out of Acrobat documents. Some data requires months of labor to clean or requires combining datasets from different sources in a way that’s never been done before.

In the Data Store you’ll find a growing collection of the data we’ve used in our reporting. For raw, as-is datasets we receive from government sources, you’ll find a free download link that simply requires you agree to a simplified version of our Terms of Use. For datasets that are available as downloads from government websites, we’ve simply linked to the sites to ensure you can quickly get the most up-to-date data.

For datasets that are the result of significant expenditures of our time and effort, we’re charging a reasonable one-time fee: In most cases, it’s $200 for journalists and $2,000 for academic researchers.

I hope it works.

Favorites

10 Best Data Visualization Projects of 2015

These are my picks for the best of 2015. As usual, they could easily appear in a different order on a different day, and there are projects not on the list that were also excellent.

The Changing American Diet

See what we ate on an average day, for the past several decades.

How You Will Die

So far we’ve seen when you will die and how other people tend to die. Now let’s put the two together to see how and when you will die, given your sex, race, and age.

Life expectancy changes

The data goes back to 1960 and up to the most current estimates for 2009. Each line represents a country.
http://flowingdata.com/2014/03/04/propublica-opened-a-data-store/
flowingdata.com
crawl=CC-MAIN-2017-04
24573
application/http; msgtype=response
sha1:CHWAQVCP6US7W5IZAZXRTB3TWULGNDGF
<urn:uuid:a95d98f1-f859-4538-b07e-5aa87666c295>
2017-01-16T12:53:13Z
104.25.98.18
sha1:R72WND56L2AWL53DYOCMA247CU5MDARU
<urn:uuid:38471f3f-2f98-4c2f-a318-b0e320d07941>
http://flowingdata.com/2014/03/04/propublica-opened-a-data-store/
length
response
<urn:uuid:9f5ee839-0aa8-4470-b281-8f405be9b493>
robots: classic
hostname: ip-10-171-10-70.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2017-04
operator: CommonCrawl Admin
description: Wide crawl of the web for January 2017
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf