Skip to content

A survey of Brazilian states' and municipalities' transparency and open data portals, as well as institutional websites, obtained from several public data sources. 🇧🇷 Levantamento de portais estaduais e municipais de transparência e dados abertos, bem como os portais institucionais, obtido a partir de diversas fontes públicas de dados.

master
Switch branches/tags
Code

A survey of Brazilian states' and municipalities' transparency and open data portals, as well as institutional websites, obtained from several public data sources.

Este texto também está disponível em português: 🇧🇷LEIAME.md.

goodtables.io

Transparency and Open Data Portals of Brazilian states and municipalities

Obtaining a list of transparency and open data portals of all of the 27 Brazilian states and 5,570 municipalities can be a daunting task. This project has the goal of consolidating and keeping up to date all of that reference information.

Besides, each state and municipality has executive and legislative branches, which effectively doubles the work required.

Steps of data discovery

We take many steps to find the actual transparency and open data portals. First, we need to find out the institutional websites of city councils and city halls in Brazil.

Finding the websites of city councils and city halls

One would expect for a database containing the official websites of all city councils and all city halls in Brazil to already exist. But alas, it does not. So we try to obtain possible links from several sources.

These data sources are in the sources directory. Check each individual data source directory for more information. Some examples of sources that are already implemented are:

The actual scripts that do the importing are in the tools/import directory. They output data that contains possible links to the websites of city administration, which is stored in the data/unverified directory. Data stored here is expected to still contain a lot of garbage and should not be committed to the repository.

Validation

The next step is link validation. This is done automatically, by a bot, and manually, with the help of a script. The code for those are in the tools/validation directory. The output, with validated data, goes to the data/valid directory.

We also apply the concept of continuous data integration to keep the integrity of data already validated. It's like continuous integration, but for data instead of software. It helps keep the incorrect data out of the repository, as every commit and pull request gets automatically verified, thanks to a hook on Github and the goodtables.io service.

Metadata is maintained using the Data Package specification, which also removes the friction out of the process of reusing the data. For more information, see the Frictionless Data website.

Finding transparency and open data portals in institutional websites

Once we have the links to the official website for city councils and city halls, we can analyze the content of the pages to look for the presence of links to transparency and open data portals.

This step is not yet implemented.

Goals

The goal of this project is to build a reference database of Brazilian states and municipalities, contemplating the legislative and executive branches, containing the following data on each one of them:

  • official website
  • transparency portal
  • open data portal

Contributing

We welcome contributions to the project. If you have an idea or want to improve something, please check out the contributing guide.

We also intend to collaborate on the monitoring of transparency of Brazilian states and municipalities in projects such as Colaboradados and the list of open data catalogs in Brazil.

About

A survey of Brazilian states' and municipalities' transparency and open data portals, as well as institutional websites, obtained from several public data sources. 🇧🇷 Levantamento de portais estaduais e municipais de transparência e dados abertos, bem como os portais institucionais, obtido a partir de diversas fontes públicas de dados.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages
https://github.com/augusto-herrmann/transparencia-dados-abertos-brasil
github.com
crawl=CC-MAIN-2022-05
221358
application/http; msgtype=response
sha1:IIINQ2P22INCNBI4PZ3YOV4T7RJMBYIG
<urn:uuid:d4ec88b8-b709-4e9b-a2f5-c74fc6588c19>
2022-01-22T15:56:30Z
140.82.114.4
text/html
sha1:IXU7OBVMVVCNW4FOV7N57GP3ATZVYMZC
<urn:uuid:233e6820-8321-4bd8-aa46-e118bf044232>
https://github.com/augusto-herrmann/transparencia-dados-abertos-brasil
response
<urn:uuid:75599731-64a7-4e1f-a975-9db4cfd9b37a>
isPartOf: CC-MAIN-2022-05
publisher: Common Crawl
description: Wide crawl of the web for January 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-11
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/