Catalog Basics

Open data catalogs - or portals - are simple in purpose but may appear complicated at first glance. A homepage will often feature a slideshow, list of blog posts, or a visualization using data from the underlying catalog; these homepage assets can easily obscure the underlying structure of the data.

Understanding the structure of an open data catalog can help users, developers, and stakeholders make the best use of open data tools.

Catalog

The catalog is a wrapper containing information, sorted into datasets. A catalog should provide the answers to basic questions such as: “Who is providing this data?” and “Under what conditions?”

DCAT - an RDF vocabulary designed to facilitate interoperability between data catalogs published on the Web - recommends providing the following fields: catalog record, dataset, description, homepage, language, license, publisher, release date, rights, spatial, themes, title and update date.

Dataset

A dataset contains individual resources as well as metadata. Metadata is the “Who, What, When, Where, Why” of each dataset. There are a number of specifications for dataset metadata and DKAN currently complies with the DCAT and Project Open Data schemas.

It is possible to add new fields to DKAN to conform to additional specifications or custom requirements.

Resource

Resources are the actual files, APIs or links that are being shared. Resource types include csv, html, xls, json, xlsx, doc, docx, rdf, txt, jpg, png, gif, tiff, pdf, odf, ods, odt, tsv, geojson and xml files. If the resource is an API, it can be used as a live source of information for building a site or application.

Datasets and Resources within DKAN and CKAN

The following images display how datasets and resources are structured within DKAN and CKAN. Screenshots are from Data.gov which is powered by CKAN. DKAN follows CKAN conventions where possible.

introduction/../images/DatasetResourceDiagram2.png
introduction/../images/datasets-resources.png
Metadata
https://docs.getdkan.com/en/stable/introduction/catalog-basics.html
docs.getdkan.com
crawl=CC-MAIN-2020-16
13824
application/http; msgtype=response
sha1:5CDCQVMVNBBEPKZ44NB2QMUXAFIVMKVS
<urn:uuid:90f56a52-db45-4f23-811d-ecadc0e4d71e>
2020-04-04T19:11:26Z
104.17.33.82
text/html
sha1:JFW4DAR23JAGY7FDTALSUY4PTSKZRZRL
<urn:uuid:59b43513-9546-478f-812f-f820d96b3c76>
https://docs.getdkan.com/en/stable/introduction/catalog-basics.html
response
<urn:uuid:c9b0abc8-da30-4330-854b-51080de5c4b0>
isPartOf: CC-MAIN-2020-16
publisher: Common Crawl
description: Wide crawl of the web for March/April 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-116.ec2.internal
software: Apache Nutch 1.16 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/