Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Engineering and Public Policy: Data Management: Documentation

MetaData

Data that provides descriptive information (content, context, quality, structure, and accessibility) about a data product and enables others to search for and use the data product. In a lab setting, much of the content used to describe data is initially collected in a notebook; metadata is a more formal, sharable expression of this information. It can include content such as contact information, geographic locations, details about units of measure, abbreviations or codes used in the dataset, instrument and protocol information, survey tool details, provenance and version information and much more. Where no appropriate, formal metadata standard exists, for internal use, writing “readme” style metadata is an appropriate strategy.

The Digital Curation Center provides a catalog of common metadata standards, organized by discipline: http://www.dcc.ac.uk/resources/metadata-standards.

Some specific examples of metadata standards, both general and domain specific are:

Data Documentation

Make sure all data generated and/or collected is easy to understand and analyze. If someone were to look at this in 20 years, they should be able to understand what and why it was done. Stable non-proprietary software should be used.  Documentation may include, but is not limited to:

  • lab notebooks
  • methodology reports
  • codebooks with full variable and value labels,
  • documenting decisions about software,
  • tracking changes to different versions of the dataset,
  • recording assumptions made during analysis. 

Cite Your Data

Datasets should have an identifier and a locator. A persistent identifier is a unique Web-compatible, alphanumeric code that points to a specific dataset that will be preserved for the long term. The dataset identifier is an identifier of the dataset such as its title, file name, or even an object ID code. Examples of identifiers are UUID (Universally Unique Identifier), OID (Object Identifier), LSID (Life Sciences Identifier).
https://guides.library.cmu.edu/c.php?g=215654&p=1422394
guides.library.cmu.edu
crawl=CC-MAIN-2021-04
37273
application/http; msgtype=response
sha1:IACELMSB6W5JJB4LQ2CRRURUFGJNWXJN
<urn:uuid:f9af5658-843f-4c59-aac0-ef8ece677029>
2021-01-23T11:27:11Z
54.85.246.96
text/html
sha1:P4WTMIXY6UF6MTIAM4UQNZYRWRUUZGPX
<urn:uuid:0e51f2d5-805d-43ad-9e35-b8b93bbda155>
https://guides.library.cmu.edu/c.php?g=215654&p=1422394
response
<urn:uuid:b30a8f9d-24f5-486b-ad1b-86b11bedcbaf>
isPartOf: CC-MAIN-2021-04
publisher: Common Crawl
description: Wide crawl of the web for January 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-33.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/