Archivists Who Code

 View Only

Social Media data dictionary document

  • 1.  Social Media data dictionary document

    Posted 06-18-2024 12:30 PM

    Hey all,

    I just finished a draft version of the standard for the social media preservation format. Those of you who made it to BPE this year will be familiar with this, probably, but others will not. I took a rough it idea of what social media normalization would look like and in a BPE session a group of about I don't know how many workshopped what a platform-agnostic preservation format would look like. I'm asking if some of you would be willing to take a look and provide feedback, if any of either what needs modification or something not thought of in terms of elements.

    The file type is json, but with standardized elements. The json file is attached, just remove the .txt from the extension and open in firefox to see what it looks like.

    I wrote the data standard to give guidelines on what everything means so other users will know what and why a thing is used rather than willy-nilly just throwing stuff into the bucket. 

    (when this is all done I'm going to see if i can get preservica buy-in for rendering and possibly automated normalization, but at least rendering.)

    Thanks everyone,

    Brian



    ------------------------------
    Brian Thomas
    ------------------------------

    Attachment(s)
https://www.statearchivists.org/discussion/social-media-data-dictionary-document
www.statearchivists.org
CC-MAIN-2024-38
198417
application/http; msgtype=response
sha1:AHMOHB3BH2DDOR44MPDARFB4VB4QO7WV
<urn:uuid:cc44c233-6519-4a5f-8603-e0fcacf13211>
2024-09-18T00:56:22Z
34.225.242.50
text/html
sha1:P6EY6P5OZTAX4BNTSUMGRBEHJRTZRG46
<urn:uuid:338fcd04-a6a3-4dc0-aab9-ee8b3d4ed8e6>
https://www.statearchivists.org/discussion/social-media-data-dictionary-document
response
<urn:uuid:320c9389-eb99-49d7-a98b-049fce4e1810>
isPartOf: CC-MAIN-2024-38
publisher: Common Crawl
description: Wide crawl of the web for September 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-228
software: Apache Nutch 1.20 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/