Difference between revisions of "Entities data"

From InventaireWiki
Jump to navigation Jump to search
(link concept to guides and glossary page)
(more detail link)
Line 156: Line 156:
  
 
<!--T:56-->
 
<!--T:56-->
* [[Guides]], some detailed explanations about how to deal with Inventaires data.
+
* [[Guides#Entities]], some detailed explanations about how to deal with Inventaire data.
 
* [[Entities issues]]
 
* [[Entities issues]]
 
* [https://www.wikidata.org/wiki/Wikidata:WikiProject_Books Wikidata:WikiProject_Books]
 
* [https://www.wikidata.org/wiki/Wikidata:WikiProject_Books Wikidata:WikiProject_Books]

Revision as of 12:35, 1 July 2021

Other languages:
English • ‎français


The Inventaire project has two pillars:

  • The book sharing web application, inviting anyone to make the inventory of their books
  • The open data project aiming at building a CC0 Wikidata-centered database on resources, to support the needs of the web app, while being enriched by its users, and more broadly speaking, to fulfill our mission: mapping resources with libre software and open knowledge, starting with books.

This article aims to document this second aspect, by:

  • making the context of open data production in Inventaire understandable by people who might be more interested in the structured data produced by the application than by its book sharing aspect
  • helping data re-use, especially, but not only, by importing data produced in Inventaire back to Wikidata

If you are here, maybe you would like to go to:

  • data.inventaire.io, home page link to send to everyone who could be interested in the data aspect of Inventaire, but not necessarily in the web app itself
  • Guides describing in detail what to do with Inventaire data

Data sources

Data about books, authors, series, subjects, etc., come from three sources:

Wikidata

Wikidata is a free, collaborative, multilingual, secondary database, collecting structured data to provide support for Wikipedia, Wikimedia Commons, the other wikis of the Wikimedia movement, and to anyone in the world.

The Wikidata community has a dedicated task force on books: WikiProject Books

Internal database

As Wikidata doesn't aim to gather data on all books and their editions, but only those that match certain notability criteria, we can't only rely on it, thus the introduction of a internal database, made of users contributions, linked to Wikidata entities. See Data contribution guidelines.

Those data will are published under a CC0 license, so that the Wikidata community can import the entities that are relevant to Wikidata following their criteria.

Web data seeds

Basic Facts on books editions extracted from the wide web, processed, and enriched to bootstrap the data set of an edition missing to the database. This is made in an attempt to ease and somewhat automatize what a user encountering missing data would have to do manually: go on the web, searching basic facts like a title, the author's name, a book cover. This data collection is done by a closed source external service nicknamed dataseed. That's a temporary solution, waiting to be replaced by more reliable data sources, like raw libraries and publishers open data.

Data structure

The term entity gathers all the concepts of works (books, comics, manga, etc.), editions of those works, their authors and series. Largely following Wikidata data model, those entities are linked together by properties.

Differences between Wikidata and Inventaire terminologies

See Glossary

Examples of relations between entities

  • I, Robot, the book edition uniquely identified by its ISBN: 978-0-00-753227-8 → is an edition of (property P629 in Wikidata) → the [I, Robot], uniquely identified in Wikidata as Q393018
  • the work I, Robot → has for author (property P50 in Wikidata) → Isaac Asimov, uniquely identified in Wikidata as Q34981

This is how this graph of relations looks like:

  • Series are made of several works
  • An author can be the author of several works
  • A work can have several editions
  • Usually an edition is the edition of a unique work. In special cases, an edition can be the edition of several works (for instance, an edition that would gather the Harry Potter volumes 1 and 2): we call those composite or multi-work editions.

The user inventory is made of items, each item being an instance of an edition.


Data editor

Entity data editor.png

To edit Inventaire entities, you first need to create an Inventaire account. In addition, to edit Wikidata entities from the Inventaire entity editor, you will be invited to connect your Wikidata account. But you can also simply click the "Edit on Wikidata" button.

To find an entity's editor page:

The editor is largely inspired by Wikidata own editor so that the Wikidata community can feel home ;) But at the same time, it is designed to make the contribution of anyone - aware of Wikidata or not - easy and welcome. Inputs values are thus more constrained than in Wikidata, a few examples:

  • you won't be able to state that a book has "mont Fuji" as an author,
  • following the WikiProject Books recommendations to separate works and their editions (the FRBR-ish way), you won't be allowed to set an ISBN on a work,
  • additionally, you won't be allowed to set on editions claims that belong to work level, as we assume editions inherit those claims from the works they are associated with, and that it would be unnecessary duplicates.

Raw data

API

See api.inventaire.io section on entities

The API should look somewhat familiar to Wikidata API users, with a few notable differences:

  • for convenience, the API returns both Wikidata and Inventaire entities
  • the Inventaire entities data model doesn't have qualifiers at the moment, so labels and claims from both Wikidata and Inventaire entities are returned simplified
  • as Inventaire knows about both Wikidata and Inventaire entities, claims properties and entity values need to be prefixed:
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX inv: <https://inventaire.io/entity/>

SPARQL endpoint

A Query Service is available at https://query.inventaire.io/ with many example queries to get you started. The Query Service is unfortunately currently only periodically updated (rather than live updated), so you might experience some differences with the canonical data you see on Inventaire.

Dumps

All dumps can be found at https://dumps.inventaire.io

Available formats JSON, NDJSON, TTL (turtle)

Data import

For anyone willing to import data to Inventaire CC0 data, here are the properties used by Inventaire, where:

If you instead import data directly to Wikidata:

Statistics

See also
https://wiki.inventaire.io/w/index.php?title=Entities_data&diff=prev&oldid=2409
wiki.inventaire.io
crawl=CC-MAIN-2022-27
41412
application/http; msgtype=response
sha1:GTXRNRLKWFV67RQW6DPEBGQPKYWYNNXA
<urn:uuid:f8448b97-f031-49a7-a1a0-dba4b7912519>
2022-06-30T04:38:40Z
78.47.243.145
text/html
sha1:V77FDS4B76ZK6YHU5IXPQRDMRDSCYWLA
<urn:uuid:71b64194-59bc-42f4-ab93-b8d8753dbf80>
https://wiki.inventaire.io/w/index.php?title=Entities_data&diff=prev&oldid=2409
response
<urn:uuid:9b15a2a6-ed98-4338-8505-24d3591b3a9b>
isPartOf: CC-MAIN-2022-27
publisher: Common Crawl
description: Wide crawl of the web for June/July 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-246
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/