Clarino-logo
COMEDI :: COMEDI documentation



COMEDI – The COmponent Metadata EDItor

Metadata Records page

On the Metadata Records page, you can

  • Create a new metadata record;
  • Clone an existing metadata record;
  • Upload a metadata record file;
  • Search for and select an existing metadata record for viewing and editing.

Existing metadata records are listed by resource name and identifier. By clicking on their Identifier you can inspect or edit the metadata.

Creating new metadata

You can create a new, initially empty metadata record by choosing a profile from the drop-down menu Choose profile. If the desired profile is not among the listed profiles you can alternatively provide the profile ID (e.g., clarin.eu:cr1:p_1407745711925).

You have to specify an identifier that uniquely identifies your metadata record in the system. Allowed are alphanumeric characters, including space, ‘_’, ‘/’, ‘@’, ‘#’, ‘+’, ‘?’, ‘.’ and ‘*’. Please keep the identifier reasonably short.

The metadata is connected to the owner user ID (which is your login user ID); normally, only you are able to edit or delete your metadata. In contrast, everybody can see your metadata.

The metadata owner should not be confused with the metadata Creator. When creating new metadata, the Creator field in the metadata (which is part of the metadata header) is filled out automatically with the user clear name (first and last name). When you upload existing metadata, the creator field already exists and might be different from your login clear name.

Cloning an existing metadata record

You can clone an existing metadata record and giving it a new unique identifier. The new record is equal to the cloned one, except that the Self link and other PIDs are cleared. In addition, the creator is set to your user name, and the creation date is adjusted.

Uploading a metadata record file

To upload a metadata record file in CMDI 1.1 XML format, first select the file on your local computer using the Choose file button, and then click Upload. See also Batch upload below.

Viewing and editing existing metadata

When you clicked on a link connected to existing metadata, you get to the Metadata Editor page.

Metadata Editor page

Viewing

The metadata can be viewed, one top-level component of the profile at a time. From the left-hand list, you can select the top-level component to be viewed, or you can use the Next and Previous buttons to navigate.

Editing

To edit the contents of a component, you shift to edit mode by clicking the Edit radio button.
Here, as in View mode, the components and their elements are shown hierarchically as boxes containing boxes. Components can be expanded and shrunk by clicking on their name. Initially, all but the top-level component are shrunk.

In the element box, its sub-components, elements and components, are shown.

Right to the component or element name, in square brackets, its minimal and maximal cardinality is shown, together with the number of actual instantiations [to be implemented for elements]. Example: [1/0-∞] means that there is one instantiation of that component, and its minimal and maximal cardinalities are zero and infinity.

If the minimal cardinality of a component or element is 1, that component or element is always present, even if it has no content. If an obligatory element has no content, it is shown in red in View mode.

If a component has max cardinality infinity, a “+” button is present adjacent to its name. By clicking the button, you create a new instance of that component.

The same is true for elements: if an element’s cardinality is infinity, there is a “+” button next to the element name, which creates a new instance of the element.

If a component or an element has min cardinality zero and has not been instantiated, this is indicated by the component name being grayed out. Again, you can create an instance of it by clicking the “+” button.

Instantiations of components and elements can be deleted if this does not result in a component or element cardinality lower than the minimal one. This is done using the “-” button.

An element can have an xml:lang attribute. Then, a lang field is shown together with the input field for the element. You should use an ISO 639-1 (two-letter) language code if available, and an ISO 639-2 (three-letter) code otherwise.

Shared editing

Normally, a metadata record that you created or imported cannot be edited by other users. A user can however request write access to a specific metadata record that you own. When this happens, the record owner gets an email notification, and the requested metadata and the users requesting write access are listed on the Metadata Records page. You may either accept or deny each single request.

Vice versa, you can request write access to a metadata record owned be another user. This can be done by clicking the Request write access button in the Metadata editor while viewing the record.

When a user is editing a record (viewing it in edit mode), this record is locked and cannot be edited by other users (and other sessions) at the same time. A user can enforce editing a record that is in edit mode by another user if there has been no editing activity for 15 minutes. (This can be useful if a user leaves the browser without quitting edit mode.)

Groups

Users can be members of one or more user groups. When you are member of a group, you can associate a metadata record with that group. This gives all members of that group write access to the record.

Groups are administered on the Metadata Editor page. There, you can see which groups the selected metadata record is associated with, you can see the groups you are member of, and you can create new groups, connect the selected record to a group, and request membership in one of the existing groups.

When you request membership in an existing group, the creator of the group will receive an email notification, and he/she/it can approve or deny the request, much like a write access request.

When you create a new identifier or group keep in mind that COMEDI is an open tool; everybody can see all identifiers and groups. Therefore, try to find names that are not too generic (e.g., “corpus1”, “mygroup”, “test”, etc.), but rather make their association to your institute or project conspicuous (e.g., “TextLab”, “INESS”, etc.).

Resources

The ResourceProxyList section of the Resources can be edited in much the same way as regular components. (Other Resource sections will be added.)
Resource proxy ids are available as values for the ref attribute of elements and components, and only declared ids can be listed in a ref attribute. Click on the ref: link next to the element or component name to edit the ref attribute.

Keyboard shortcuts

  • Ctrl-Arrow down: Go to next component or element (put it into focus)
  • Ctrl-Arrow up: Go to previous component or element
  • Ctrl-Arrow left: Go to parent component
  • Ctrl-Arrow right: When an element is in focus, put element input field into focus
  • Ctrl-Return: Expand/shrink focused component
  • Ctrl-L: Center focused element/component on the screen
  • Ctrl-A: Toggle between View and Edit mode
  • Ctrl-H: Select component
  • Ctrl-+, Ctrl-Space: Add an element or component (same as clicking the “+” button)
  • Ctrl-Shift-Arrow left: Go to previous main component
  • Ctrl-Shift-Arrow right: Go to next main component

Copying and cloning

A component can be filled with the contents of a component of the same type from a stored record. When you click on select component next to the component name, a list of existing components of the same type (ComponentId) appears. You can scroll through the list using Ctrl-Arrow left and Ctrl-Arrow right. When you select one of these, its content is copied into the component in focus.

To make this feature safe to use, a component can only be filled with new content if it is empty. If a component already has content, you can clear it by clicking Remove content.

Useful components can be marked as favorites in the component selection box; they will appear first next time you evoke the component list for this component type.

  • TODO: Batch replace; more

Saving and storing

After each edit operation, the record is automatically stored in the database, overwriting the previous version.

You can save a snapshot of the record by pressing the Save button. You can at any time revert to an earlier snapshot of the metadata record.

Creating new metadata

When you have chosen a profile to create a new metadata record, you are presented on the Metadata Editor page with empty metadata. Again, obligatory components are shown, and obligatory elements of them are shown with empty values. You proceed creating the metadata by providing values for those elements, adding element and component instances, and cloning from existing metadata component instantiations, as described above.

Consistency checking, validation

When you have entered a value of an element, the value is validated against the element description on the fly. (To be exact, validation happens when the input field loses focus, e.g. when you move the caret out of the field.) If the value is invalid, an error message is displayed below the value.

Validity of your metadata is indicated in the Status row below Metadata Record. When your metadata is not valid , clicking on Show details displays a list of all validation errors. You can click on any of the errors and jump right to the place of the error.

Status

Metadata that is being created has status in progress. When your metadata is complete and valid, you can change the status to published. This means that your metadata is available for hosting via the OAI-PMH endpoint. You can continue editing published metadata, but when you introduce a validation error, the status is set back to in progress, and you have to change it manually back to published when you have corrected the error.

Batch upload

There is a JSON/REST endpoint for batch uploading metadata to COMEDI.

The URL is http://clarino.uib.no/comedi/upload, and the following GET and POST parameters have to be provided:

  • GET:
    • login-code, if you are using local login, or
    • session-index, if you are using federated login (Feide, eduGAIN, OpenIdP, Clarin-IdP)
    • group, the user group the metadata should be attached to. This is optional.
  • POST:
    • file, with the metadata XML as attachment.

The metadata identifier is equal to the filename minus extension.

A valid login-code or session-index connected to your user ID is displayed on the Metadata Records page once you are logged in. login-code and session-index expire after eight hours.

The response is a JSON string that either reports success or an error message, like this:

{"success":true,"identifier":"tiger-treebank"}
{"error":"No valid session-index or login-code has been supplied."}

You may use the Unix tool curl to upload metadata. An example request would look like this:

curl -F "file=@tiger-treebank.xml" "http://clarino.uib.no/comedi/upload?group=CLU&session-index=_a19b86a94e9d54d8fa11c480bb51024c86953a9149"

Download

Metadata can be downloaded either via OAI-PMH (see below), or as plain CMDI XML files.

CMDI XML download

The URL for downloading CMDI metadata is http://clarino.uib.no/comedi/metadata-record?identifier=<identifier> .

The format of the returned metadata depends of the value of the accept parameter in the request header. If the value is application/x-cmdi+xml or application/xml, the CMDI record is returned in XML format; otherwise, a HTML page is returned. In order to download the CMDI XML record you can either right-click or Alt-click (MacOS) on the download link, or use the unix programs curl or wget to fetch the record.

OAI-PMH harvesting of metadata

The URL for harvesting metadata via OAI-PMH is

Values of the set parameter in ListRecords are the defined user groups.

Some sample requests:


Design & implementation: Paul Meurer, 2020
http://clarino.uib.no/comedi/page?page-id=Metadata_Editor
clarino.uib.no
crawl=CC-MAIN-2020-29
43070
application/http; msgtype=response
sha1:MGJFV5PBLUFMLRW2K7SD3BUZ6XFVP5OP
<urn:uuid:9208c720-937c-42f1-82ac-195a072cd63b>
2020-07-06T23:46:36Z
129.177.234.50
text/html
sha1:SIQKPQEZMOOOSNO4SHFTZLG2BA2ZVDKY
<urn:uuid:c4c24716-f212-42ef-a8da-deb5a1a0052e>
http://clarino.uib.no/comedi/page?page-id=Metadata_Editor
response
<urn:uuid:d508174a-25eb-432a-a74e-07df875f9ac7>
isPartOf: CC-MAIN-2020-29
publisher: Common Crawl
description: Wide crawl of the web for July 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-176.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/