Date: Sun, 25 Oct 2020 08:11:26 -0500 (CDT) Message-ID: <1924069592.81572.1603631486343@duraspace-app01.managed.contegix.com> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_81571_980324251.1603631486342" ------=_Part_81571_980324251.1603631486342 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html Batch Metadata Editing

Batch Metadata Editing

=20 =20

Batch Metadata Edi= ting Tool

DSpace provides a batch metadata editing tool. The batch editing tool is= able to produce a comma delimited file in the CSV format. The batch editin= g tool facilitates the user to perform the following:

For information about configuration options for the Batch Metadata Editi= ng tool, see Batch Metadata Editing Configuration

Export Function

Web Interface Export

Batch metadata exports (to CSV) can be performed from the Administrative= menu:

Please see below documentation for more information on the CSV format and actions that can be perfo= rmed by editing the CSV= .

Command Line Export

The following table summarizes the basics.

Command used:

[dspace]/bin/dspace metadata-export

Java class:

org.dspace.app.bulkedit.MetadataExport

Arguments short and (long) forms):

Description

-f or --file

Required. The filename of the resulting CSV.<= /p>

-i or --id

The Item, Collection, or Community handle or = Database ID to export. If not specified, all items will be= exported.

-a or --all

Include all the metadata fields that are not = normally changed (e.g. provenance) or those fields you configured in the [dspace]/config/modules/bulkedit.cfg to be ignored on export.

-h or --help

Display the help page.

To run the batch editing exporter, at the command line:

=20
[dspace=
]/bin/dspace metadata-export -f name_of_file.csv -i 1023/24
=20

Example:

=20
[dspace=
]/bin/dspace metadata-export -f /batch_export/col_14.csv -i /1989.1/24=20

In the above example we have requested that a collection, assigned handl= e '1989.1/24' export the entire collection to the file 'col_14= .csv' found in the '/batch_export' directory.

Please see below documentation for more information on the CSV format = and actions that can be performed by editing the CSV .

Import Function

Importing large CSV files

It is not recommended to import CSV files of more than 1,000 lines (i.e.= 1,000 items). When importing files larger than this, it may be diffic= ult for an Administrator to accurately verify the changes that the import t= ool states it will make. In addition, depending on the memory available to = the DSpace site, large files may cause 'Out Of Memory' errors part way thro= ugh the import process.

Web Interface Import

Batch metadata imports (from CSV) can be performed from the Admini= strative menu:

  • First, complete all edi= ting of the CSV and save your changes
  • Login as an Administrative User
  • Click "Import Metadata" and select the CSV file
    • In XMLUI, "Import Metadata" can be found under the "Administrative" men= u on any page
    • In JSPUI, "Import Metadata" can be found under the "Administer" menu (u= nder your user account dropdown). On the Adminstration Tools page, select "= Import Metadata" from the "Content" dropdown
  • After uploading the CSV, you will be presented with a summary of all ch= anges that will be performed in the system. You can review these changes an= d choose whether to apply them or cancel.

Command Line Import

The following table summarizes the basics.

Command used:

[dspace]/bin/dspace metadata-import

Java class:

org.dspace.app.bulkedit.MetadataImport

Arguments short and (long) forms:

Description

-f or --file

Required. The filename of the CSV file to loa= d.

-s or --silent

<= /td>

Silent mode. The import function does not pro= mpt you to make sure you wish to make the changes.

-e or --email

The email address of the user. This is only r= equired when adding new items.

-w or --workflow

When adding new items, the program will queue= the items up to use the Collection Workflow processes.

-n or --notify

<= /td>

when adding new items using a workflow, send = notification emails.

-t or --template

When adding new items, use the Collection tem= plate, if it exists.

-h or --help

Display the brief help page.

Silent Mode should be used carefully. It is possible (and probable) that= you can overlay the wrong data and cause irreparable damage to the databas= e.

To run the batch importer, at the command line:

=20
[dspace=
]/bin/dspace metadata-import -f name_of_file.csv
=20

Example

=20
[dspace=
]/bin/dspace metadata-import -f /dImport/col_14.csv
=20

If you are wishing to upload new metadata without bitst= reams, at the command line:

=20
[dspace=
]/bin/dspace metadata-import -f /dImport/new_file.csv -e joe@user.com -w -n=
 -t
=20

In the above example we threw in all the arguments. This would add the m= etadata and engage the workflow, notification, and templates to all be appl= ied to the items that are being added.

CSV Format

The CSV (comma separated values) files that this tool can import and exp= ort abide by the RFC4180 CSV format. This means that new = lines, and embedded commas can be included by wrapping elements in double q= uotes. Double quotes can be included by using two double quotes. The code d= oes all this for you, and any good csv editor such as Excel or OpenOffice w= ill comply with this convention.

All CSV files are also in UTF-8 encoding in order to support all languag= es.

File Structure

The first row of the CSV must define the metadata values that the rest o= f the CSV represents. The first column must always be "id" whic= h refers to the item's internal database ID. All other co= lumns are optional. The other columns contain the dublin core metadata= fields that the data is to reside.

A typical heading row looks like:

=20
id,coll=
ection,dc.title,dc.contributor,dc.date.issued,etc,etc,etc.
=20

Subsequent rows in the csv file relate to items. A typical row might loo= k like:

=20
350,229=
2,Item title,"Smith, John",2008
=20

If you want to store multiple values for a given metadata element, they = can be separated with the double-pipe '||' (or another character that you d= efined in your modules/bulkedit.cfg file). For example:

=20
Horses|=
|Dogs||Cats
=20

Elements are stored in the database in the order that they appear in the= CSV file. You can use this to order elements where order may matter, such = as authors, or controlled vocabulary such as Library of Congress Subject He= adings.

Editing the CSV

If you are editing with Microsoft Excel, be sure to open= the CSV in Unicode/UTF-8 encoding

By default, Microsoft Excel may not correctly open the CSV in Unicode/UT= F-8 encoding. This means that special characters may be improperly displaye= d and also can be "corrupted" during re-import of the CSV.

You need to tell Excel this CSV is Unicode, by importing it as follows. = (Please note these instructions are valid for MS Office 2007 and 2013. = Other Office versions may vary)

  • First, open Excel (with an empty sheet/workbook open)
  • Select "Data" tab
  • Click "From Text" button (in the "External Data" section)
  • Select your CSV file
  • Wizard Step 1
    • Choose "Delimited" option
    • Start import at row: 1
    • In the "File origin" selectbox, select "65001 : Unicode (UTF-8)"
      • NOTE: these encoding options are sorted alphabetically, so "Unicode (UT= F-8)" appears near the bottom of the list.
    • Click Next
  • Wizard Step 2
    • Select "Comma" as the only delimiter
    • Click Next
  • Wizard Step 3
    • Select "Text" as the "Column data format" (Unfortunately, this must= be done for each column individually in Excel)
      • At a minimum, you MUST ensure all date columns (e.g. dc.date.issued) ar= e treated as "Text" so that Excel doesn't autoconvert DSpace's YYYY-MM-DD f= ormat into MM/DD/YYYY
      • To avoid such autoconversion, it is safest to ensure each column is tre= ated as "Text".  Unfortunately, this means selecting each column one-b= y-one and choosing "Text" as the "Column data format".
    • Click Finish
  • Choose whether to open CSV in the existing sheet or a new one

Tips to Simplify the Editing Process

 When editing a CSV, here's a couple of basic tips to keep in mind:=

  1. The "id" column MUST remain intact. This column also must always have a= value in it.
  2. To simplify the CSV, you can simply remove any columns you do NOT wish = to edit (except for "id" column, see #1). Don't worry, removing the entire = column won't delete metadata (see #3)
  3. When importing a CSV file, the importer will over= lay the metadata onto what is already in the repository to = determine the differences. It only acts on the contents of the CSV= file, rather than on the complete item metadata. This means that the CSV f= ile that is exported can be manipulated quite substantially before being re= -imported. Rows (items) or Columns (metadata elements) can be removed and w= ill be ignored. 
    1. For example, if you only want to edit "dc.subject", you can remov= e ALL columns EXCEPT for "id" and "dc.subject" so that you can just manipul= ate the "dc.subject" field. On import, DSpace will see that you've only inc= luded the "dc.subject" field in your CSV and therefore will only update the= "dc.subject" metadata field for any items listed in that CSV.=
  4. Because removing an entire column does NOT delete metadata value(= s), if you actually wish to delete a metadata value you should leave the co= lumn intact, and simply clear out the appropriate row's value (in that colu= mn).

 

Editing Collect= ion Membership

Items can be moved between collections by editing the collection handles= in the 'collection' column. Multiple collections can be included. The firs= t collection is the 'owning collection'. The owning collection is the prima= ry collection that the item appears in. Subsequent collections (separated b= y the field separator) are treated as mapped collections. These are the sam= e as using the map item functionality in the DSpace user interface. To move= items between collections, or to edit which other collections they are map= ped to, change the data in the collection column.

Adding Metadata-On= ly Items

New metadata-only items can be added to DSpace using the batch metadata = importer. To do this, enter a plus sign '+' in the first 'id' column. The i= mporter will then treat this as a new item. If you are using the command li= ne importer, you will need to use the -e flag to specify the user email add= ress or id of the user that is registered as submitting the items.

Deleting Metadata

It is possible to perform metadata deletes across the board of certain m= etadata fields from an exported file. For example, let's say you have used = keywords (dc.subject) that need to be removed en masse. You would = leave the column (dc.subject) intact, but remove the data in the correspond= ing rows.

Performing 'acti= ons' on items

It is possible to perform certain 'actions' on items.  This is achi= eved by adding an 'action' column to the CSV file (after the id, and collec= tion columns).  There are three possible actions:

  1. 'expunge' This permanently deletes = an item.  Use with care!  This action must be enabled by setting = 'allowexpunge =3D true' in modules/bulkedit.cfg
  2. 'withdraw' This withdraws an item from the ar= chive, but does not delete it.
  3. 'reinstate' This reinstates an item that has = previously been withdrawn.

If an action makes no change (for example, asking to withdraw an item th= at is already withdrawn) then, just like metadata that has not changed, thi= s will be ignored.

Migrating Dat= a or Exchanging data

It is possible that you have data in one Dublin Core (DC) element and yo= u wish to really have it in another. An example would be that your staff ha= ve input Library of Congress Subject Headings in the Subject field (dc.subj= ect) instead of the LCSH field (dc.subject.lcsh). Follow these steps and yo= ur data is migrated upon import:

  1. Insert a new column. The first row should be the new metadata element. = (We will refer to it as the TARGET)
  2. Select the column/rows of the data you wish to change. (We will refer t= o it as the SOURCE)
  3. Cut and paste this data into the new column (TARGET) you created in Ste= p 1.
  4. Leave the column (SOURCE) you just cut and pasted from empty. Do not de= lete it.

Common Issues

Metadata values in CSV export seem to have = duplicate columns
DSpace responds with "No changes were detected" w= hen CSV is uploaded

Unfortunately, this response may be caused in many ways

  • It's possible the CSV was not saved properly after editing. Check = that the edits are in the CSV, and that there were no backend errors in the= DSpace logs (which would be an indication of an invalid or corrupted CSV f= ile)
  • Depending on the version of DSpace, you may be encountering this known = bug with processing linebreaks in CSV files: DS-32= 45
  • If you are setting a new embargo date in the CSV, ensure that the embar= go lift date is a future date.  It's been reported that past dates may= cause DSpace to ignore item changes.

------=_Part_81571_980324251.1603631486342--
https://wiki.lyrasis.org/exportword?pageId=68064808
wiki.lyrasis.org
crawl=CC-MAIN-2020-45
32386
application/http; msgtype=response
sha1:IV2PNTKBGHD2AHIKUQOVRXCOERSYETXZ
<urn:uuid:f4132773-1bef-4ff9-9f6b-dc1078de7f16>
2020-10-25T13:11:26Z
199.119.127.15
message/rfc822
sha1:XUOZOOGZRZIMH573MPD6YGXHCGYYJI4B
<urn:uuid:254a0f16-ed17-43f9-8186-313b00f7847a>
https://wiki.lyrasis.org/exportword?pageId=68064808
response
<urn:uuid:f8a828f6-8d21-4291-be92-783217b17638>
isPartOf: CC-MAIN-2020-45
publisher: Common Crawl
description: Wide crawl of the web for October 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-188.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/