Documentation:Library:Circle/FROGBEAR Organizing content for upload

From UBC Wiki
Jump to navigation Jump to search

Overview

This document describes the steps that need to be taken in order to organize content for upload into FROGBEAR repository.

All files (images, video, audio, etc.) to be included in the repository must be accompanied by a metadata spreadsheet (.xls or .xlsx format). Files that are not accompanied by a metadata spreadsheet, or that are not referenced in the spreadsheet, will not be included in the repository.
In addition to the files and the metadata, two kinds of permission forms are required. First, permission is needed from anyone who created a file being added to the repository (the person who operated the camera, audio recorder, or other device; or the person who wrote a text or other document). This allows us to make the files publicly available through UBC Library Open Collections. Second, anyone (cluster participant or not) who appears in media files must grant permission to have their image and/or voice made public. This permission is granted by signing the forms listed below.

Checkbox

Document Checklist

cIRcle License: This license must be signed by anyone who creates content for the project including metadata, images, videos, scans, etc.
  • See guidelines in Step 1 for completing the cIRcle license. [NB: This license was updated on November 16, 2021 to version 3.0.]
Consent for Disclosure of Personal Information (Release form) – release required from every identifiable person appearing in a file (face visible in an image/video, voice heard in an audio recording)

The above forms must be provided for all relevant material (one form per person per trip). They will be kept on file by project staff, but will not be part of the public repository.

Step 1: Prepare the documents

cIRcle License

  • Language: The cIRcle License must be in English.
  • Names: The content creators name should be written in the Latin alphabet (pinyin, romaji, etc.). Check that the same spelling is used in both the metadata spreadsheet and the all forms
  • Description of work(s): Include the following statement and add relevant details for your trip, "All digital objects created as part of the From the Ground Up: Buddhism and East Asian Religions project [research cluster ID] [trip name] [trip dates]."
  • Creative Commons: Content creators must answer YES to both questions in the Creative Commons License so that the UBC Library can host and display the files and others can reuse them

See this example of a completed license for reference.

Upload forms

In order to transfer the paper copy on to the computer, it has to be scanned and uploaded as a PDF

Name files

Ideally, the filename for the form should contain the name of the person signing them

  • ex. “ZhangMing_License.pdf”

Alternatively, upload a single PDF containing all the signed forms.

Step 2: Organize your content

For general OwnCloud instructions, see Documentation:Library:Circle/FROGBEAR OwnCloud

Decide which files should be included in the repository. Within OwnCloud, there are separate folders for

  • material to include in repository (for research and educational use) -ex. Item record in Open Collections
  • material for project publicity and other purposes, but not intended for the library repository (for example, group photos of participants) -ex. Frogbear website

We have organized the upload area on OwnCloud into separate areas for each project cluster.

Owncloud screenshot1.PNG

When uploading material, look for the folder of repository material for your cluster, and create a new folder for each trip (e.g., “2019 Trip to Kyoto”). Within that folder, create folders for licenses from the content creators (“cIRcle Licenses”) and for permission forms signed by the individuals appearing in the media (“Individual Releases”).

Owncloud screenshot2.PNG

Step 3: Organize folders for upload

Data for the repository (images, video, audio, transcripts, etc.) can be placed in a single folder, or it can be organized into subfolders, for example by site or by day.

Each folder of content should contain one metadata spreadsheet (“Metadata.xls” or “Metadata.xlsx”) describing the files (multiple kinds of media can be included).
The same basic folder structure should be used for material not intended for the repository (“Other Uploads”). No metadata spreadsheet is required for these, but the same permission requirements apply, and you should use a metadata spreadsheet or other document to explain the content of the images..

Items marked with a bullet (●) are files, the rest are folders. For printable version, see File:FROGBEAR Repository file folder structure pdf.pdf

Link=http://wiki.ubc.ca/File:File_folder_structure_edit.PNG
FROGBEAR filefolder structure edit.PNG


Step 4: Link to Metadata Spreadsheet

For information on how to fill out the metadata spreadsheet and to download the file, see Documentation:Library:Circle/FROGBEAR Describing your data.

The field that links the metadata in the spreadsheet to the material in the data files is the Filename(s) field. This must contain one or more filenames, including the extension (.jpg, .mp4, .mp3, etc.) but without no folder/path information. The files must be in the same folder as the metadata spreadsheet. Multiple filenames should be separated by double vertical lines (||).

Filenames

It is important that filenames appear in the spreadsheet exactly as they do in the folder, otherwise we will not be able to link the two. In general, we do not recommend trying to give the files meaningful names—the metadata provides the description, so the filename does not need to. Using the default filenames that come from recording devices (e.g., IMG_12345.jpg) is fine. There are two circumstances in which you may need to change filenames manually:

Checkbox

Filename Checklist

No spaces (ex. T1.001.01.Tudi Gong.jpg)
No commas (ex. T1.001.01,Tudi _Gong,jpg)
No Asian characters (ex. T1.001.01.土地公_Tudi_Gong.jpg)
No double underscore (ex. T1.001.01.Tudi__Gong.jpg)

To ensure that filenames are correctly inserted into the spreadsheet, we recommend the following workflow:

  1. Organize data files into folders, following the above guidelines
  2. Copying an empty metadata template spreadsheet (metadata.xls or metadata.xlsx) into each folder
  3. Copying the file list and pasting it into the Filename(s) column of the metadata spreadsheet (see instructions below)
  4. Filling in metadata records, consolidating multiple filenames into single records as needed

Alternatively, in Step 3, instead of copying the list directly into the metadata spreadsheet, you can create a separate spreadsheet and copy the files into it. This may be easier if you anticipate having many records with references to multiple files.

Step 5: Create a File List

Creating a file list is needed in order to transfer the data files to the metadata spreadsheet. Copying filenames individually, or typing them in by hand, is tedious and error-prone. We recommend following one of the two procedures below:

Method 1: FilelistCreator (free software)

FilelistCreator is a free program for creating lists of files. It is available for Windows, Mac, and Linux at http://www.sttmedia.com/filelistcreator-download

  1. Run FilelistCreator by following the instructions on the website above. On Windows, it does not need to be installed, just copied to a convenient location on your hard drive.
  2. Save the attached profile in a convenient location (e.g., the folder with the FilelistCreator program). This profile contains the appropriate settings (set to export only filenames, with no additional information such as file size, folder name, etc.). This can also be found in the “FROGBEAR Documentation and Utilities” folder on OwnCloud.
  3. Run FilelistCreator, and load the profile by choosing from the top menu Settings  Save or Load Settings. In the dialogue that appears, under “Save or Load Settings,” click the “Load file” button, then choose the FROG_Metadata.fls file. This will load the settings and a pop-up window will confirm that they have been loaded. On the settings page, under “Default Settings,” click “Save.” This will save these settings so that they are the default every time you start the program. Close the Settings dialogue.
    Filelistcreator1.png
    Filelistcreator2.png
  4. You are now ready to move file lists into the metadata spreadsheet. Drag the folder of material onto the program window (from Windows Explorer, OS X Finder, etc.). It will fill the Files box with a list of files in that folder. Alternatively, you can use the “Search Folder…” button to choose a folder to add to the list. The top part of the window shows the files to be listed; the bottom Preview area shows the list as it will be formatted. There should be only one column, with the filename, and no folder or filesize information. If other information appears here, reload the FROG_Metadata.fls profile as described above
    Filelistcreator3.png
  5. To copy the list to the metadata template, click the Clipboard button on the bottom right of the FilelistCreator window to copy the list to the clipboard. Then open the template spreadsheet in Excel or another program. Find the filename column and select the first blank cell under the header column. Then paste (ctrl-v on Windows, command-v on Mac). The rows below will fill with the filenames.

Method 2: Copy-and-Paste

If you are not able to run FilelistCreator, you may also copy a list of files from your system directly into the spreadsheet.

In Windows, open the folder in Windows Explorer and select the files whose names you wish to copy, hold down Shift and right-click on the selected filenames, then from the menu that appears choose “Copy as path.” Go to the Filenames column in the spreadsheet and paste. This will paste the complete filenames include the full folder path, e.g., “C:\Users\YourName\Photos\FrogBear\2017-06-15\IMG_12345.jpg”, but the metadata should not include the path, just the filename. To fix this, in your spreadsheet choose a filename, copy the text in the cell, then perform a search-and-replace throughout the spreadsheet for the folder portion (in this case, “C:\Users\YourName\Photos\FrogBear\2017-06-15\”).

In OS X, open the folder in Finder, select the files (CMD-A to select all), copy (CMD-C), then in the first blank cell in the Filenames column paste (CMD-V). This will paste a file list including extensions.

Step 6: Creating Records

Some individual files will become individual records in the repository, but others will be grouped into records describing multiple files, for example multiple images of the same object or multiple audio segments from the same conversation. To do this, in the spreadsheet choose the row with the first filename that will be part of the record and start filling in the metadata. To add more files to the record, insert a separator of two vertical lines (“||”), then cut and paste the filename from another row.

If you find that you end up with gaps between records (blank row between entries), you can follow the instructions on this page to delete the blanks: http://www.techrepublic.com/blog/microsoft-office/a-quick-way-to-delete-blank-rows-in-excel/
https://wiki.ubc.ca/Documentation:Library:Circle/FROGBEAR_Organizing_content_for_upload
wiki.ubc.ca
crawl=CC-MAIN-2022-21
42308
application/http; msgtype=response
sha1:3IAG7PBSFUDNPNJPQYOICD6BINIZZXHZ
<urn:uuid:91e2bcb7-c1a8-4378-b38b-8be4021615f7>
2022-05-19T09:21:22Z
206.87.224.38
text/html
sha1:Y5GAKRMO7QAKUROSTXQ6L7EGX3XJAQCS
<urn:uuid:d1d76757-f9b4-4611-9ad9-8f7847b1fa9f>
https://wiki.ubc.ca/Documentation:Library:Circle/FROGBEAR_Organizing_content_for_upload
response
<urn:uuid:c705bba7-cfc4-4cfd-84fe-f2939ffa8b15>
isPartOf: CC-MAIN-2022-21
publisher: Common Crawl
description: Wide crawl of the web for May 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-253
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/