Metadata Editor - project - DML-CZ

quaggahooliganInternet and Web Development

Feb 5, 2013 (4 years and 7 months ago)

140 views

DML
-
CZ

Metadata Editor


Miroslav Bartošek


Masaryk University, Brno

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

2

DML
-
CZ Workflow

1.
Preparation

2.
Scanning

3.
OCR

4.
Metadata harvesting (MR, ZBL)

5.
Integration

6.
Digital Library

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

3

Page

images

Page

images

DML
-
CZ Integration

page

images

OCR

texts

MR/ZBL

refMD

Metadata

Editor

journal

volume

digital

article

content + MD

operators

mathematicians

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

4

Metadata Editor


Metadata Creation & DL Integration


Developed for DML
-
CZ


Web
-
based application


web interface


suite of scripts


files in directories


internal database







http:
//editor.dml.cz

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

5

ME workflow

1.
Input data loading

2.
Articles building

3.
Metadata editing

4.
References processing

5.
Verification

6.
pdf
-
compilation

7.
Export to DML
-
CZ

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

6

ME workflow

1.
Input data loading

2.
Articles building

3.
Metadata editing

4.
References processing

5.
Verification

6.
pdf
-
compilation

7.
Export to DML
-
CZ

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

7

1. Input Data


Scanned documents


19
th

century


1991


printed documents


full workflow


Retro
-
born
-
digital


1992


2007


some digital content & MD


transformation, integration


Born
-
digital


since 2008


on
-
fly new journal issues


integration

main focus

of ME

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

8

1. Input Data


checks


completeness, consistency, page ordering


restructuring


files in hierarchical directories


internal database


internal identifiers


hierarchical structure


serials
:


journal/volume/issue/article


proceedings
:

series/volume/article


monographs
:

collection/book/chapter

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

12

ME workflow

1.
Input data loading

2.
Articles building

3.
Metadata editing

4.
References processing

5.
Verification

6.
pdf
-
compilation

7.
Export to DML
-
CZ

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

13

2. Articles building


pages
-
> articles (issue structure)


initial structure build automatically


metadata from MR/Zbl


article’s start
-
end located in OCR
-
pages


manual check/corrections


visual article editor


page thumbnails (cards on desk)


reshuffling


grouping


green

= article


red



= excluded pages

edit

issue structure

(articles building)

article
1

article
2

pages to be

excluded

click to inspect

page details

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

18

2. Articles building


auxiliary functions


page cloning


page download/upload


page reshuffling within article/journal


page number editing


physical number


logical number


sequential number


issue sections

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

19

ME workflow

1.
Input data loading

2.
Articles building

3.
Metadata editing

4.
References processing

5.
Verification

6.
pdf
-
compilation

7.
Export to DML
-
CZ

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

20

3. Metadata editing


article
descriptive metadata


pre
-
filled with MR/Zbl metadata


manual checking + augmenting


DM editor

1.
page preview

2.
editing form


editing


pre
-
emptive input


non
-
latin
-
1 characters input


cut&paste editing (OCR)

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

23

3. Metadata editing

Journal article
-

metadata elements


Title

(original + English)


Author

(name authority base)


Language


MSC


Summary Language


Article Type

(math, editorial, review, …)


idMR
,
idZBL
,
idJFM


Accessibility


Link


Status

(in progress, completed)

initial

structure

edited

structure

edited

metadata

1

2

3

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

25

3.1 Authority base


Named checked against AB


author’s record


several name forms

(initials, full name, transliterated, pseudonym)


author personal data


consistent names

(even if author’s name written differently)


many different name forms of one author

(still the same author)


search, display


different authors with the same name

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

29

ME workflow

1.
Input data loading

2.
Articles building

3.
Metadata editing

4.
References processing

5.
Verification

6.
pdf
-
compilation

7.
Export to DML
-
CZ

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

30

4. Bibliographical references


Semi
-
automated process


Automat


block of references localized in article OCR
-
text


individual references identified


Manual


OCR errors correction


simple markup (//title//)


Automat


structured reference records


links to MR/ZBL generated

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

32

ME workflow

1.
Input data loading

2.
Articles building

3.
Metadata editing

4.
References processing

5.
Verification

6.
pdf
-
compilation

7.
Export to DML
-
CZ

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

33

5. Verification

MD automated verification


set of tests


missing MD elements


data storage integrity


page ordering


TeX syntax


article language detection


reference markup


article pdf completeness


statistics (work progress)



and other …

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

34

ME workflow

1.
Input data loading

2.
Articles building

3.
Metadata editing

4.
References processing

5.
Verification

6.
pdf
-
compilation

7.
Export to DML
-
CZ

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

35

6. PDF compilation


DML
-
CZ: article
-
oriented delivery


article PDF


composed of page PDFs


2 layers

(page pictures + OCRed full
-
text)


title page

(article identification, full paper citation, persistent
URL, copyright notice)


slate compression


digital signature?

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

36

ME
-

other features


search module


easy and powerful element
-
property
-
relation search


batch update


links


continuation articles


derived works


article review


suggested relevant papers


users/rights management


distribution of work (students x mathematicians)

M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

37

ME
-

implementation

Three
-
tier web application on Nitro (Ruby) framework

System architecture:


M.Bartošek,
10.
-
12.6

2008

Launching DML
-
CZ

38

Live

demonstration
https://www.techylib.com/en/view/quaggahooligan/metadata_editor_project_dml-cz
www.techylib.com
crawl=CC-MAIN-2017-34
46884
application/http; msgtype=response
sha1:5YRYWSCQIQCKFHMDIRINOI53KMVX32DJ
<urn:uuid:8cecb1d8-c382-4068-9e3b-a64ba6545ce5>
2017-08-16T15:13:23Z
54.225.207.13
application/xhtml+xml
sha1:VWC3JXFW77JFOEZV33THO3CAWM5OJIR7
<urn:uuid:bca0815e-2668-49c9-9540-2491c6683adf>
https://www.techylib.com/en/view/quaggahooligan/metadata_editor_project_dml-cz
length
response
<urn:uuid:6cd5ade2-b9a5-42b0-890e-44255caeee9c>
robots: classic
hostname: ip-10-65-97-209.ec2.internal
software: Nutch 1.6 (CC)
isPartOf: CC-MAIN-2017-34
operator: Common Crawl Admin
description: Wide crawl of the web for August 2017
publisher: Common Crawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf