
About Features Downloads Getting Started Documentation Events Support GitHub

Site Tools


indexing:tracking_record_changes

Tracking Record Changes

IMPORTANT: This page refers to features that were added in VuFind 1.1. If you are using an earlier version, you will have to upgrade.

Introduction

For certain types of functionality, VuFind needs to be aware not only of the content of the records in its index, but also the times at which these records were first indexed and most recently changed. By understanding the history of its own records, VuFind can provide RSS feeds that show recent changes, and it can export its contents using the popular OAI-PMH protocol.

Keeping track of the information needed to support these features adds some overhead – more data in your MySQL database, and slower indexing times. Because not everyone needs the features, they are disabled by default. This page explains how to activate them when you need them.

Enabling Change Tracking

The most important thing that you need to do in order to enable VuFind's change tracking is to edit the local MARC mappings file and uncomment the lines that fill in the “first_indexed” and “last_indexed” fields of your Solr index. If you are using VuFind 2.x or higher, this usually consists of:

1. Copy import/marc_local.properties into the import subdirectory of your local_settings_directory (usually $VUFIND_HOME/local/import), if you don't already have a local copy of marc_local.properties.

2. Uncomment the *_indexed lines in the local copy of marc_local.properties. The example lines assume that your record IDs come from the 001 field; if you use IDs in a different place, change 001 to an appropriate pattern (e.g. 999c for Koha).

If you are indexing non-MARC records, some change tracking facilities also exist in the XSLT index tool. Depending on your data source, you may need to customize your XSLT and/or properties file to implement support.

After making the necessary configuration changes, you will need to reindex all of your records. This will take a long time, as it will be generating a database so that VuFind can correctly populate these fields in the future. “first_indexed” contains the time that the record was first indexed by VuFind, while “last_indexed” contains the time that the record was most recently changed. For consistency and appropriate OAI-PMH behavior, it is STRONGLY RECOMMENDED that all times be stored in UTC; VuFind's provided tools will already do this, but if you build your own custom indexer, you should conform to this standard. VuFind stores information in its database so that even if you upgrade Solr and destroy your entire index, it will still be able to rebuild the correct “first_indexed” and “last_indexed” values. In the future, when you reindex existing records, the process should be faster than the first time around, since for records that have not actually changed, VuFind will not have to do much extra database work.

A Note on Memory Usage

SolrMarc may require more memory in order to update the database and the Solr index at the same time. If you get error messages about running out of heap space during the index process, try editing import-marc.sh or import-marc.bat (depending on your platform) to change the Java memory settings. See the performance page for more details on Java memory tuning.

OAI-PMH Server Functionality

Location

VuFind's OAI-PMH server can be found at http://[your_server]/[your_vufind_base_path]/OAI/Server.

Requirements

The server will not return any records unless the first_indexed and last_indexed fields of the Solr index are properly filled in (see Enabling Change Tracking, above).

Configuration

Several aspects of the server (including ID prefixes, administrative email addresses and set behavior) can be configured through the [OAI] section of config.ini. See the comments in that file for details on how the settings work.

Metadata Formats

Out of the box, VuFind's OAI-PMH server supports basic oai_dc in a minimalistic way and is capable of exporting MARC records in XML format.

To add additional metadata formats, simply make changes to the getXML() method of the record driver(s) you are using.

Important: The server is currently coded to assume that all supported metadata formats are supported by all records in the index. In order to display only a subset of records based on the metadata format requested, it would be necessary to make some minor adjustments to the code and the underlying Solr index. If there is any demand for this feature, please open a JIRA ticket.

Why use OAI-PMH in VuFind?

Not everyone is going to want to expose their VuFind records through OAI-PMH, but in some cases it may be convenient – if VuFind is how you present your holdings to the world, it makes sense to offer the data to automated agents as well as to humans. If you have already gone through the complicated work of getting data out of your ILS and into VuFind, using OAI-PMH may be a way of saving you from having to repeat that effort to send records to other places.

If you have found this feature useful, feel free to add comments and advice here!

More on OAI-PMH

For a general introduction to the OAI-PMH protocol, see this tutorial.

For more on OAI-PMH, see the OAI-PMH Server and Harvest Functionality video.

Enhanced RSS Functionality

When record change tracking is enabled, RSS feeds will be more effective. Without tracking, RSS dates are based on publication dates, which may or may not correspond with the dates that items were added to your collection. With change tracking turned on, RSS feeds will be sorted by index date and will display index dates within the feed. This means that patrons can truly track searches in order to be notified of new additions within their chosen areas of interest.

Note that you can customize some of the RSS behavior in VuFind – see the [RSS] section of searches.ini.

Solr-driven New Items

:!: This feature was added in VuFind 2.3.

By default, VuFind tries to use the ILS driver to power the “new items” search. However, not all ILS drivers support this functionality. If you turn on record change tracking, you can modify the method setting to “solr” in the [NewItem] section of searches.ini to gain new item searches without an ILS dependency.

Frequently Asked Questions

This section contains answers to frequently asked questions about the technical details of the change-tracking implementation.

If I reindex my existing records, will VuFind think that they are new?

No, the change tracking system will only update dates in its internal database if it encounters a record ID it has never seen before, or if the record itself has changed. If you index the same MARC file over and over again, that record will only be recorded in the database once. However, if you edit the MARC record and export a new version, when you index the changed record, VuFind will update the “last indexed” value to reflect that something is different.

This behavior is dependent on records containing last-changed dates. In the case of MARC records, VuFind checks the 005 and 008 fields for modification dates. Note that if no dates are found, VuFind will always assume that the record has changed as a precaution; thus, it is important to ensure that your ILS is including one of these fields in exported records if you want reliable change tracking. In the case of XSLT records, the custom PHP methods used by the XSLT expect a modification date parameter, and you should be sure to extract it from an appropriate place in your XML to ensure consistent behavior.

What happens when records are deleted?

If you remove records from your ILS, you should inform VuFind about the deleted records so it can update its change tracking database. This enables the system to correctly report deleted records through OAI-PMH, and it also allows the system to treat records as new if they later are added again after being previously deleted. You can use the util/deletes command line utility for this purpose.

For systems that have no easy way of reporting deleted records, a common solution is to simply delete VuFind's index, and then reindex from a fresh full export. If you do this, the deleted records will be removed from VuFind's index correctly, so normal search behavior will work as expected. However, the change tracking system will not be aware that records have been removed, and the now-missing records will not be reported as deleted through OAI-PMH (or any other feature that might rely on delete tracking functionality). This may or may not be a problem for you, depending on your use case.

indexing/tracking_record_changes.txt · Last modified: 2021/08/03 13:59 by demiankatz
https://vufind.org/wiki/indexing:tracking_record_changes?do=
vufind.org
crawl=CC-MAIN-2021-49
31897
application/http; msgtype=response
sha1:3CWMVN5UJAEPCOBUJ3R7DHPDUO4G3OOP
<urn:uuid:263fdcab-4137-4c84-9caa-90f2dd1b7653>
2021-12-04T17:24:52Z
153.104.6.41
text/html
sha1:T7VG2PU4BPBK2LISO2OGMOMEJRH5M3XO
<urn:uuid:9df0ee78-9d28-42ae-9511-5b4b0aa8b93a>
https://vufind.org/wiki/indexing:tracking_record_changes?do=
response
<urn:uuid:1e280cd8-f7cb-4bf2-9d7c-df1af45a789d>
isPartOf: CC-MAIN-2021-49
publisher: Common Crawl
description: Wide crawl of the web for November/December 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-6
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/