
About Features Downloads Getting Started Documentation Events Support

Site Tools


indexing:solrmarc

SolrMarc

SolrMarc is used to import MARC metadata to the “biblio” index on Solr used by VuFind. See Solr Index Schema for notes on the layout of the index.

Upgrading SolrMarc

There should rarely be a need to do so, but if you wish, you can update to the latest version of the tool, available here. Instructions on setting up SolrMarc with VuFind are available here.

Customizing Import Mappings

The import process is controlled by the settings in the import/marc.properties file under your VuFind installation directory. The default settings should be fine for a first-time user, but if you want to change or expand the set of MARC fields that are used to build VuFind's search indexes, you can edit this file to make adjustments. See the SolrMarc documentation for details on how this works.

As of VuFind 1.0RC2, a second properties file is also available called import/marc_local.properties. Any lines added to this file will override the equivalent settings in marc.properties. Use of this file is optional, but it is an easy way to separate your local customizations from the default settings packaged with VuFind. For more details, see local MARC mappings.

You can use dynamic_field_suffixes as part of your custom field names in your marc_local.properties file. This will enable you to add fields to the marc_local.properties file without having to modify schema.xml or restart Solr. To do this, name the custom field with the appropriate suffix for the data type you need. Otherwise, be sure to update schema.xml to define the custom fields, and also restart Solr.

Customizing Translation Maps

One of the features of SolrMarc is the ability to translate values found in MARC into different strings using translation map files (i.e. language_map.properties). See the SolrMarc documentation for details on how to specify a translation map in the marc.properties file.

Starting with VuFind 1.0RC2, translation maps are found in the import/translation_maps directory.

Prior to RC2, translation maps are embedded in the SolrMarc .jar file, making them more difficult (but certainly not impossible) to modify.

Customizing Format Determination

One of the most commonly-requested VuFind customizations involves changing the way record formats are assigned.

By default, formats are generated using the getFormat function built into SolrMarc. However, the logic used by getFormat is also replicated in a BeanShell script packaged with VuFind. If you want to customize the behavior, here are the steps to follow:

  1. Edit import/index_scripts/format.bsh to customize the behavior as needed. BeanShell borrows its syntax and libraries from Java, so the code should look familiar to many developers. The MARC access is accomplished with the MARC4J library.
  2. If your new custom script returns different values than the old script, or if you want to change the way the existing values are mapped into your index, edit the translation map in import/translation_maps/format_map.properties.
  3. Uncomment the format line (format = script(format.bsh), getFormat, format_map.properties) in import/marc_local.properties to ensure that VuFind imports using the custom BeanShell version of getFormat instead of the built-in SolrMarc version.

Custom Indexing Functions

Compiled Custom Functions

Sometimes, it is necessary to perform special data manipulation beyond the capabilities of the built-in SolrMarc functions. See this page for details on writing custom Java code to extend the capabilities of SolrMarc.

BeanShell Scripts

Starting with VuFind 1.0RC2, it is also possible to write custom indexing functions as BeanShell scripts. This allows you to extend SolrMarc without having to rebuild the entire Java package. You can simply add scripts to the import/scripts directory and call them from your marc.properties file; here's an example to show the syntax:

format = script(format.bsh), getFormat, format_map.properties

SolrMarc Mixins

Starting with VuFind 1.4, you can also write compiled Java mixin objects to define custom functionality. These should offer better performance than BeanShell, though they are slightly more complex to implement. See the README in the mixin development kit at the SolrMarc download page for more details.

Customizing Record IDs

It is sometimes useful to add a prefix to a record ID (for example, if you are importing numeric IDs from multiple systems and want to prevent collisions). This can be achieved through a regular expression trick in import/marc_local.properties:

id = 001, (pattern_map.id_prefix), first
pattern_map.id_prefix.pattern_0 = (.+)=>bib_$1

(just replace “bib_” in the second line with the prefix you desire).

Here is the original thread from which this trick was derived.

Pre-Processing Records

It may occasionally be useful to manipulate records after you export them from your ILS but before you load them into SolrMarc. See the Code4Lib Working with MARC page for some tools to help you with MARC manipulation.

indexing/solrmarc.txt · Last modified: 2015/12/14 11:57 by demiankatz
https://vufind.org/wiki/indexing:solrmarc
vufind.org
crawl=CC-MAIN-2016-07
23425
application/http; msgtype=response
sha1:RCYRPEVCLORUTPO4J455EM42KFKLMFEX
<urn:uuid:2be84598-3c59-4e81-8fc0-48cdf437f20d>
2016-02-07T17:06:35Z
153.104.4.76
sha1:RBO3K3ZXQOO5SPWBOPYVEJBRO3JOC4YV
<urn:uuid:4ca5dc1a-0ba3-4025-913a-91fb19c48540>
https://vufind.org/wiki/indexing:solrmarc
length
response
<urn:uuid:6c4f1a97-395e-4456-a066-0e80a73904d7>
robots: classic
hostname: ip-10-236-182-209.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2016-07
operator: CommonCrawl Admin
description: Wide crawl of the web for February 2016
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf