Comments on: Rasmuson Library DVD Browser: Fun with Screen Scraping and Drupal http://journal.code4lib.org/articles/469 Fri, 20 Nov 2015 14:28:57 +0000 hourly 1 http://wordpress.org/?v=4.2.4 By: mark411 http://journal.code4lib.org/articles/469/comment-page-1#comment-2557 Mon, 31 Aug 2009 13:32:07 +0000 http://journal.code4lib.org/?p=469#comment-2557 Hi! Is it possible to retrieve information from covers.hearsay24.com ?

]]>
By: Mark Morlino http://journal.code4lib.org/articles/469/comment-page-1#comment-1301 Mon, 22 Dec 2008 16:50:20 +0000 http://journal.code4lib.org/?p=469#comment-1301 Hi Jonathan,

Thanks for your interest in our article.

We are just matching by title. IMDB and Rotten Tomatoes are both usually pretty good. If there are multiple matches, the first one is very often the correct one. The development and testing was time consuming because when I encountered false positives or false negatives I modified the code to prevent them, and that would require dumping all of the data and starting from the first DVD to make sure the change to the screen scraping code did not affect the processing of any of the other DVDs.

Freecovers.net is the only one that offers a public API. Basically, the API offers the same searching functionality as the web page but returns an XML document rather than an HTML document. So we can parse the XML into a data structure easily and consistently to look for matches. It is considerably easier to code and less error prone than the screen scraping. Unfortunately, the perl module we are using to parse the XML stores the list of possible title matches as a hash, which means we do not maintain the order in which they were sent, and Freecovers appends various information to the titles (to indicate a release, a language, a region, or sometimes a video game by the same name) so there are many DVDs that have cover images available from Freecovers that the program doesn’t because it cannot determine which one to use.

The DVDs in our catalog all have call numbers that begin with “DVD” the rest of the call number is based on the order in which the DVD was cataloged and whether or not it is part of a multiple dvd set. So it is fairly easy for us to search for them in the catalog by number. The browser can just search for the most recent DVD that it knows about and keep processing results until it finds a call number that does not begin with DVD.

I hope this helps.

-Mark

]]>
By: Jonathan Rochkind http://journal.code4lib.org/articles/469/comment-page-1#comment-1298 Sat, 20 Dec 2008 20:09:42 +0000 http://journal.code4lib.org/?p=469#comment-1298 Oh, it also occurs to me that I’m not sure how to identify _which_ bib records in my catalog represent movies. How are you determining that, by call number including ‘DVD’ at your library?

]]>
By: Jonathan Rochkind http://journal.code4lib.org/articles/469/comment-page-1#comment-1297 Sat, 20 Dec 2008 20:06:39 +0000 http://journal.code4lib.org/?p=469#comment-1297 I would love more information about how you retrieve the information from imdb, rottentomatoes, and FreeCovers.net. You are just matching on title? Do you have much trouble with false positives or false negatives matching on title keyword? Do these three services offer any kind of an API, or are you screen-scraping to find content?

]]>
http://journal.code4lib.org/articles/469/feed
journal.code4lib.org
crawl=CC-MAIN-2015-48
7828
application/http; msgtype=response
sha1:Y3UFMQEEXSWCZNZVJ3S7P4T26S76RK6V
<urn:uuid:63b8c438-8717-446a-b304-de55cc240c0f>
2015-11-26T23:11:00Z
152.19.134.41
sha1:74M7XQQLLHWD7XSGMDB53NRQVDOKRVG4
<urn:uuid:0ea9c453-0ec1-4e32-a366-aa5977af4377>
http://journal.code4lib.org/articles/469/feed
response
<urn:uuid:5fef44fd-d3e1-4d2e-84d1-c7b9331201f2>
robots: classic
hostname: ip-10-71-132-137.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2015-48
operator: CommonCrawl Admin
description: Wide crawl of the web for Nov 2015
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf