Seek and Hide Irritant

The move to Canada has meant a near constant immersion in new information: new resources, Web sites, facts about people. The fantastic thing about librarians (a.k.a information professionals) is that, if you are tool-oriented, your tools are often portable from one environment to another. For the most part, that has been true here. I spent a bit of time last week trying to hone some search tweaks that would enable me to search a variety of primary law sources. You can imagine how I felt when one of them blocked my search!

Now, I don’t take it personally.  The site maintainers have blocked all robot activity, to stop Google and other engines from indexing the content that the maintainers have created.  How?  By using the noindex, nofollow, and noarchive attributes (here’s an official Google blog posting about the exclusion protocol).  But since the goal of this and similar sites is public access to information, you would think that they would LIKE to have major engines finding their content.

No.  And one day I’ll find out why – when I’m no longer the new kid on the street – because it’s a weird twist.  I guess I expected that when there was a large and otherwise-public data store available (you can go to the site and search it directly), the content providers would be happy to have remote searches executed on the site.

So for now I am limited to having this particular site as a custom Firefox search (using the Opensearch plugin) and cannot use tools like Rollyo to aggregate the site into a larger search.

    David Whelan

    I improve information access and lead information teams. My books on finding information and managing it and practicing law using cloud computing reflect my interest in information management, technology, law practice, and legal research. I've been a library director in Canada and the US, as well as directing the American Bar Association's Legal Technology Resource Center. I speak and write frequently on information, technology, law library, and law practice issues.
https://ofaolain.com/blog/2007/11/10/seek-and-hide-irritant/
ofaolain.com
crawl=CC-MAIN-2018-13
51563
application/http; msgtype=response
sha1:3R5MIULOMMSHVE3IETWFYPGD75WZ2GN2
<urn:uuid:1e4e7352-ea59-425b-b2d7-09883ba2026f>
2018-03-20T08:13:52Z
37.60.225.118
text/html
sha1:4F4YMQXGSW4MB7T37MZG37KSPHEQC4JP
<urn:uuid:f95cf0bc-0c1b-4dea-8835-29aac1350dc1>
https://ofaolain.com/blog/2007/11/10/seek-and-hide-irritant/
response
<urn:uuid:5bf2f34c-fafc-4c7a-bfc7-38bf7c779487>
robots: classic
hostname: ip-10-155-52-209.ec2.internal
software: Nutch 1.6 (CC)
isPartOf: CC-MAIN-2018-13
operator: Common Crawl Admin
description: Wide crawl of the web for March 2018
publisher: Common Crawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf