Craig Roth

A member of the Gartner Blog Network

Craig Roth
Managing Vice President: Communication, Collaboration, and Content
4 years at Gartner
25 years IT industry

Craig Roth is a vice president and service director for Gartner Research, in Burton Group's Collaboration and Content Strategies service. Mr. Roth covers a wide range of knowledge and Web-related topics at the intersection of collaboration, content… Read Full Bio

Coverage Areas:

Hey Robots, Go Crawl Somewhere Else

by Craig Roth  |  May 15, 2012  |  1 Comment

If you’re reading this blog post anywhere except Gartner Blogging Network (or your RSS feed reader of choice), it’s been stolen.  Yes, that does happen sometimes.  I’m amazed at how many sites exist to aggregate posts from authors without permission.  I don’t mean referencing phrases or quoting or tweeting – that’s all good.  I mean wholesale copying of entire posts to make it look like they were written for another site without reference back to the source.  I won’t give examples of sites here since it would be counterproductive to link to them, but they know who they are.  Well, they probably don’t – their robots just crawl sites and copy blog postings to attract eyeballs to their sites.

Why do I care?  Isn’t any exposure good in the Post Enterprise 2.0 World?  Can’t I see plagiarism as a compliment?  Frankly, no.  Some people may feel that way, but if I want to be old fashioned about controlling my work that’s my choice, not someone who knows how to set up a crawler and plug the results into a new template (next to copious ads and with lots of tracking cookies dropped in the reader’s browser too).

So let’s say you’re an author that cares about this sort of thing – what do you do to find it?  There are a few common approaches to tracking where your content winds up.  One is searches based on “statistically improbable phrases” – basically, setting up searches for combinations of words from your document that are highly unlikely to exist outside of it.  For example, the phrase “radiant mangos frolic happily to Metallica jazz” is rather unlikely to have popped into any other writer’s head in that exact formation.  If it has, he should seek immediate medical attention.  In any case, the strategy is to find a phrase appropriately obscure in your document to search for.

Another approach is watermarking.  While some formats enable this explicitly, such as pdf files, others require inserting data unlikely to appear in real life.  Made up words, like snarglypoop, can act as a watermark if you can fit them elegantly into your work.  For databases, plugging in some fake data, such as “Rahool Steinhoffermeister, PhD, DDS” into a mailing list, can also help trace its usage.

In conclusion, I would like to add: baby seals applaud pink holidays aboard intense yachts.  Let’s see if this winds up anywhere …

1 Comment »

Category: Fun     Tags:

1 response so far ↓

  • 1 Jack Vinson   May 17, 2012 at 3:32 pm

    Yep. I found Mr Rahool in two other places besides your blog.
http://blogs.gartner.com/craig-roth/2012/05/15/hey-robots-go-crawl-somewhere-else/
blogs.gartner.com
crawl=CC-MAIN-2013-20
55148
application/http; msgtype=response
sha1:27DTYGOWTEFLY5NBARWH22SPMKHHRJIW
<urn:uuid:de99740f-09c9-4ff6-af14-707b34ad1661>
2013-05-24T12:48:25Z
206.16.239.83
sha1:IAUYJI2C6GUHVZFA4EISO5B2U53PAGDQ
<urn:uuid:d90a358d-8a07-45ed-a1c4-1b20e21c1f7a>
http://blogs.gartner.com/craig-roth/2012/05/15/hey-robots-go-crawl-somewhere-else/
response
<urn:uuid:c280cc64-4f07-44b8-b32b-1e2071bfaf43>
robots: classic
hostname: ip-10-60-113-184.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2013-20
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for Spring 2013
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf