Techdirt is off for Memorial Day. We'll be back with regularly scheduled posting tomorrow!Hide

News Publishers Want To Change Robots.txt; Want To Make Sure Their Content Is Less Useful

from the deep-misunderstandings dept

Following on the speech given earlier this month by the head of the Associated Press, where it was made clear that the AP and news organizations still think that they can be gatekeepers of news, a bunch of publishers along with the AP are now trying to revise robots.txt so that they can hide content on a more selective level. Now, it is true that robots.txt can be rather broad in its sweep. But it's rather telling that it's the publishers who banded together and are telling search engines what changes are needed, rather than working with the search engines to come up with a reasonable solution. In the meantime, there really are some simple solutions if you don't want content indexed by search engines -- but we've yet to fully understand why publishers are so upset that Google, Yahoo and others are sending them so much traffic in the first place.

Reader Comments (rss)

(Flattened / Threaded)

  1. icon
    JS Beckerist (profile), Nov 30th, 2007 @ 10:25am

    Simple solutions:

    A: Put the content you don't want crawled on a different server and blacklist all known bots from connecting to that server.
    B: Use a login script.

    reply to this | link to this | view in thread ]

  2. identicon
    Johan, Nov 30th, 2007 @ 11:12am

    I've said it before...

    and I'll say it again - Google, Yahoo, and the others need to completely remove news publishers that complain from their listings. Poof! Gone! Problem solved!

    Unfortunately for the publishers no more listings equals no more traffic. Bye-bye AP, bye-bye archaic newspapers that can't make the move into the 21st century.

    If the search engines would do this the news publishers would quickly see the value of the listings.

    reply to this | link to this | view in thread ]

  3. identicon
    data64, Nov 30th, 2007 @ 11:18am

    More info on proposed changes ?

    Wish there was more information on the proposed changes. The Acap website is also not very clear. There is some information on their reasoning in their FAQs.

    update:
    A pdf document with the proposed changes.

    reply to this | link to this | view in thread ]

  4. identicon
    Carme, Nov 30th, 2007 @ 12:41pm

    Why it does make sense

    The fact that there are no business models that make use of this fine-grained control now doesn't mean there won't be such in the future. They are smartly making sure the technology is already in place so they are free to experiment with such business models, and if they find a successful one stick with it. While it's possible they won't find a successful model, they certainly won't find it without the technology to back it up.

    You may believe that no such model exists, but bear in mind that a failed experiment is as important as a successful one. For example, if the music industry started off selling unprotected MP3, they would probably get scared of the inevitable piracy, however minor, that would ensue and fold out of the digital media market. Now that they tried various DRM schemes - and failed miserably - they are much more committed to selling unprotected digital media and much more pragmatic as to what is an acceptable level of piracy. Similarly, while such an experimentation period may not help the publishing industry actually find a better Internet strategy, it will certainly make sure they are more informed and committed when they do pick their strategy.

    And anyway, having more control can never be bad. Since the courts seem to agree the publishers do have a say as to how their content is used, they might as well use that power to put in place the technology to enforce control. Then they can decide to just continue with the status quo, and change it when they find it beneficial.

    reply to this | link to this | view in thread ]

  5. identicon
    T, Dec 3rd, 2007 @ 9:45am

    Be patient. ACAP is a proposal and voluntary. Those that that would need to use it will decide if it has value.

    reply to this | link to this | view in thread ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
Insider Shop - Show Your Support!

Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.
https://www.techdirt.com/articles/20071130/005506.shtml
www.techdirt.com
crawl=CC-MAIN-2016-22
52423
application/http; msgtype=response
sha1:UPJ2DQQWGY2JKU6CWIKSS6A4FPJJCRAD
<urn:uuid:fcd7d098-9ce0-434e-b489-c2d918312255>
2016-05-30T09:21:53Z
104.25.105.28
sha1:EWHXEQK2J5PJEYYVO4FXAGJQ5DTPR7JB
<urn:uuid:f1f42168-5db0-43e7-ae35-16dbfeebbf22>
https://www.techdirt.com/articles/20071130/005506.shtml
length
response
<urn:uuid:f1e90365-366c-42a5-8dbb-70d7b08b3cae>
robots: classic
hostname: ip-10-185-217-139.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2016-22
operator: CommonCrawl Admin
description: Wide crawl of the web for May 2016
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf