Wikipedia:Controlling search engine indexing

There are a variety of ways in which Wikipedia attempts to control search engine indexing, commonly termed "noindexing" on Wikipedia. The default behavior is that pages are indexed. All of the methods rely on using the noindex HTML meta tag, which tells search engines not to index certain pages. Respecting the tag, especially in terms of removing already indexed content, is up to the individual search engine, and in theory the tag may be ignored entirely.

The control methods are:

  1. Controlling an entire namespace, via MediaWiki software settings
  2. Controlling classes of pages, via MediaWiki:Robots.txt (Wikipedia's Robots.txt file)
  3. Controlling individual pages by adding the __NOINDEX__ magic word into them, either directly or using the {{NOINDEX}} template, however articles are a special case, see #Indexing of articles ("mainspace").
  4. Controlling multiple pages by adding the __NOINDEX__ magic word into standard templates used in certain situations (same caveat as in the third point).

Indexing of articles ("mainspace")Edit

Articles older than 90 days are automatically indexed.[1] The __NOINDEX__ magic word and the {{NOINDEX}} template do not work on them. Articles younger than 90 days are not indexed, unless they have been patrolled and do not have the {{NOINDEX}} template on them (or a template that transcludes the {{NOINDEX}} template, such as the speedy deletion templates).[2][3][4] The __NOINDEX__ magic word does not work on them. Note that &action=info will incorrectly state that they are indexed.[5] Articles that include the {{NOINDEX}} template are listed at Category:Noindexed articles.

This patrolling may be done automatically by the software, as in the case of articles created by editors with the autopatrolled user right, or by another editor with the new page reviewer user right (not to be confused with the pending changes reviewer user right).

Other namespaces and robots.txtEdit

Namespace controlEdit

On English Wikipedia the entire User:[6] namespace, User talk:, Draft: and Draft talk: namespaces are automatically noindexed via a software setting.[7]

At the same time, __NOINDEX__ and __INDEX__ are disabled, in addition to article space, on the Draft namespace, and the Draft talk namespace; they have no effect there.[8]

Robots.txt noindexingEdit

MediaWiki:Robots.txt forbids analytic tools from visiting sensitive or potentially sensitive types of pages, primarily in the Wikipedia namespace – for example deletion debates. A side effect of not visiting is normally that a page cannot be indexed. Where possible, you should in addition use __NOINDEX__ for those pages.

NOINDEX magic wordEdit

Individual pagesEdit

Individual pages can be noindexed by adding the __NOINDEX__ magic word into that page, either directly or using the {{NOINDEX}} template. As explained above, this magic word doesn't work in mainspace (on articles).

Pages with the keyword are listed in Category:Noindexed pages.[9]

Standard template noindexingEdit

Some standard templates include the __NOINDEX__ keyword, thereby noindexing pages to which the templates are applied. Such templates should be listed in Category:Wikipedia templates which apply NOINDEX.

Biographies of Living Persons talkpage noindexingEdit

The templates {{BLP}} and {{BLP others}} include the {{NOINDEX}} parameter. The {{BLP}} template is added automatically by the {{WikiProject Biography}} talkpage template, if given the parameter |living=yes; see the documentation of that template for more details. Pages using these templates are automatically categorised in Category:Biography articles of living people.

Other templatesEdit

These templates include {{NOINDEX}}:

See also Category:Wikipedia templates which apply NOINDEX.

  • {{Uw-userspacenoindex}} provides a user warning message for inappropriate use of userspace which required noindexing.

INDEX magic wordEdit

Individual pagesEdit

Individual pages can override namespace noindexing by adding the __INDEX__ magic word into that page, either directly or using the {{INDEX}} template. Such pages appear in Category:Indexed pages. However, INDEX does not override noindexing via MediaWiki:Robots.txt.[10] As explained above, this magic word doesn't work in mainspace (on articles).

The ability to add the INDEX magic word to user spaces (User:, User talk:) has been restricted by an edit filter to extended confirmed users following a community discussion.[11]

Nofollow HTML attributeEdit

Since 2007, all links to other websites from English Wikipedia have the nofollow HTML attribute set.[12] This means that on pages that are indexed by search engines, any links found by a search engine on those pages should not influence the link target's ranking in the search engine's index.

Past discussionsEdit

Namespace discussionsEdit

Individual template discussionsEdit

Current issuesEdit

  • T26169 / phab:T24251 – "Create an __NOINDEX__ equivalent to prevent indexing by internal search engine"

See alsoEdit

NotesEdit

  1. ^ 2017 switch from 30 to 90 days
  2. ^ T147544
  3. ^ PageTriage source code
  4. ^ Value of $wgRCMaxAge on WMF wikis
  5. ^ See T157747
  6. ^ Decided at Wikipedia:Village pump (proposals)/Archive 126#Userpage drafts shown in search engines, implemented at phab:T104797.
  7. ^ This is $wgNamespaceRobotPolicies. See Wikimedia's $wgNamespaceRobotPolicies setting for enwiki
  8. ^ This is controlled by the MediaWiki software setting $wgExemptFromUserRobotsControl. On other projects, the exempt namespaces are the same as $wgContentNamespaces, which is set to main space on almost all Wikimedia projects – see here and here.
  9. ^ The listing is done by MediaWiki tracking the keyword. The category name is determined by MediaWiki:Noindex-category.
  10. ^ It does override mw:Manual:$wgArticleRobotPolicies, but on English Wikipedia this is only used for two pages anyway: Wikimedia's $wgArticleRobotPolicies setting for enwiki
  11. ^ Special:PermaLink/862856598#Prevent_new_users_from_allowing_search_engine_indexing_of_user_pages
  12. ^ Controlled by $wgNoFollowLinks, set to true in Wikimedia's settings file
https://en.m.wikipedia.org/wiki/Wikipedia:Controlling_search_engine_indexing
en.m.wikipedia.org
crawl=CC-MAIN-2022-05
63680
application/http; msgtype=response
sha1:4AMVUNY4P5K4LEL3IDRGKHZKYOKSTNTB
<urn:uuid:3f4a203e-3507-4865-a366-17b252970737>
2022-01-22T00:10:35Z
208.80.154.224
text/html
sha1:LPJMMY756URKAX5P6CZ5YNOUNQRPKXIF
<urn:uuid:3fefbe55-5e88-4e5e-aee6-b730107c444a>
https://en.m.wikipedia.org/wiki/Wikipedia:Controlling_search_engine_indexing
response
<urn:uuid:881bbd6f-3cac-4763-9018-846afc0d837d>
isPartOf: CC-MAIN-2022-05
publisher: Common Crawl
description: Wide crawl of the web for January 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-84
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/