Robots meta tags noindex nofollow

Robots meta tags are special species of meta tags. The robots tag tells an internet robot to perform or not perform certain operation on a page. That is, the robots meta tag can for example tell an incoming robot not to index the content of a page. The tag can also tell a robot whether is should scan the page for links to follow.

In general, a meta tag provides the browser, let it be the user interface, a robot, indexing spider, or a server, some top-level information about the content of a page. This can be for example information about the language the website is written in or the description of it, or robot behavior instructions in this case.

What is the important implication of the robots noindex, nofollow meta tag?

Website popularity...

If your link is on a web page that uses this robots meta tag, your link would not be counted towards your website’s link popularity.

Search engine penalization...

The NOINDEX and NOFOLLOW tags are often used on a links page or directory page. In this case, search engine robots will not index (NOINDEX) or follow (NOFOLLOW) the links on that page. There is a reason for this. Some search engines penalize a website ranking if they find too many links or too many terms on a page with no substantial content. That is why some webmasters specify these tags in their code.

Where can I find the NOINDEX, NOFOLLOW robots meta tags?

In general, meta tags are placed in the header of a web page. The noindex and nofollow meta tags are no exception to this rule. It is the code between the <HEAD> and </HEAD> tags.

You would find this code in the source code of the web page near the top of the source code.

Are the robots tags required?

The answer is no. If you do not include the robots tag at your website, it will be equivalent to the robots directives of index, follow or all. These are the default behavior of indexing spiders.

Syntax of robots tags

The general syntax of the robots tag is the following

The content="robots-terms" is a comma separated list used in the robots meta tag that may contain one or more of the following keywords: none, noindex, nofollow, all, index, and follow.

Note, the keywords are not case sensitive.

An example would be:

Are there any other robots tags (keywords) besides the NOINDEX and NOFOLLOW?

Yes, the table below provides a quick summary:

NOINDEX tag tells a robot not to index a specific page
INDEX tag tells a robot to index a specific page
NOFOLLOW tag tells a robot not to follow the links on a specific page
FOLLOW tag tells a robot to follow the links on a specific page
NOARCHIVE tag tells a robot not to store a cached copy of the page
NOSNIPPET tag tells a Google robot not to show a snippet (description) under your Google listing, it will also not show a cached link in the search results
NONE this tells the robots to ignore this page (this is equivalent to noindex, nofollow)
ALL this means no restrictions on indexing the page, or following links from the page to determine pages to index (this is equivalent to index, follow)

What is important to keep in mind when using the noindex, nofollow robots tag?

Robots can ignore...

One important consideration when using the noindex, nofollow robots tag is that web robots can ignore the tag.

Especially malware robots and email address harvesters are known to ignore them.

Applies to links on the page only...

The NOFOLLOW directive only applies to links on the page where it is. The tag does not prevent robots from finding the page through other venues. It is possible that a robot can find the same links on some other page without the NOFOLLOW tag. This can be for example from some other website.

Where can I find more information about the robots meta tags?

The META tag is also described in the HTML 4.01 specification, Appendix B.4.1. You might be interested in these articles too:

How to make money with AdSense? (AdSense make money.)
How to increase Google crawl rate
How to find out the last time Google crawled and indexed a website
How to create a Google sitemap
How to add URL to Google

You are also welcome to ask questions in our SEO discussion forum.

Internet
IT

Discuss this article or this topic in our discussion forum:

(The table bellow shows a list of 8 most recent topics posted in our discussion forum. Visit our discussion forum to see more. It is possible the links below are not related to this page, but you can be certain you will find related posts in the discussion forum. You can post one yourself too.)

Email this article to a friend:

How can I link to this web page?

It is easy, just include the code provided below into your HTML code.

<a href="http://www.maxi-pedia.com/robots+meta+tags+noindex+nofollow" title="www.Maxi-Pedia.com: Robots meta tags noindex nofollow" target="_blank">Robots meta tags noindex nofollow</a>

Bookmark this article with