Robots tag

Home

Robots tag

Custom Search

The metatag <robots> is needed to indicate which page can be indexed and which one can not.
If you want your site to be completely indexed, you can do it using this tag. But don't expect them to do it with obedience.
Aim at the contents and remember to create a sitemap.
This tag is not followed by all the spiders: some of them, for some reason, ignore it completely.
In any case, teach the spider what to do at least on the home page and the sitemap.

Some examples:

<meta name="robots" content="index,follow">
<meta name="robots" content="noindex,follow">
<meta name="robots" content="index,nofollow">
<meta name="robots" content="noindex,nofollow">

Meta robots

index it tells the robots that this page can be indexed
noindex using this tag, you tell the robots not to index the page. Therefore this page will not appear in the search engines
follow it tell the robots to follow the links from this page
nofollow it tells the robots not to follow the links and not to index the other pages

The tag must be inserted in the <HEAD> content of the page, between <head> and </head>.

Other robots metatags

A search engine adds the content of indexed pages to its database. The saved content is called CACHE VERSION. If you don't want your content to be saved in the database, use this tag:

<meta name="robots" content="noarchive">

This tag will not prevent your page from being indexed, but the content will not be saved.

Another procedure can be to communicate it only to a certain spider:

<meta name="googlebot" content="noarchive">

It will allow other search engines, but Google, to archive data.

A strange tag

<meta name="revisit-after" content="15days">

It is a tag that no robot follows.
You can insert it, but the spider will come back when it wants and not when you tell it to. Tags:robot meta robots tag

Bookmark this page

< Prev		Next >

[ Back ]

The ultimate search engine would basically understand everything in the world and it would always give you the right thing. And we're a long, long way from that.

Larry Page

Meta robots

Other robots metatags

A strange tag

Content Links

RSS Newsfeed