Meta Robots Tag

One other meta tag worth mentioning is the robots tag. This lets you specify that a particular page should NOT be indexed by a search engine. To keep spiders out, you can simply add a code in your robots tag on each page you don't want indexed.

You do NOT need to use variations of the meta robots tag to help your pages get indexed. They are unnecessary. By default, a crawler will try to index all your web pages and will try to follow links from one page to another.

Most major search engines support the meta robots tag. However, the robots.txt convention of blocking indexing is more efficient, as you don't need to add tags to each and every page. If you do use a robots.txt file , there is no need to also use meta robots tags as at first search engines hit the robots.txt file and according to it index pages.

The meta robots tag also has some extensions offered by particular search engines to prevent indexing of multimedia content.

Other Meta Tags

There are many other meta tags that exist beyond those explained. For example, if you were to view the source code of this web page, you would find "author," "channel" and "date" meta tags. These mean nothing to web wide crawlers such as Googlebot, Scooter etc. They are specifically for an internal search engine used by websites to index its own content.

There are also "Dublin Core" meta tags. The intent is that these can be used for both "internal" search engines and web-wide ones. However, no major web-wide search engine supports these tags.

How about the meta revisit tag? This tag is not recognized by the major search engines as a method of telling them how often to automatically return. They have never supported it.

 In Conclusion

Overall, just remember this. Of all the meta tags you may see out there:

Meta Robots: This tag enjoys full support, but you only need it if you DO NOT want your pages indexed.

Meta Description: This tag enjoys much support, and it is well worth using.

Meta Keywords: This tag is only supported by some major crawlers and probably isn't worth the time to implement.

Meta Everything Else: Any other meta tag you see is ignored by the major crawlers, though they may be used by specialized search engines.