Disadvantages of the robots.txt file

Careless handling of directory and filenames can lead hackers to snoop around your site by studying the robots.txt file, as you sometimes may also list filenames and directories that have classified content. This is not a serious issue as deploying some effective security checks to the content in question can take care of it. For example if you have your traffic log on your site on a URL such as www.domain.com/stats/index.htm which you do not want robots to index, then you would have to add a command to your robots.txt file. As an example:

User-agent: *
Disallow: /stats/

However, it is easy for a snooper to guess what you are trying to hide and simply typing the URL www.domain.com/stats in his browser would enable access to the same. This calls for one of the following remedies -
Change file names:

  Change the stats filename from index.htm to something different, such as stats-    new.htm so that your stats URL now becomes www.domain.com/stats/stats-new.htm

  Place a simple text file containing the text, “Sorry you are not authorized to view this    page”, and save it as index.htm in your /stats/directory.

This way the snooper cannot guess your actual filename and get to your banned content.

Use login passwords:

  Password-protect the sensitive content listed in your robots.txt file.