Personally I think it's a good idea to use both the robot meta tag and the robots.txt. If you only want to use one, then use robots.txt, if you can.The reason you need to use a robots.txt file is that most search engine spiders will look for it and if it's not there, some get confused and go away. They can also generate a lot of no file found or page 404 entries in your web log files.
The robots.txt file The robots.txt file as the name suggests is a simple text file that contains instructions for the search engine robot ( spider ) as to what files it's not allowed to access.The file can be created in a simple text editor like notepad then uploaded to the root directory on your web host where your web site resides. Check with your web host that you can use a robots.txt file. It must be called robots.txt in lower case letters only and be saved in the root directory. If you put it anywhere else the spider will not find it. If your web site is at http://www.yourdomain.com then with the robots.txt file in the root directory on your web server space the spider will find it as http://www.yourdomain.com/robots.txt A simple robots.txt file might look something like: User-agent: *
Disallow: /cgi-bin/
Disallow: /forms/
Disallow: /contact.html In the above example Disallow: /cgi-bin/ tells the robot to stay out of the cgi-bin directory and away from your perl scripts etc. The next Disallow tells the robot to stay out of a forms directory where there are form templates and pages stored that we don't want indexed. The final line tells the robot not to index the single web page contact.html. Each directory or file to be disallowed needs a separate Disallow: command line. The User-agent:* means any robot, it is not a wildcard character as one might use in a search query. So you can not use it as Disallow: /*.jpg It's not possible to stop the robot accessing jpg image files with any filename but file extension .jpg. To exclude all robots to the site would require: User-agent: *
Disallow: / There is not an Allow command except that to do this you would use User-agent: *
Disallow: You can also disallow only particular robots by giving the name of the search engine robot in the User-agent command line. User-agent: Marvin
Disallow: / This would allow all robots access except Marvin which happens to be Infoseek.de (Germany). NOTE: Not all robots observe the robots.txt file. The robots meta tag The Robots Meta tag is a simple way to tell a visiting robot if a page should be indexed, or links on the page followed.It has four basic commands, INDEX, NOINDEX, FOLLOW, NOFOLLOW and two additional commands ALL=INDEX,FOLLOW and NONE=NOINDEX,NOFOLLOW. These commands are not case sensitive so can be written in either upper or lower case. Examples would be:
<meta name="robots" content="index,follow"> or <meta name="robots" content="all">
<meta name="robots" content="noindex,follow">
<meta name="robots" content="index,nofollow">
<meta name="robots" content="noindex,nofollow"> or <meta name="robots" content="none"> As a META tag it should be placed in the HEAD section of an HTML page: <html>
<head>
<meta name="robots" content="index,nofollow">
<title>...</title>
<meta name="description" content="This page ....">
</head>
<body> You can generate a robots meta tag using the Meta tag optimization tool | 
