The robot meta tag and the robots.txt file are used to advise search engine robots on what areas of your site you want them to access.There are literally hundreds of search engine spiders, robot's, bots, what ever term you prefer. Many of the robot's are country specific so if your site does not exist within a domain for that country you will not see it. Some global robot's also have local country variants. There are about 200 main robot's, but most web sites will only ever see just a few of them from the major search engine's. For the site owner & optimizer robots are welcome because they collect information about web pages to become indexed by the search engine. We can then start down that frequently long road toward recognition & placement in a top ranking of our chosen search engine's. To a web server and its administrator the robot is frequently not a friend but a foe, an unwanted nuisance that consumes valuable resources. It's one of the main reasons for wanting to control access of a robot. When they come calling they can make thousands of page requests that can tie up processors, slow down access times to the server for its customers and also consume valuable bandwidth. Of course on a modern well-designed & managed web server, the impact is minimal to its performance, but none the less it's an additional burden or overhead. As an optimizer and web site owner you will need to develop an understanding of your web log files or web statistics. That I cover in another section, but for now let's just say that when you analyze these log files there will be entries for user access to the site. Among the names & domains from which your visitors come you will find names like Slurp, MantraAgent, Scooter, and Architextspider. These are the search engine spiders, some call daily, others weekly, some just once a month. Common robot names and associated search engine  As already mentioned two ways exist to allow or disallow a robot. The robot meta tag advises the search engine robot about an individual web page and its links. The robots.txt file advises the search engine robot about what it should access in an individual file (page), several files or entire directories of files. This Robots Meta Tag article continues at: Robot Meta Tag or Robots Txt - Part 2 | 
