Apr 28
Website Marketing Tips #17 – robots.txt
What is a robots.txt file?
A robots.txt file basically tells the search engine spiders what not to look at. Which directories to ignore or skip.
The robots.txt file is locate in the root directory of your web site. Example: http://www.yourwebsite.com/robots.txt
How do you create a robots.txt file?
The robots.txt file is a plain text file. This is the basic format:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /admin/
As of recently many of the major search engines adopted a new function, you can add the sitemap.xml file to the robots.txt file:
Sitemap: http://www.yourwebsite.com/sitemap.xml
The search engine spiders do look for the robots.txt file.
For more information about robots.txt file creation:
http://www.robotstxt.org/wc/robots.html
Comments Off
