Robots.txt Generator is a handy tool that uses a web form to generate a Robots.txt file for your website. There are lots of options that make the file generation a snap! For those of you wondering, Robots.txt is a very simple text file that is placed on your root directory and tells search engines and other robots which parts of your site they are allowed to visit and index. Don’t worry if you don’t have one at the moment because by default search engines will crawl all parts of your site that they can reach. However, if there are parts of your site that you don’t want crawled for whatever reason (i.e. kept out of search engine indexes), it’s surprisingly easy to restrict robots through the use of the Robots.txt file.
Robots.txt Generator
Previous post: New Web IDE
Next post: Palette Grabber Firefox Extension
