Robots.txt Validator

24 January 2004   1 comment   Web development

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

Mind that age!

This blog post is 16 years old! Most likely, its content is outdated. Especially if it's technical.

"robots.txt" is a file you can create on your site to help indexing bots to index your site correctly. These bots first scans your robots.txt file to see which pages to ignore.

This page is a good tool to keep in mind to validate your robots.txt files. robotstxt.org has more information about the wannabe standard.

Comments

Peter

Just found out how to exclude my printer-friendly version and PDF version (see bottom righthand corner)

Disallow: /pv$
Disallow: /pv/pdf$

Let's hope it works.

Your email will never ever be published

Related posts