24 January 2004 1 comment Web development
"robots.txt" is a file you can create on your site to help indexing bots to index your site correctly. These bots first scans your robots.txt file to see which pages to ignore.
This page is a good tool to keep in mind to validate your robots.txt files. robotstxt.org has more information about the wannabe standard.
Disallow: /pv$
Disallow: /pv/pdf$
Let's hope it works.