Robots.txt Validator

24 January 2004   1 comment   Web development

Mind That Age!

This blog post is 14 years old! Most likely, its content is outdated. Especially if it's technical.

"robots.txt" is a file you can create on your site to help indexing bots to index your site correctly. These bots first scans your robots.txt file to see which pages to ignore.

This page is a good tool to keep in mind to validate your robots.txt files. has more information about the wannabe standard.


Just found out how to exclude my printer-friendly version and PDF version (see bottom righthand corner)

Disallow: /pv$
Disallow: /pv/pdf$

Let's hope it works.

Your email will never ever be published

Related posts

MathML and displaying Math on the web 23 January 2004
Labels in HTML forms 26 January 2004
Related by Keyword:
django-html-validator 20 October 2014
Interesting float/int casting in Python 25 April 2006
Related by Text:
Be very careful with your add_header in Nginx! You might make your site insecure 11 February 2018
jQuery and Highslide JS 08 January 2008
I'm back! has been renewed 05 June 2005
Anti-McCain propaganda videos 12 August 2008
I'm Prolog 01 May 2007