This blog post is 14 years old! Most likely, its content is outdated. Especially if it's technical.
"robots.txt" is a file you can create on your site to help indexing bots to index your site correctly. These bots first scans your robots.txt
file to see which pages to ignore.
This page is a good tool to keep in mind to validate your robots.txt files. robotstxt.org has more information about the wannabe standard.
- Previous:
- MathML and displaying Math on the web
23 January 2004
- Next:
- Labels in HTML forms 26 January 2004
- Related by Keyword:
- django-html-validator now supports Django 2.x 13 August 2018
- django-html-validator 20 October 2014
- Interesting float/int casting in Python 25 April 2006
- Related by Text:
- jQuery and Highslide JS 08 January 2008
- I'm back! Peterbe.com has been renewed 05 June 2005
- Anti-McCain propaganda videos 12 August 2008
- Ever wondered how much $87 Billion is?
04 November 2003
- Guake, not Yakuake or Yeahconsole 23 January 2010
Just found out how to exclude my printer-friendly version and PDF version (see bottom righthand corner)
Disallow: /pv$
Disallow: /pv/pdf$
Let's hope it works.