This blog post is
13 years old! Most likely, its content is outdated.
Especially if it's technical.
"robots.txt" is a file you can create on your site to help indexing bots to index your site correctly. These bots first scans your robots.txt file to see which pages to ignore.
This page is a good tool to keep in mind to validate your robots.txt files. robotstxt.org has more information about the wannabe standard.
- Previous:
- MathML and displaying Math on the web
23 January 2004
- Next:
- Labels in HTML forms
26 January 2004
- Related by Keyword:
- django-html-validator
20 October 2014
- Interesting float/int casting in Python
25 April 2006
- Related by Text:
- The importance of being findable
15 April 2004
- Google hardware history
16 May 2004
- Autocompeter.com
02 April 2015
- ElasticSearch, snowball analyzer and stop words
25 September 2015
- CSS Bloat Comparison
03 June 2016
Disallow: /pv$
Disallow: /pv/pdf$
Let's hope it works.