Robots.txt Validator

24 January 2004   1 comment   Web development

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

Peterbe.com Robots.txt Validator
Archive About Contact
Menu
Archive About Contact Search
Mind That Age!

This blog post is 13 years old! Most likely, its content is outdated. Especially if it's technical.

Powered by Fusion×

"robots.txt" is a file you can create on your site to help indexing bots to index your site correctly. These bots first scans your robots.txt file to see which pages to ignore.

This page is a good tool to keep in mind to validate your robots.txt files. robotstxt.org has more information about the wannabe standard.

Comments

Peter
24 January 2004
Just found out how to exclude my printer-friendly version and PDF version (see bottom righthand corner)

Disallow: /pv$
Disallow: /pv/pdf$

Let's hope it works.
Reply
Thank you for posting a comment

Your email will never ever be published


Related posts

Previous:
MathML and displaying Math on the web 23 January 2004
Next:
Labels in HTML forms 26 January 2004
Related:
django-html-validator 20 October 2014
Interesting float/int casting in Python 25 April 2006

© peterbe.com 2003 - 2017 ⋅ Hosted on Digital Ocean

Home Archive About Contact Search

Check out my latest side project Song Search