URL: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

"robots.txt" is a file you can create on your site to help indexing bots to index your site correctly. These bots first scans your robots.txt file to see which pages to ignore.

This page is a good tool to keep in mind to validate your robots.txt files. robotstxt.org has more information about the wannabe standard.

Comments

Peter January 24, 2004

Just found out how to exclude my printer-friendly version and PDF version (see bottom righthand corner)

Disallow: /pv$
Disallow: /pv/pdf$

Let's hope it works.

Your email will never ever be published.

Previous:
MathML and displaying Math on the web January 23, 2004 Mathematics, Web development
Next:
Labels in HTML forms January 26, 2004 Web development
Related by category:
Fastest way to find out if a file exists in S3 (with boto3) June 16, 2017 Web development
Be very careful with your add_header in Nginx! You might make your site insecure February 11, 2018 Web development
<datalist> looks great on mobile devices August 28, 2020 Web development
How to get all of MDN Web Docs running locally June 9, 2021 Web development
Related by keyword:
django-html-validator October 20, 2014 Python, Web development, Django
Interesting float/int casting in Python April 25, 2006 Python
Check your email addresses in Python, as a whole May 22, 2020 Python, MDN
django-html-validator now supports Django 2.x August 13, 2018 Python, Web development, Django