Features / Usability

Posted by John Hadjisky 07 Dec 2005 04:40 GMT-0000 posts: 9

While the formal grammer for robots.txt ( http://www.robotstxt.org/wc/norobots-rfc.html ) does not appear to require a leading / in Disallow: statements, in literally every example given, Disallow: is followed by a leading /. I don't know why they didn't make this a requirement.

Furthermore, parsers such as http://tool.motoricerca.info/robots-checker.phtml will specifically warn you if your disallow line doesn't have a leading /. This parser warns that the standard TikiWiki robots.txt file, for example http://dupli.tikiwiki.org/robots.txt, should make better use of leading /. Typical output:

Line 22 	Disallow: tiki-install.php
We advise you to start a file/directory name with a leading slash char (Example: /private.html).

I am making this change to my robots.txt ( http://ihuck.com/robots.txt ) and will report the results after my next visit from the GoogleBot

But, since the leading / might help, and can hardly hurt, I'd recommend using it for all disallows, anyways.