Loading...
 
Features / Usability

Features / Usability


Fixed! Re: Googlebot ignoring robots.txt

posts: 9 United States

OK, I changed my robots.txt so that there are leading slashes in front of each page (previously, there were only leading slashes in front of vdir's).

See the attached "robotsneedroot.png" for a graph of my bandwith usage.

  • Late Oct - bots appear to have changed their behavior!
  • Nov - I had to pay for extra bandwidth this month eek
  • Early Dec - I made the changes to robots.txt. Shortly after,
  • PROBLEM SOLVED


More details at my bug report