Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?
Google states
Anyone?
Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl.A site owner posted logs where Googlebot fetches files although these are disallowed in robots.txt.
Anyone?
 Stumble It! |
 Post it to del.icio.us |
posted by Sebastian @ Monday, March 12, 2007 · PERMANENT LINK
-->
Subscribe to my blog
0 Comments:
Post a Comment
<< Home