Sebastian's Pamphlets

If you've read my articles somewhere on the Internet, expect something different here.

MOVED TO SEBASTIANS-PAMPHLETS.COM

Please click the link above to read actual posts, this archive will disappear soon!

Stay tuned...

Monday, March 12, 2007

Q: Does Googlebot obey ftp://ftp.example.com/robots.txt?

Google states
Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl.
A site owner posted logs where Googlebot fetches files although these are disallowed in robots.txt.

Anyone?
Share this post at StumbleUpon
Stumble It!
    Share this post at del.icio.us
Post it to
del.icio.us
 


-->

0 Comments:

Post a Comment

Links to this post:

Create a Link

<< Home