Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl.A site owner posted logs where Googlebot fetches files although these are disallowed in robots.txt.
Anyone?
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.