Watch those invasive robots

I recently had a client who noticed some strange activity on their website.  It turns out google robots were hitting URL’s that were sent out in newsletter emails.  The URL’s were not published anywhere else.  Further research uncovered others complaining about similar activity.  So the bottom line is that google (and potentially other search engines) are looking for new pages to crawl by scanning emails, or recording your browsing history.  This means that if you are testing a new area of your website, you should lock it down to prevent the search engines from finding it.

Category: Web Maintenance Comment »

Leave a Reply

You must be logged in to post a comment.

Back to top