September 29, 2004

Google’s new Googlebot

by brian_turner

There’s plenty of talk about Google’s new Googlebot visiting websites:

Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)

It’s been suggested that the Googlebot might be parsing Javascript and other scripting methods normally used to hide links from search engines.

However, in this thread at DigitalPoint, Shaun makes a point of commenting that it’s using the HTTP 1.1 protocol, as the Googlebot/Test did – but it seems to be spidering normally, albeit from an IP range he normally associates with Google Adsene’s MediaPartner’s bot.

Dirk of fantomaster, one of the biggest providers of cloaking software, and who personally has to put up with tracking bots across nearly 10,000 websites, had this to say on the Search Engine Watch forums:

During the last 2 weeks we noticed lots of new spiders (about 200) coming from 3 new IP ranges:

66.249.64.xxx
66.249.65.xxx
66.249.66.xxx

It’s not unusual that search engines use new IP ranges from time to time. We have often seen this in the past.

It’s interesting to note that Brett Tabke at WebmasterWorld recently commented that Google was “panic-indexing” the internet – effectively, building a new index from scratch.

With the PR update drought still in process, it’ll be interesting to see if these changes in Google indexing are indicative of a major new update in the pipeline…

Questions? Discuss this in our Internet Business forums for help and advice

Story link: Google’s new Googlebot

Add to Bookmarks: