We all know that yesterday was a monumental day for the Internet, with major sites such as Reddit and Wikipedia “blacking out” in solidarity against SOPA and PIPA. But there were likely some unintended consequences of the blackout, namely when it comes to the search results that you’ll see today.
In looking at traffic through its network, CloudFlare found the hard numbers about which crawlers were moving faster or slower yesterday versus the day before. Here are the results, given as the percentage higher or lower than the day previous:
- Google was down 60.7%
- Baidu was down 11.23%
- Bing was up 2.35%
To go even deeper, though, it appears that Google was crawling sites at varied speeds, depending upon the size of the site in question. According to CloudFlare CEO Matthew Prince:
“The drop in Google’s crawl traffic varies substantially across small/medium/large sites with the relative drops being 80.84%/48.67%/24.92% respectively.”
In my head, the situation plays out like this – Google had publicly stated that it would crawl at a lower rate so as to not harm search results on sites that were choosing to participate in the blackout. Given that information, Bing saw an opportunity to have fresher results and increased its crawl rate.
The 💜 of EU tech
The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!
The unintended conseqnece here is that today Google perhaps has results that are not as “fresh” as Bing. However, it’s also possible that Bing’s increased crawl rate has left it with a few more results that are indicative of the blackout, instead of what a site’s content actually is.
So the question to you, Internet, is this – Have you noticed any odd search results today? Over on Search Engine Roundtable, Barry Schwartz noted some pretty drastic results in the Google Webmaster Help. Given that Schwartz is my go-to guy when it comes to all things search, I asked him about the increase in Bing results:
“A 2% increase might be due to blacking out sites and confusing BingBot. So, it tries to crawl more often in order to figure out the problem.”
Schwartz gives the disclaimer, however, that he’s not certain as to the specific reasoning behind the increased crawl frequency from Bing, though I think his theory is sound.
Get the TNW newsletter
Get the most important tech news in your inbox each week.