The heart of tech

This article was published on May 6, 2015


Apple details its Web crawler, possibly indicating a future push into search

Apple details its Web crawler, possibly indicating a future push into search
Owen Williams
Story by

Owen Williams

Former TNW employee

Owen was a reporter for TNW based in Amsterdam, now a full-time freelance writer and consultant helping technology companies make their word Owen was a reporter for TNW based in Amsterdam, now a full-time freelance writer and consultant helping technology companies make their words friendlier. In his spare time he codes, writes newsletters and cycles around the city.

Apple has quietly added a new page to its support site that details its first Web crawler.

A crawler is a bot, used by search engines such as Google and Bing, that automatically scans the Web for pages and information to add to an index and provides information when pages change.

Apple’s crawler, which existed previously but was never detailed by the company, identifies itself as “Applebot” and could indicate a future push into full Web search.

Right now, Apple says its crawler is used to add data such as movie information to Siri and Spotlight. You can exclude Applebot from searching your site by updating the robots.txt file on your site.

The company’s deal to use Google as the default search engine on iOS is reportedly due to expire soon and rumors have indicated in the past that Apple may be quietly building a search engine of its own.

Applebot

Image credit: Songquan Deng / Shutterstock.com

Also tagged with