Story by
Owen Williams
Former TNW employeeOwen was a reporter for TNW based in Amsterdam, now a full-time freelance writer and consultant helping technology companies make their word Owen was a reporter for TNW based in Amsterdam, now a full-time freelance writer and consultant helping technology companies make their words friendlier. In his spare time he codes, writes newsletters and cycles around the city.
Apple has quietly added a new page to its support site that details its first Web crawler.
A crawler is a bot, used by search engines such as Google and Bing, that automatically scans the Web for pages and information to add to an index and provides information when pages change.
Apple’s crawler, which existed previously but was never detailed by the company, identifies itself as “Applebot” and could indicate a future push into full Web search.
Right now, Apple says its crawler is used to add data such as movie information to Siri and Spotlight. You can exclude Applebot from searching your site by updating the robots.txt file on your site.
The company’s deal to use Google as the default search engine on iOS is reportedly due to expire soon and rumors have indicated in the past that Apple may be quietly building a search engine of its own.
➤ Applebot
Image credit: Songquan Deng / Shutterstock.com
Get the TNW newsletter
Get the most important tech news in your inbox each week.
