The robots.txt file on any site is how search engines discover what a site wants indexed and what it doesn’t.
Generally you won’t find many people checking the files unless they’re technically minded, passionate about SEO and curious. The Daily Mail of all sites thought it would be an ingenious idea to place a job advert in the file and SEO consultant Malcolm Coles discovered it.
F**k it, we'll do it live!
Our biggest ever edition of TNW Conference is fast approaching! Join 10,000 tech leaders this May in Amsterdam.
This is most likely the work of The Mail Online’s MD James Bromley.
I’m rather impressed…The Daily Mail thinking well outside of the box.