For those of you not aware, the Robots.txt is a file that lies on the server of a website and tells search engines which parts of a site they can index and which parts they can’t.
Have a look at a comparison of Obama’s new site in comparison to Bush’s:
F**k it, we'll do it live!
Our biggest ever edition of TNW Conference is fast approaching! Join 10,000 tech leaders this May in Amsterdam.
and Bush’s continues to here….
Hopefully a sign of greater transparency & change indeed.