Comparing Robots.txt Files From Obama’s New Site and Bush’s Old Site
For those of you not aware, the Robots.txt is a file that lies on the server of a website and tells search engines which parts of a site they can index and which parts they can’t.
Have a look at a comparison of Obama’s new site in comparison to Bush’s:
Obama:
![]()
Bush:

and Bush’s continues to here….
Hopefully a sign of greater transparency & change indeed.
via Codeulate.
Discussion - 39 Comments/Pingbacks RSS feed for comments on this post
-
Comparing Robots.txt Files From Obama’s New Site and Bush’s Old Site « TheNextWeb.com | Michaels Blog
[...] Comparing Robots.txt Files From Obama’s New Site and Bush’s Old Site « TheNextWeb.com: “” [...]
-
Διαφορές στο site του Λευκού Οίκου - jooblers.gr
[...] [via] [via] Tags: barack [...]
-
Inauguration Stuff | All about MICROSOFT
[...] collage from the inauguration courtesy of CNN (I mentioned the amazing TED video way back when), a seriously cut back robots.txt file and a brand new Presidential signature to add to the [...]


















Actually, Bush’s robots.txt file is a recommended practice for removing duplicate content.
See http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66359
I’m guessing that Bush’s webmaster wanted only the full-html pages to be indexed and linked to, not the plaintext versions.
awww…. :( You had to spoil the party…. :P
Actually, This is from Feb 2, 2001. This is courtesy of wayback machine and as you will see, Mr. Bush did not have a big robots.txt file at that time either
****NOT A BUSH SUPPORTER HERE****
http://web.archive.org/web/20010419161754/whitehouse.gov/robots.txt
GOBAMA!!!
Also, if they moved most of the content to the /includes/ folder, the situation actually got worse…
Haha dammit, I thought it was a brilliant post until I read “Famous Web Search Engines” spoiler!
Still, a creative comparison. :)
This is a perfect example of an overly partisan person grasping to make a point and making an idiot of themselves in the process.
You hear that guys?? I’m a perfect example! Perfect I tell you…blushing.