Comparing Robots.txt Files From Obama’s New Site and Bush’s Old Site

For those of you not aware, the Robots.txt is a file that lies on the server of a website and tells search engines which parts of a site they can index and which parts they can’t.

Have a look at a comparison of Obama’s new site in comparison to Bush’s:

Obama:

Bush:

 and Bush’s continues to here….

Hopefully a sign of greater transparency & change indeed.

via Codeulate.

Discussion - 39 Comments/Pingbacks RSS feed for comments on this post

1. 

   Famous Web Search Engine says January 20, 2009

   Reply

   Actually, Bush’s robots.txt file is a recommended practice for removing duplicate content.

   See http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66359

   I’m guessing that Bush’s webmaster wanted only the full-html pages to be indexed and linked to, not the plaintext versions.

   • 

     Zee says January 20, 2009

     Reply

     awww…. :( You had to spoil the party…. :P

2. 

   Sherlock says January 20, 2009

   Reply

   Actually, This is from Feb 2, 2001. This is courtesy of wayback machine and as you will see, Mr. Bush did not have a big robots.txt file at that time either

   ****NOT A BUSH SUPPORTER HERE****

   http://web.archive.org/web/20010419161754/whitehouse.gov/robots.txt

   GOBAMA!!!

3. 

   Tjeerd Kramer says January 21, 2009

   Reply

   Also, if they moved most of the content to the /includes/ folder, the situation actually got worse…

4. 

   Brent Shepherd says January 21, 2009

   Reply

   Haha dammit, I thought it was a brilliant post until I read “Famous Web Search Engines” spoiler!

   Still, a creative comparison. :)

5. 

   jerry winkel says January 21, 2009

   Reply

   This is a perfect example of an overly partisan person grasping to make a point and making an idiot of themselves in the process.

6. 

   Zee says January 21, 2009

   Reply

   You hear that guys?? I’m a perfect example! Perfect I tell you…blushing.

1. Comparing Robots.txt Files From Obama’s New Site and Bush’s Old Site « TheNextWeb.com | Michaels Blog

   [...] Comparing Robots.txt Files From Obama’s New Site and Bush’s Old Site « TheNextWeb.com: “” [...]

2. Διαφορές στο site του Λευκού Οίκου - jooblers.gr

   [...] [via] [via] Tags: barack [...]

3. Inauguration Stuff | All about MICROSOFT

   [...] collage from the inauguration courtesy of CNN (I mentioned the amazing TED video way back when), a seriously cut back robots.txt file and a brand new Presidential signature to add to the [...]