For those of you not aware, the Robots.txt is a file that lies on the server of a website and tells search engines which parts of a site they can index and which parts they can’t.
Have a look at a comparison of Obama’s new site in comparison to Bush’s:
Obama:
Bush:
and Bush’s continues to here….
Hopefully a sign of greater transparency & change indeed.
via Codeulate.
Actually, Bush’s robots.txt file is a recommended practice for removing duplicate content.
See http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66359
I’m guessing that Bush’s webmaster wanted only the full-html pages to be indexed and linked to, not the plaintext versions.
Actually, This is from Feb 2, 2001. This is courtesy of wayback machine and as you will see, Mr. Bush did not have a big robots.txt file at that time either
****NOT A BUSH SUPPORTER HERE****
http://web.archive.org/web/20010419161754/whitehouse.gov/robots.txt
GOBAMA!!!
Also, if they moved most of the content to the /includes/ folder, the situation actually got worse…
Haha dammit, I thought it was a brilliant post until I read “Famous Web Search Engines” spoiler!
Still, a creative comparison. :)
This is a perfect example of an overly partisan person grasping to make a point and making an idiot of themselves in the process.
You hear that guys?? I’m a perfect example! Perfect I tell you…blushing.
awww…. :( You had to spoil the party…. :P