The To-Don’t List: Things You Will NOT Do
10 Offbeat Holidays to Celebrate in June
Fashism's Online Community Caters to the More Fickle Fashionistas

Channels
- Apple
- Apps
- Daily Dose
- Design & Dev
- Entrepreneur
- Events
- Facebook
- Gadgets
- Google
- Insider
- LifeHacks
- Media
- Microsoft
- Mobile
- Sessions
- Shareables
- Social Media
- Twitter
- Video

Editions
- Africa
- Asia
- Australia
- Canada
- Deutschland
- Europe
- India
- Latin America
- Middle East
- UK
- United States

Languages
- France
- Nederland
- Polska
- Portugal
- Romania
- Russia

Quite possibly the best Robots.txt file ever.

27th July 2010 by Zee

Courtesy of Last.fm, h/t Nick Halstead.

Picture 1433 Quite possibly the best Robots.txt file ever.

Join TNW Shareables on: Follow @shareables OR rss

About the Author

CEO of The Next Web. A fan of startups, entrepreneurship, getting things done faster, penning the occasional blog post, taking photos, designing, listening to good music and making lurrrve.

	FRONTROW Lets You Snap Photos During YouTube LIvestreams
	Mujjo Cases for iPad, iPhone and MacBook in Leather and Felt
	Silicon valley cannot be replicated
	Andrew Keen: 'I'm Not a Luddite'
	Facebook Introduces Trending Videos Feature to News Feed
	Distil Wins the GLUE Conference Demo Pavilion Competition
	The To-Don’t List: Things You Will NOT Do
	10 Offbeat Holidays to Celebrate in June

	Google Chrome Leapfrogs Internet Explorer as the Web's Top Browser
	Facebook To Buy Opera? Maybe. Here's What We Hear.
	How Toy Story 2 Got Deleted Twice, Once on Accident, Again on purpose
	Infogr.am Makes Infographic Creation and Sharing Simple
	Microsoft's So.cl - It's a Trap!
	London Fashion: Meet 5 Startups Re-Shaping the Industry
	China Confirms it Now Has More Than 1 Billion Mobile Phone Users
	Microsoft Pushes Data Sorting to the Next Level

	Google Chrome Leapfrogs Internet Explorer as the Web's Top Browser
	Abraham Lincoln Filed a Patent For Facebook First
	Facebook To Buy Opera? Maybe. Here's What We Hear.
	Apple's Siri: Nokia Lumia 900 Is The Best Smartphone Ever
	This Inspiring Note Greets Apple's New Hires on Their First Day
	55,000 Twitter Accounts Hacked
	Woz on Windows Phone: Compared to Android, There's No Contest
	Apple Linked With $113m Bid for German TV Maker Loewe

	Silicon valley cannot be replicated
	How Long can Windows Phone Stay Obscure?
	Yahoo! Employees Seem to Like Axis
	Folders for Facebook Apps Could be a Good Thing
	Yahoo! Shuts Down its Flipboard Clone Livestand
	Take A Sneak Peek at the Gorgeous Design of SlideDeck 2.1
	WorkFu Taps Your Social Connections to Find the Perfect Job
	Google's Blogger Updates Its Signup Flow

	Google Chrome Leapfrogs Internet Explorer as the Web's Top Browser
	Facebook To Buy Opera? Maybe. Here's What We Hear.
	How Toy Story 2 Got Deleted Twice, Once on Accident, Again on purpose
	75% of Top 10K Sites on the Web Served by Open Source Software
	Pakistan Turns to Interpol after Twitter Declines to Help Censoring
	Jony Ive: Apple is Working on 'Most Important' Work it's Done
	Microsoft's So.cl - It's a Trap!
	Silicon valley cannot be replicated

Quite possibly the best Robots.txt file ever.

About the Author

New and notable from across TNW

Fashism's Online Community Caters to the More Fickle Fashionistas

10 Offbeat Holidays to Celebrate in June

The To-Don’t List: Things You Will NOT Do

Tears, Tantrums & Tech 6: Keep It Simple, Stupid [Video]

Andrew Keen: 'I'm Not a Luddite'

Facebook Introduces Trending Videos Feature to News Feed

Apple Releases Revised iOS 5.1.1 Update for GSM iPhone 4

TETRIS The Movie [video]

Airbnb iOS App Now Has Offline Access

Yahoo! Employees Seem to Like Axis

A Rare Video From Inside an iPhone Factory

How Long can Windows Phone Stay Obscure?

Partners



Save to read later		or copy the link
		Copy-paste link