Official Google Blog: Controlling how search engines access and index your website

Hey—we've moved. Visit The Keyword for all the latest news and stories from Google

Official Blog

Insights from Googlers into our products, technology, and the Google culture

Controlling how search engines access and index your website

January 26, 2007

Posted by Dan Crow, Product Managerrobots.txtrobots.txtrobots.txtrobots.txtWhat does robots.txt do?The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams)robots.txtrobots.txtrobots.txtFine-grained controlrobots.txtMETAMETArobots.txtMETAA simple examplerobots.txt

User-Agent: Googlebot
Disallow: /logs/

User-Agentrobots.txtDisallowlogs logsPreventing access to a fileMETA

<html>
<head>
<meta name="googlebot" content="noindex">
...

METArobots.txtLearn morehttp://www.robotstxt.orgGoogle's Webmaster help center, which contains lots of helpful information, including:

How to create a robots.txt file

Descriptions of each user-agent that Google uses

How to use pattern matching

How often we recrawl your robots.txt file

webmaster blog

Using robots.txt files

All about Googlebot

There is also a useful list of the bots used by the major search engines: http://www.robotstxt.org/wc/active/html/index.html Next time...Update:

Labels: search

Labels

Archive

2016
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2015
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2014
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2013
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2012
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2011
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2010
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2009
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2008
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2007
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2006
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2005
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2004
- Dec
- Nov
- Oct
- Sep
- Aug
- Jul
- Jun
- May
- Apr

Feed

Give us feedback in our
Product Forums.

Company-wide

Public Policy Blog
Research Blog
Student Blog

Products

Official Android Blog
Chrome Blog
Lat Long Blog

Developers

Developers Blog
Ads Developer Blog
Android Developers Blog

Google
Privacy
Terms