Hey—we've moved. Visit
The Keyword
for all the latest news and stories from Google
Official Blog
Insights from Googlers into our products, technology, and the Google culture
More on Robots Exclusion Protocol (REP)
3 juin 2008
Posted by Prashanth Koppula, Product Manager
Web publishers often ask us how they can maximize their visibility on the web. Much of this has to do with search engine optimization -- making sure a publisher's content shows up on all the search engines.
However, there are some cases in which publishers need to communicate more information to search engines -- like the fact that they
don't
want certain content to appear in search results. And for that they use something called the
Robots Exclusion Protocol (REP)
, which lets publishers control how search engines access their site: whether it's controlling the visibility of their content across their site (via robots.txt) or down to a much more granular level for individual pages (via META tags).
Since it was introduced in the early '90s, REP has become the de facto standard by which web publishers specify which parts of their site they want public and which parts they want to keep private. Today, millions of publishers use REP as an easy and efficient way to communicate with search engines. Its strength lies in its flexibility to evolve in parallel with the web, its universal implementation across major search engines and all major robots, and in the way it works for any publisher, no matter how large or small.
While REP is observed by virtually all search engines, we've never come together to detail how we each interpret different tags. Over the last couple of years, we have worked with Microsoft and Yahoo! to bring forward standards such as
Sitemaps
and offer additional tools for webmasters. Since the original announcement, we have, and will continue to, deliver further improvements based on what we are hearing from the community.
Today, in that same spirit of making the lives of webmasters simpler, we're releasing detailed documentation about how we implement REP. This will provide a common implementation for webmasters and make it easier for any publisher to know how their REP directives will be handled by three major search providers -- making REP more intuitive and friendly to even more publishers on the web.
To see the major REP features currently implemented by Google, Microsoft, and Yahoo!, please see our
detailed post
on the Webmaster Central blog.
Libellés
accessibility
41
acquisition
26
ads
131
Africa
19
Android
59
apps
419
April 1
4
Asia
39
books + book search
48
commerce
12
computing history
7
crisis response
33
culture
12
developers
120
diversity
35
doodles
68
education and research
144
entrepreneurs at Google
14
Europe
46
faster web
16
free expression
61
google.org
73
googleplus
50
googlers and culture
202
green
102
Latin America
18
maps and earth
194
mobile
125
online safety
19
open source
19
photos
39
policy and issues
139
politics
71
privacy
66
recruiting and hiring
32
scholarships
31
search
505
search quality
24
search trends
118
security
36
small business
31
user experience and usability
41
youtube and video
140
Archive
2016
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2015
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2014
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2013
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2012
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2011
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2010
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2009
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2008
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2007
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2006
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2005
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
mars
févr.
janv.
2004
déc.
nov.
oct.
sept.
août
juil.
juin
mai
avr.
Feed
Google
on
Follow @google
Follow
Give us feedback in our
Product Forums
.