![\"\"](\"https:\/\/web.archive.org\/web\/20130902000726im_\/https:\/\/semseo.md\/images\/stat\/robots.jpeg\")
<\/p>\nThe robots.txt file consists of records. Each record has at least two lines: a line with the client application name \u2013 User-agent, and one or more lines starting with the Disallow directive. Empty lines are significant, they separate records with different User-agent values.<\/p>\n
<\/p>\n
User-agent<\/p>\n
The User-agent line specifies the name of the crawler. For example, the following line specifies Google\u2019s bot \u2013 \u201cgooglebot\u201d:<\/p>\n
User-agent: googlebot<\/p>\n
Yandex bot \u2013 \u201cYandex\u201d
\nRambler bot \u2013 \u201cStackRambler\u201d
\nYahoo! bot \u2013 \u201cYahoo! Slurp\u201d
\nMSN bot \u2013 \u201cmsnbot\u201d<\/p>\n
Other crawler names can be found in your server logs.<\/p>\n
If you want to block indexing of files and\/or folders by all search engines, you can use the wildcard \u201c*\u201d:<\/p>\n
User-agent: *
\nDisallow<\/p>\n
The second part of the record contains Disallow lines. These directives tell the bot which files and\/or folders must not be indexed. The paths in Disallow are relative, not absolute, so you do not need to enter the domain name.<\/p>\n
Example: the following directive forbids indexing of the file \u201cdownload.htm\u201d located in the root directory:<\/p>\n
Disallow: download.htm<\/p>\n
The directive can also specify a folder. Example: forbid indexing of the \u201ccgi-bin\u201d directory:<\/p>\n
Disallow: \/cgi-bin\/<\/p>\n
The next directive blocks both the file \u201ccatalog.html\u201d and the \u201ccatalog\u201d folder:<\/p>\n
Disallow: \/catalog<\/p>\n
If the Disallow line is empty, the bot may index everything. At least one Disallow directive must be present for each User-agent, otherwise robots.txt may not be read correctly. A completely empty robots.txt is equivalent to its absence.<\/p>\n
Allow full indexing for all search engines:<\/p>\n
User-agent: *
\nDisallow:<\/p>\n
Block all indexing for all search engines:<\/p>\n
User-agent: *
\nDisallow: \/<\/p>\n
Block indexing of the \u201ccgi-bin\u201d folder:<\/p>\n
User-Agent: *
\nDisallow: \/cgi-bin\/<\/p>\n
Block indexing of the \u201cdownload.htm\u201d file:<\/p>\n
User-Agent: *
\nDisallow: download.htm<\/p>\n
Block indexing of \u201cdownload.htm\u201d and the \u201ccgi-bin\u201d folder:<\/p>\n
User-Agent: *
\nDisallow: \/cgi-bin\/
\nDisallow: download.htm<\/p>\n
Block indexing of \u201cdownload.htm\u201d only for Google\u2019s bot \u2013 \u201cgooglebot\u201d:<\/p>\n
User-agent: googlebot
\nDisallow: download.htm<\/p>\n
Comments<\/p>\n
Any text following the \u201c#\u201d symbol to the end of the line is considered a comment and ignored by bots. Example:<\/p>\n
# Yahoo! No index.
\nUser-agent: Yahoo! Slurp
\nDisallow: \/<\/p>","protected":false},"excerpt":{"rendered":"
Managing site indexing by all search engines is possible using the robots.txt file, located in the root directory of the server. This file tells search engine crawlers (bots) which files they may index and which they may not. The robots.txt file consists of records. Each record has at least two lines: a line with the […]<\/p>\n","protected":false},"author":1,"featured_media":5186,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"yoast_head":"\n