Robots.txt meaning in English


What is a Robots.txt file?

Robots.txt files are used as a communication method to control which pages can be crawled by bots.

It is possible to allow and disallow all, or specific search engines/bots on areas of a website.

The robots.txt file should be found at the root of a website, for example

Robots.txt files can also include a references to sitemaps.


The following example robots.txt file contains a rule blocking search engines from a specific page, and a reference to an XML sitemap file.

User-agent: *
Disallow: /blocked-page/

Sitemap: language: plaintext (plaintext)



Citation URL

Last modified: Created:

What is this content? This article is part of an SEO glossary and reference guide created by Search Candy - an SEO consultancy based in the UK. The Search Candy team is committed to providing content that adheres to the highest editorial standards. The date this article was created and last checked for accuracy is posted above. To reuse this content please get in touch via our contact form.