The Robots.txt file is used as a communication method to control indexing from search engine bots.

It is possible to allow and disallow all or specific search engines/bots on areas of a website.

The robots.txt file should be found at the root of a website, for example

Created: Last modified: