define robots.txt

Definition:

A special file in the root directory of a website used to control how and which search engine spiders access pages within a website. When a spider or robot connects to a website, it checks for the presence of the robots.txt file and uses it to index or avoid specific or all web pages within the site.