A text file that tells web crawlers which parts of a site to avoid.
It is a "Staff Only" sign for a website. Nice crawlers see it and stay out of the kitchen.
Search bots use it before they scan a site. It is a request, not a lock. Rude bots can ignore it.
Agent-readable Web
It tells rule-following web agents which paths they should not crawl.
Common Crawl
Common Crawl usually checks these rules before collecting web pages.