“A file used by websites to instruct search engines on what to index, but optional and often ignored by scrapers.”

“A file websites use to communicate with web crawlers, mentioned as something good citizens of the internet should obey.”

“A voluntary standard for controlling web crawlers discussed as a legacy tool.”