index.md

---
title: Robots.txt
slug: Glossary/Robots.txt
page-type: glossary-definition
---

{{GlossarySidebar}}

Robots.txt is a file which is usually placed in the root of any website. It decides whether {{Glossary("crawler", "crawlers")}} are permitted or forbidden access to the website.

For example, the site admin can forbid crawlers to visit a certain folder (and all the files therein contained) or to crawl a specific file, usually to prevent those files being indexed by other search engines.

## See also

- [Robots.txt](https://en.wikipedia.org/wiki/Robots.txt) on Wikipedia
- <https://developers.google.com/search/reference/robots_txt>
- Standard specification: [RFC9309](https://www.rfc-editor.org/rfc/rfc9309.html)
- <https://www.robotstxt.org/>