What is Robots.txt?

Robots.txt is a plain text file at the root of a website that tells search engine crawlers and other bots which parts of the site they may crawl.

A common belief is that robots.txt hides pages from Google. It does not, at least not in the way most people expect. Robots.txt is a small text file that sits at the root of your domain, for example yoursite.com/robots.txt, and it tells crawlers which areas of the site they are allowed to visit.

In simple words, it is a notice board at the gate. Well-behaved bots read it and follow it. It controls crawling, not indexing, and it is not a security lock.

How it works

The file is made of simple groups of rules. Each group starts with a user-agent line, which names the bot, followed by allow and disallow lines for folders or URL patterns. You can also add a line pointing to your XML sitemap.

For example, an online store might tell all bots not to crawl its cart, checkout and internal search pages, because those pages create endless URL variations and offer nothing useful in search results. That keeps crawlers focused on product and category pages.

The mistake that surprises people

If you block a page in robots.txt, Google cannot read that page. But if other sites link to it, Google may still index the bare URL and show it with little or no description. And because Google cannot crawl the page, it also cannot see a noindex tag placed on it.

This is why the better approach for keeping a page out of search results is a noindex tag on a page that crawlers are allowed to reach, or password protection for anything truly private. The file itself is public, so anyone can read it.

Robots.txt and AI crawlers

Robots.txt has become a business decision again because of AI. Companies that build AI models and AI search tools run their own crawlers, and many publish user-agent names, such as OpenAI's GPTBot, that you can allow or block. Some businesses block AI training bots to protect their content. Others welcome them because they want their brand to appear in AI answers. There is no single right answer. That said, think it through before blocking, since content that AI tools cannot read is less likely to be cited by them. Some sites also add an llms.txt file, which is a separate and newer idea.

What to check

Open yoursite.com/robots.txt and confirm it exists and loads.
Make sure there is no "Disallow: /" rule for all bots on your live site. It sometimes stays behind after a website moves from a staging server.
Do not block the CSS and JavaScript files that Google needs to render your pages properly.
Review the robots.txt report in Google Search Console after making changes.

Simple takeaway

Robots.txt is a small file with a big reach. One wrong line can stop search engines from crawling an entire website. Keep it short, keep it intentional, and remember its job: guiding crawlers, not hiding pages. It is a foundational piece of technical SEO worth checking every few months.