Robots.txt is the file at yoursite.com/robots.txt that tells Google (and other crawlers) which parts of your site they're allowed to crawl. It's three lines of text. It can also single-handedly tank your entire SEO if it goes wrong.
The "I broke staging into production" mistake
The single most common SEO catastrophe is shipping a robots.txt with User-agent: * followed by Disallow: / — the universal "block everyone from everything" rule. This was meant for the staging server. It got pushed to production. The site disappears from Google over the following weeks. Always check production robots.txt after every deploy.
The "I blocked /wp-content" mistake
Blocking /wp-content/ or /assets/ used to be common to hide CMS internals. It also blocks the CSS and JavaScript files Google needs to render your pages. Result: Google sees a broken layout and may decide your pages are low quality.
Fix: only block actual sensitive paths, never asset folders.
The "blocked + sitemapped" mistake
You list a URL in your sitemap and ALSO disallow it in robots.txt. Google can't crawl it but you keep telling it to. Search Console flags this as an error. Pick one: either the URL is allowed and indexed, or it's not.
The "I forgot the Sitemap line" mistake
Robots.txt is also where you tell crawlers where your sitemap lives. If you don't include a Sitemap: https://yoursite.com/sitemap.xml line, you're skipping a free signal.
A clean baseline robots.txt
For most sites, all you need is: User-agent: *, then block /admin/, internal search results, and tracking parameter URLs, and finish with a Sitemap: line pointing at your sitemap.xml. Three or four meaningful lines total.
Test before you ship
Google's Robots Testing Tool in Search Console lets you paste a URL and see whether your current robots.txt blocks it. Use it after every change.
What robots.txt does NOT do
Robots.txt blocks crawling, not indexing. If someone else links to a page you've disallowed, Google can still index the URL (without the content). To truly prevent indexing, use a noindex meta tag — which means Google has to be able to crawl the page to see the tag. So don't combine Disallow + noindex — it cancels out.