Sitemaps & robots.txt: What They Do and Why They Matter
An XML sitemap and a robots.txt file are the two standard ways your website talks to search engine crawlers. The sitemap is a machine-readable list of the URLs you want discovered and indexed, with optional last-modified dates. The robots.txt file, served at yourdomain.com/robots.txt, tells crawlers which parts of the site not to crawl. Neither one forces indexing or guarantees privacy — they are instructions to well-behaved bots, and they matter most for discovery, crawl efficiency, and avoiding self-inflicted SEO damage.
- Sitemap limits
- One sitemap file may contain up to 50,000 URLs and 50MB uncompressed (sitemaps.org protocol)
- Protocol adoption
- The Sitemaps protocol was jointly adopted by Google, Yahoo, and Microsoft in 2006 (sitemaps.org)
- robots.txt standardization
- In use since 1994; formally standardized as RFC 9309 in 2022 (IETF)
- Mobile-first indexing
- Google predominantly crawls and indexes the mobile version of sites (Google)
Crawling vs. indexing: two different jobs #
The distinction that unlocks this whole topic: crawling is a search engine fetching your pages; indexing is deciding to store and rank them. They are separate steps with separate controls, and mixing them up causes real damage. robots.txt controls crawling — it can stop Googlebot from fetching a URL. A noindex meta tag controls indexing — it tells Google not to include a fetched page in results. Here is the trap: if you block a page in robots.txt, Google cannot fetch it, which means it can never see a noindex tag on it. The URL can still appear in search results, shown bare with no description, if enough external links point at it. So the tools are not interchangeable: use robots.txt to keep crawlers out of wasteful or infinite URL spaces, and use noindex to keep specific pages out of results. The sitemap plays offense on the other side — actively advertising the URLs you do want crawled and indexed.
What is an XML sitemap? #
An XML sitemap is a structured file — usually at yourdomain.com/sitemap.xml — listing the canonical URLs of your site. Each entry sits in a url element with the address in loc and, optionally, a lastmod date telling crawlers when the page last changed. Google has said it largely ignores the old priority and changefreq tags, but lastmod matters when it is accurate: it helps crawlers revisit updated pages sooner. The protocol allows 50,000 URLs and 50MB per file; bigger sites use a sitemap index file pointing to multiple sitemaps, and most platforms — WordPress with an SEO plugin, Shopify, Squarespace — generate all of this automatically, splitting by content type. Be clear about what a sitemap is not: it is a discovery aid, not a command. Google treats it as a strong hint about what exists and what you consider canonical. Listing a page guarantees consideration, not indexing — thin or duplicate pages get skipped no matter how they were discovered.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://yourbusiness.com/services/</loc>
<lastmod>2026-06-14</lastmod>
</url>
</urlset>What belongs in your sitemap (and what doesn't) #
A sitemap should be a curated list of pages you would be proud to see ranking — think of it as your site's official menu, not its junk drawer. Include: the homepage, service and product pages, location pages, substantive blog posts, and key category pages. Every URL should be canonical, return a 200 status, and be indexable. Exclude: redirected URLs, 404s, pages carrying noindex, non-canonical duplicates, tag archives, paginated fragments, thank-you and cart pages, and internal search results. Why the strictness? Search engines use sitemap quality as a trust signal about your site's hygiene — Search Console will report indexing errors when your sitemap is full of redirects and dead ends, and a noisy sitemap makes those reports useless for spotting real problems. It also wastes crawl attention on a small site's limited budget of crawler visits. The practical test we apply in audits: pull ten random URLs from the sitemap; if any redirect, 404, or carry noindex, the generation logic needs fixing, not the individual URLs.
What can robots.txt actually do? #
robots.txt is a plain-text file at the root of your domain that well-behaved crawlers fetch before crawling anything else. Its vocabulary is small: User-agent lines name which bot the rules apply to (* means all), Disallow and Allow lines mark URL paths, and a Sitemap line advertises your sitemap's location. Its legitimate jobs: keeping crawlers out of infinite or worthless URL spaces — faceted filter combinations, internal search results, cart and checkout flows, admin paths — so crawl attention goes to pages that earn revenue. On large sites this is real crawl-budget management; on a 30-page local business site, crawl budget is rarely a problem, and the file's main job is simply to avoid mistakes. Two properties to internalize: robots.txt is voluntary, honored by reputable crawlers and ignored by malicious ones, and it is public — anyone can read yours at /robots.txt. A missing robots.txt is fine, by the way: crawlers treat that as permission to crawl everything, which is a safe default for most small sites.
Why robots.txt is not a privacy tool #
This deserves its own section because the mistake is common and the consequences are serious. Disallowing a path in robots.txt does not hide it — it does the opposite of hiding it. The file is publicly readable, so listing /internal-pricing/ or /staging/ in Disallow lines literally publishes a map of what you consider sensitive; attackers routinely read robots.txt files for exactly this reason. Meanwhile the blocked URLs can still end up in Google's index if other sites link to them, appearing as bare URLs without descriptions. And robots.txt does nothing whatsoever against scrapers, vulnerability scanners, or anyone who simply types the URL. Actual access control requires actual authentication: password protection, IP restrictions, or server-level rules. The correct pairings: content that must be private gets authentication; pages that may be crawled but should not appear in search results get a noindex tag; only wasteful-but-harmless URL spaces get robots.txt. If a URL would embarrass you on a search results page, robots.txt was never the right tool.
A sane robots.txt for most small business sites #
Most small business sites need a robots.txt that is short, boring, and hard to get wrong. The example here reflects what we deploy for a typical WordPress site: allow everything by default, block the admin area while explicitly allowing admin-ajax.php (which front-end features legitimately call), keep crawlers out of internal search results and checkout flows, and declare the sitemap location. What is deliberately absent matters as much: no blanket blocks on /wp-content/ or /wp-includes/, which would cut Google off from your CSS and JavaScript, and no attempt to enumerate private things. One syntax note that bites people: Disallow: / with nothing after the slash blocks your entire site — a single character separates normal operation from total deindexing, which is why every edit deserves a second look. If you would rather not hand-write it, our free robots.txt Generator builds a correct file from a few plain-English questions and flags dangerous patterns before you deploy.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://yourbusiness.com/sitemap.xmlWhat about AI crawlers like GPTBot and ClaudeBot? #
A newer question in every audit: AI companies now run crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI training control), among others — and the reputable ones honor robots.txt, so you can allow or disallow them by user-agent like any other bot. Whether you should block them is a genuine business decision, not a technical default. Blocking training crawlers keeps your content out of future model training; it may also reduce your visibility in AI assistants and AI-powered search experiences, which are a growing discovery channel — for a local business that lives on being found, we generally lean toward allowing them, but it is your call. Note that Google-Extended controls AI training use without affecting normal Google Search crawling. You may also encounter llms.txt, a proposed convention for offering AI systems a curated, markdown-friendly summary of your site. It is early and adoption is not guaranteed, but it is cheap to add and signals nothing negative — a reasonable low-priority experiment.
Common mistakes: blocking CSS, stale sitemaps, and more #
The recurring failures we find in technical audits, in rough order of damage: 1) Disallow: / left over from a developer's staging environment after launch — the site quietly falls out of the index while everyone wonders why traffic died. 2) Blocking CSS and JavaScript paths; Google renders pages like a browser, and blocked assets can make a fine mobile site look broken to mobile-first indexing. 3) Using robots.txt to remove already-indexed pages — the block prevents Google from seeing the noindex, freezing the URL in the index instead. 4) Stale or broken sitemaps full of redirects, 404s, and noindexed URLs, which erode crawler trust and bury real errors in noise. 5) Sitemaps listing http:// or non-canonical www variants after a migration. 6) Multiple conflicting sitemaps from stacked SEO plugins. Almost all of these are silent — the site looks normal to humans while crawlers hit walls. A quarterly check of Search Console's page-indexing report catches every one of them early.
Submitting to Search Console, and when to get help #
Google Search Console is free and is where this all becomes measurable. Verify your site, open Sitemaps, submit your sitemap URL once, and Google will refetch it automatically from then on. The Pages (indexing) report then shows exactly how many URLs are indexed and, more usefully, why others are not — Excluded by noindex, Blocked by robots.txt, Crawled, currently not indexed each point to a specific fix. The URL Inspection tool tests any single page live, and Bing Webmaster Tools deserves the same one-time setup. For a typical small business site, this is an hour of setup and a quarterly fifteen-minute review. Bring in help when the symptoms outrun the tooling: pages that stay unindexed for months, an indexing collapse after a redesign or migration, or plugin conflicts generating contradictory signals. Technical SEO cleanup — sitemap hygiene, robots rules, redirect mapping — is part of our local SEO service, and our free Website Grader plus our free robots.txt Generator will tell you quickly whether the basics are sound.
FAQ
Do I need both a sitemap and a robots.txt file?
They do opposite jobs, so most sites benefit from both. The sitemap advertises the pages you want crawled and indexed; robots.txt marks the paths crawlers should skip. A small site can survive without either — Google discovers pages through links — but the pair gives you explicit control and better diagnostics in Search Console.
Will submitting a sitemap get my pages indexed faster?
It speeds discovery, which is the first step, and an accurate lastmod date helps Google recrawl updated pages sooner. But indexing is a quality decision: Google still evaluates each page and skips thin or duplicate content regardless of the sitemap. Think of it as making sure Google knows your pages exist, not as a ranking lever.
Can I use robots.txt to hide a page from Google?
No — this backfires. Blocking a URL stops Google from crawling it, but the URL can still be indexed from external links, and Google can never see a noindex tag on a page it cannot fetch. Use a noindex meta tag for keep-out-of-search pages, and real authentication for anything genuinely private.
Should I block AI crawlers like GPTBot in robots.txt?
It is a business decision. Blocking GPTBot, ClaudeBot, or Google-Extended keeps your content out of AI training, but may reduce your visibility in AI assistants and AI search features that customers increasingly use for discovery. For local businesses that depend on being found, we generally recommend allowing them, and revisiting as the landscape evolves.
How often should my sitemap update?
Automatically, every time content changes — which is how modern CMS platforms and SEO plugins already behave. You submit the sitemap URL to Search Console once; Google refetches it on its own schedule. What needs periodic human attention is accuracy: a quarterly check that it contains no redirects, 404s, or noindexed URLs keeps it trustworthy.
What happens if I have no robots.txt at all?
Crawlers treat a missing robots.txt (a 404 at /robots.txt) as permission to crawl everything, which is a safe default for most small sites. The riskier situation is a server error at that URL — if /robots.txt returns a 5xx error, Google may pause crawling the site entirely until it can read the file.
Was this helpful?