What Are Meta Robots Tags?
Meta robots tags are HTML instructions placed in a page's head that tell search engine crawlers how to index the page and handle its links. Common directives include index or noindex, which control whether the page appears in results, and follow or nofollow, which control whether links on the page pass signals. They give page-level control over crawling behavior and can also be delivered as an X-Robots-Tag HTTP header for non-HTML files.
- Syntax
- <meta name="robots" content="noindex, nofollow"> in the head (Google Search Central)
- Default behavior
- With no tag, crawlers assume index, follow (Google Search Central)
- Bot-specific targeting
- Use name="googlebot" to target Google specifically instead of all bots (Google Search Central)
- Header equivalent
- X-Robots-Tag delivers the same directives for PDFs and other non-HTML files (Google Search Central)
What is a meta robots tag? #
A meta robots tag is a line of HTML in a page's head section that gives search engine crawlers page-level instructions about indexing and link handling. It sits alongside other meta tags like the description and viewport, and its content attribute holds one or more directives separated by commas. The name attribute usually reads robots to address all crawlers, but it can name a specific bot such as googlebot to target Google alone. If a page has no meta robots tag at all, crawlers apply the default of index, follow, meaning they will list the page and follow its links. The tag matters because it lets you override that default on a per-page basis without touching server files or affecting the rest of the site. It is one of the most direct levers in technical SEO, and getting it right across a template-driven site is a core part of any build handled through /services/web-design, where each page type needs a deliberate indexing decision before launch.
What are the main meta robots directives? #
The two most important directives are the index pair and the follow pair. Index means the page may appear in search results, while noindex keeps it out. Follow means crawlers should follow the links on the page and pass signals through them, while nofollow tells them not to. These combine, so noindex, follow keeps a page out of results but still lets its links pass value, a common setting for utility hub pages. Beyond these, several useful directives exist. Noarchive prevents a cached copy from being shown. Nosnippet suppresses the text snippet and preview in results. Max-snippet, max-image-preview, and max-video-preview set limits on how much of your content can appear in a result or rich preview. Notranslate discourages offering a translated version, and noimageindex stops images on the page from being indexed. Most local sites need only index, noindex, and occasionally the snippet controls, but knowing the full set helps when a specific presentation problem arises.
Meta robots tag versus robots.txt: what is the difference? #
These two mechanisms are constantly confused because both involve robots, yet they operate at different stages and are not substitutes for each other. Robots.txt is a single file at your domain root that controls crawling, telling bots which URLs they may request before they fetch anything. A meta robots tag lives inside an individual page's HTML and controls indexing and link handling after the page has been fetched. The critical consequence is that if you block a URL in robots.txt, crawlers never fetch the page and therefore never see its meta robots tag, so a noindex directive on a robots-blocked page is silently ignored. To reliably keep a page out of results, you must leave it crawlable and use noindex, not block it in robots.txt. The interplay between crawl directives and index directives trips up many site owners, and the full picture is laid out in /wiki/sitemaps-and-robots-txt-explained alongside the related meta robots concepts.
How do you implement meta robots tags? #
Implementation depends on how your site is built. On a hand-coded or template-driven site, you add the tag directly to the head of the relevant template or page. On WordPress, SEO plugins expose per-page toggles for indexing and following, writing the tag for you, which is why sites built through /services/wordpress-development rarely need manual edits for routine cases. For non-HTML files such as PDFs, spreadsheets, or images, there is no head to hold a meta tag, so you deliver the same directives through the X-Robots-Tag HTTP response header configured on the server. A frequent oversight is JavaScript that injects or changes a meta robots tag after the initial HTML loads; search engines that render the page will honor the final rendered value, so a noindex added by script is fully binding and easy to miss when auditing raw source. Always verify the rendered tag, not just the source, when confirming a page's directives.
<!-- Keep a thank-you page out of results but pass link signals -->
<meta name="robots" content="noindex, follow">
<!-- Index normally but limit the text snippet length -->
<meta name="robots" content="index, follow, max-snippet:160">
<!-- Target only Google, allowing a large image preview -->
<meta name="googlebot" content="index, max-image-preview:large">
<!-- Prevent a cached copy and hide the snippet -->
<meta name="robots" content="noarchive, nosnippet">When should you use noindex, follow? #
The noindex, follow combination is one of the most useful and misunderstood settings. It keeps a page out of search results while still allowing crawlers to follow its links and pass ranking signals to the pages those links point to. This is ideal for hub or utility pages that have little search value themselves but connect to pages that matter. Classic examples include certain paginated archives, internal category listings you would rather not rank, and filter pages that aggregate links to valuable products or articles. By using noindex, follow instead of noindex, nofollow, you avoid bottling up internal authority behind a hidden page. That said, Google has noted that a page kept noindexed for a very long time is crawled less often, and its links may eventually be treated more like nofollow simply because the page is rarely visited, so do not rely on a permanently hidden page as a critical link conduit. When internal links matter, the cleaner answer is often to rethink architecture, which is where /services/conversion-optimization audits add value.
What are the snippet control directives? #
Beyond indexing, meta robots tags let you shape how your content appears in results. Nosnippet removes the descriptive text snippet entirely, which is rarely desirable but occasionally used for sensitive content. Max-snippet sets a character limit on the snippet, letting you cap how much text Google shows. Max-image-preview controls the size of any image preview, with values of none, standard, or large, and setting it to large is common for content you want to surface prominently in visual results and rich features. Max-video-preview limits video preview length in seconds. Noarchive prevents Google from showing a cached version of the page. These controls matter most for publishers and content-heavy sites, but even local businesses use max-image-preview large to encourage richer presentation of service photos and portfolio images. Because these directives influence how AI-driven results and previews draw on your content, they intersect with the concerns covered in /wiki/what-are-ai-overviews, where controlling how much of your page is surfaced can be a deliberate strategic choice.
How do you audit meta robots tags across a site? #
Auditing meta robots at scale means crawling every URL and recording its indexability, not checking pages one at a time. A crawler reports each page's robots directives, flags conflicts such as a noindexed page included in your sitemap, and highlights pages that should be indexed but carry an accidental noindex. Search Console's URL Inspection tool confirms Google's own view of a specific page, including whether a robots directive is keeping it out of the index. Our /tools/website-grader surfaces indexability signals across a site quickly, which is the practical starting point for most owners. Two failure modes deserve special attention. First, a site-wide noindex accidentally left on after launch, which can erase an entire business from search. Second, JavaScript-injected directives that differ from the raw HTML, requiring you to check the rendered page. Regular audits, built into ongoing /services/care-plans, catch these before they quietly cost visibility, since a single misapplied template directive can affect thousands of pages at once.
Why do meta robots tags matter for local businesses? #
For a local business, meta robots tags decide which pages compete in local search and which stay hidden. Service pages, location pages, and the homepage must be indexable, while thank-you pages, internal search results, thin tag archives, and staging copies should carry noindex so they do not dilute the site with low-value URLs. The stakes are concrete: a template that applies noindex too broadly can knock ranking service pages out of results, while forgetting to noindex junk URLs floods the index and wastes crawl budget on pages no customer wants. Because these sites are often built quickly and handed between developers, leftover directives from a build or migration are a recurring hazard. Teams behind industry pages such as /web-design-for-roofers and /web-design-for-law-firms treat the indexing decision for each template as a launch requirement, pairing it with strong /services/local-seo so that every page competing in search is one a customer would be glad to find, and nothing else clutters the results.
FAQ
What is the default if there is no meta robots tag?
If a page has no meta robots tag, search engines apply the default of index, follow. That means the page is eligible to appear in results and crawlers will follow its links and pass signals. You only need a meta robots tag when you want to change this default behavior for a specific page.
Can I use meta robots and robots.txt together?
Yes, but carefully. Robots.txt controls crawling and meta robots controls indexing, so they work at different stages. Never block a page in robots.txt that carries a noindex tag, because the crawler will not fetch the page and will never see the noindex, leaving it potentially still listed in results.
What does noindex, nofollow do?
Noindex, nofollow keeps the page out of search results and tells crawlers not to follow its links or pass signals through them. Use it sparingly, for pages with no search value whose links also lead nowhere useful, such as certain login or account areas. For most hidden pages, noindex, follow is the better choice to preserve link flow.
Do meta robots tags work for PDFs?
Not directly, because PDFs and other non-HTML files have no head section to hold a meta tag. To apply the same directives, use the X-Robots-Tag HTTP response header configured on your server. It supports noindex, nofollow, and the other directives, delivering identical instructions to crawlers for downloadable files.
Will a nofollow directive stop Google finding my links?
A page-level nofollow tells Google not to follow links on that page or pass signals through them, but Google may still discover those URLs through other links elsewhere on the web. Nofollow controls signal passing and following intent, not absolute discovery. To keep a destination out of the index, apply noindex on that destination page.
How do I check what meta robots tag a page has?
View the page source and search the head for a robots meta tag, use browser developer tools to inspect the rendered head, or run Search Console's URL Inspection tool for Google's own view. Watch for JavaScript that changes the tag after load, since the final rendered value is what search engines honor.
Was this helpful?