What Is a Crawl Error?
A crawl error occurs when a search engine tries to access a page on your site but cannot, preventing it from being read and potentially indexed. Errors fall into two groups: site-level problems that block access to the whole site, such as DNS or server failures, and URL-level problems affecting individual pages, such as 404s, server errors, or redirect issues. Google reports crawl errors in Search Console, and unresolved ones can keep pages out of search results.
- Two categories
- Site errors affect the whole site; URL errors affect individual pages (Google Search Central)
- Where reported
- Google Search Console's Page indexing and Crawl stats reports (Google Search Central)
- Common causes
- DNS failures, server timeouts, robots.txt blocks, 404s, and 5xx errors (Google Search Central)
- Consequence
- Pages that cannot be crawled cannot be indexed or ranked (Google Search Central)
What is a crawl error? #
A crawl error is what happens when a search engine's crawler, such as Googlebot, attempts to reach a page on your site and fails. Crawling is the first step in how pages get into search results: a bot must successfully fetch a page before that page can be read, indexed, and ranked. When a crawl attempt fails, the page may never enter the index, or an already-indexed page may eventually be dropped if the error persists. Crawl errors come in two broad categories. Site-level errors affect access to the entire site, meaning the crawler cannot reach your server at all, which is the more serious case. URL-level errors affect specific pages while the rest of the site remains reachable, such as an individual page returning a 404 or a server error. Google surfaces these problems in Search Console, giving site owners a direct window into what the crawler is experiencing. Diagnosing and clearing crawl errors is a foundational technical SEO task and a standard part of the health monitoring in /services/care-plans.
What are site-level crawl errors? #
Site-level crawl errors are the most serious because they block a crawler from reaching your whole site, not just one page. The main types are DNS errors, server connectivity errors, and robots.txt fetch failures. A DNS error means the crawler could not resolve your domain name to a server address, often due to a misconfigured or unreachable DNS setup; understanding this depends on how domains map to servers, explained in /wiki/what-is-dns. A server connectivity error means the crawler reached your DNS but the server did not respond in time, refused the connection, or was overloaded, which points to hosting problems, timeouts, or resource limits. A robots.txt fetch failure is subtle but important: if Google cannot retrieve your robots.txt file, for instance because the server returns an error rather than the file or a clean 404, it may pause crawling the whole site until it can confirm what it is allowed to access. Because these errors can take an entire site offline for search, they are top priorities in any /services/website-rescue engagement, usually tracing back to hosting or DNS configuration.
What are URL-level crawl errors? #
URL-level crawl errors affect individual pages while the rest of the site remains crawlable, and they are the more common day-to-day type. The most familiar is the not found error, where a page returns a 404 because it was deleted, moved without a redirect, or linked incorrectly. Server errors at the page level, in the 5xx range, mean the crawler reached the page but the server failed to deliver it, often due to a code bug, a plugin fault, or a resource limit hit under load. Redirect errors include redirect chains that grow too long, redirect loops that never resolve, and redirects pointing to broken destinations. Blocked-by-robots errors occur when a page you want indexed is disallowed in robots.txt. Soft 404s, where a missing page returns a 200 status instead of a proper 404, are a related URL-level problem. Access-denied errors from authentication requirements can block crawlers too. Because each type has a distinct fix, identifying the specific error is the first step, and tools like /tools/broken-link-checker and /tools/website-down-checker help pinpoint them.
How do you find crawl errors in Search Console? #
Google Search Console is the authoritative source for crawl errors because it shows exactly what Googlebot experiences on your site. The Page indexing report lists URLs Google could not index and groups them by reason, including not found 404s, server errors, redirect problems, and pages blocked by robots.txt, so you can see both the type and scale of each issue. The Crawl stats report, found under settings, shows crawl request volume over time and breaks responses down by status code and host status, revealing spikes in server errors or DNS problems that signal site-level trouble. The URL Inspection tool lets you test a single URL live, showing whether Google can currently fetch and index it and why not. Together these give a complete picture from the crawler's own perspective, which is far more reliable than guessing. For a quick independent read, our /tools/website-grader surfaces crawlability and status issues, and the interplay of crawl directives is covered in /wiki/sitemaps-and-robots-txt-explained.
How do you fix common crawl errors? #
Fixing crawl errors starts with matching each error type to its remedy. For not found errors, restore the page if it should exist, add a 301 redirect to a relevant replacement if it moved, or accept the 404 if the page is genuinely gone and no equivalent exists. For server errors, investigate hosting logs for the underlying fault, whether a plugin conflict, a code bug, or a resource limit, and stabilize the server, which often means upgrading or reconfiguring hosting. For redirect errors, collapse chains so each redirect points directly at its final destination and reconcile conflicting rules that cause loops. For robots.txt blocks on pages you want indexed, edit the file to allow crawling. For soft 404s, return the correct status code or add real content. After making fixes, use Search Console's validation feature to ask Google to recheck the affected URLs and confirm the errors clear. Because the causes span content, redirects, robots rules, and hosting, resolving them well often draws on both /services/managed-hosting and technical audit work.
# A reachable, correctly formatted robots.txt prevents
# whole-site crawl blocks. It must return 200, not an error.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Point crawlers to your sitemap for efficient discovery
Sitemap: https://example.com/sitemap.xmlHow do crawl errors affect indexing and rankings? #
Crawl errors matter because crawling is the gateway to everything else in search. A page that cannot be crawled cannot be indexed, and a page that is not indexed cannot rank or appear in results at all, so an unresolved crawl error on an important page effectively removes it from search. For pages already in the index, persistent errors erode standing over time: if Google repeatedly fails to fetch a page, it may eventually drop it, and a pattern of errors across a site can slow the overall crawl rate as Google conserves effort on an unreliable site. Site-level errors are the most damaging because they can block crawling of the entire site, causing broad drops if they persist. Even scattered URL-level errors waste crawl budget and can bury genuinely broken pages that disappoint users. Because the consequences ripple from crawling through indexing to rankings, monitoring and clearing crawl errors is not optional maintenance but a core part of protecting search visibility, which is why it is built into ongoing /services/care-plans and /services/local-seo work.
What is the difference between a crawl error and an indexing issue? #
Crawl errors and indexing issues are related but distinct stages of the same pipeline. A crawl error means Google could not fetch the page in the first place, whether because the server failed, the URL returned an error, or robots.txt blocked access. An indexing issue means Google successfully crawled the page but chose not to index it, or indexed it in a way you did not intend. A page can be crawled fine yet excluded from the index because it carries a noindex tag, because Google selected a different canonical, because the content is judged thin or duplicate, or because it was classified as a soft 404. Search Console's Page indexing report actually blends both, listing pages that were not indexed and grouping them by reason, some of which are crawl failures and some of which are indexing decisions. Telling them apart matters because the fixes differ: a crawl error needs the access problem solved, while an indexing issue needs a directive changed or the content improved. This distinction connects to the meta directive concepts in /wiki/what-are-ai-overviews and canonical handling covered elsewhere in this technical series.
Why crawl errors matter for local business websites #
For a local business, crawl errors translate directly into lost visibility and lost customers. If a key service or location page cannot be crawled, it never appears in the local searches that drive calls and bookings, so a single unnoticed error can quietly remove a page that used to bring in work. Site-level errors are especially dangerous: a DNS misconfiguration or an overloaded server can make the entire site uncrawlable, and because these problems often coincide with the site also being down for visitors, they hit revenue and rankings at once. Local sites are prone to these issues because many run on budget hosting that struggles under traffic spikes, and many change hands between developers who leave behind stray robots.txt blocks or broken redirects. The remedy is proactive monitoring rather than reacting after rankings fall. Pairing reliable /services/managed-hosting with routine crawl-error checks through /services/care-plans keeps the site reachable, and teams building for verticals like /web-design-for-electricians and /web-design-for-auto-repair-shops treat clean crawlability as the foundation that all their local SEO stands on.
FAQ
Where do I see crawl errors for my site?
Google Search Console is the authoritative source. Its Page indexing report groups URLs Google could not index by reason, the Crawl stats report shows response codes and host status over time, and the URL Inspection tool tests individual pages live. Together they show exactly what Googlebot experiences, which is more reliable than external guesses.
What is the difference between a site error and a URL error?
A site-level error blocks the crawler from reaching your whole site, such as a DNS failure or an unresponsive server, and is the most serious kind. A URL-level error affects individual pages while the rest of the site stays crawlable, such as a single 404 or a page-level server error. Site errors take priority because they can affect everything.
Do crawl errors hurt my rankings?
They can. A page that cannot be crawled cannot be indexed or ranked, so an error on an important page effectively removes it from search. Persistent errors may cause already-indexed pages to drop, and a pattern of errors can slow your overall crawl rate. Site-level errors are especially damaging because they affect the entire site.
How do I fix a 404 crawl error?
Decide whether the page should exist. If it should, restore it. If it moved, add a 301 redirect to the relevant new URL. If it is genuinely gone with no equivalent, letting it return a 404 is correct. Then validate the fix in Search Console so Google rechecks the affected URLs.
Can robots.txt cause crawl errors?
Yes, in two ways. If Google cannot fetch your robots.txt file at all because the server errors, it may pause crawling the whole site. And if the file disallows a page you actually want indexed, that page is blocked from crawling. Ensure robots.txt returns a valid response and does not block important content.
How often should I check for crawl errors?
Review Search Console regularly, at least monthly for a stable small site and more often after any change like a redesign, migration, or hosting switch. Crawl errors can appear suddenly from a server issue, a broken redirect, or a stray robots rule, so ongoing monitoring catches them before they cost rankings or traffic.
Was this helpful?