Google says a known URL is typically recrawled (or sometimes known as refreshed) every ~30 days.
On weareroast.com, our 178 URLs had gone an average of 39.2 days since Google last crawled them, checked every day for 30 days.
At Search Central Live Deep Dive in Barcelona, Gary Illyes shared Google’s own data on how long crawling, indexing and serving take. In this session called How Long does it take to..? it was the first time I’ve seen Google put a typical and slowest time against most of these processes.
The catch is that the number Gary explained are averages across the whole web. A 20 page brochure site and a publisher with 2 million URLs both feed into the same number.
So I wanted to share what it looks like on a real site. Ours.
What Google shared in Barcelona
Gary’s session on day 3 was called “How long does it take to…?” and it was the highlight of the event for me. I covered the full set of tables in my Day 3 recap, so here I’m only pulling out the crawling numbers.
| Process | Typical | Slowest |
| Discovery (new URL) | ~20 hours | Weeks to never |
| Refresh (known URL) | ~30 days | Weeks to never |
| Sitemap processing | ~24 hours | Up to 14 days, or never (quality) |
| Crawl demand update | ~20 hours | Weeks to months |
The row this article is about is refresh. That is how long Google takes to come back to a page it already knows about.
Gary’s caveat was that these processes are linked. E.g. a page can’t be reindexed until it has been recrawled, so a slow recrawl delays everything after it. If you want the background on how crawling works, my Day 1 recap on crawling and Day 2 recap on indexing go through it.
Why an average only tells you so much
An average of ~30 days hides a huge range. Some pages get crawled several times a day, others go months without a visit.
Google decides how often to come back based on two things: how much it can crawl your site without hurting the server (crawl capacity), and how much it wants to (crawl demand). Google explains both in its crawl budget documentation.
Every site is different on both. The main differences are
- Size: a site with millions of URLs has to compete with itself for Googlebot’s time.
- Age: an established domain with a history of useful content has built up demand.
- Performance: a fast, stable server lets Google crawl more without backing off.
- Demand: pages people search for and link to get revisited more.
E.g. a homepage that changes every day and has hundreds of links pointing at it will be treated very differently to a 2019 blog post nobody links to. Both still count towards a single site-wide average.
That is why I wanted to break our site down by section rather than stop at one number.
How we gathered the data
We used ROAST Index Checker, an internal tool we built in ROAST Labs. It runs every URL on a site through Google’s URL Inspection API in Search Console every morning.
The API returns the same information you see when you inspect a URL by hand. The field we care about is lastCrawlTime, the date and time Googlebot last fetched the page.
The process is
- Add in URLs – either upload a list or pull every URL from the XML sitemap.
- Inspect every URL through the API every day and store the last crawl date.
- For each daily check, work out the crawl lag. That is the number of days between the check and Google’s last crawl.
- Average the daily lag for each URL over 30 days, then group the URLs by folder, e.g. /services/ or /news/.
We ran it daily for the last 30 days across 178 URLs. That gave us 5,306 samples. The URLs didn’t change e.g. added or removed.
The daily checks are what make this work. The API only returns the most recent crawl, not a history, so one check is a snapshot. Thirty in a row show the lag building up and resetting each time Google comes back.
How to read crawl lag. It’s the average number of days since Google last visited, not the gap between visits. E.g. if a URL is crawled on days 1, 15 and 30, the lag climbs from 0 to 13, resets, climbs from 0 to 14, then resets again. That averages 6.5 days, even though the gap between crawls is ~14.5 days. So crawl lag is roughly half the gap between visits, and a lag above ~15 days suggests Google is coming back less often than its typical ~30 day refresh.
One caveat. If Google crawls a page more than once in a day we only see the latest recrawl.
The API also has a daily limit per Search Console property. On a site our size, checking every URL each day fits within it. On a site with hundreds of thousands of URLs we check a sample each day instead.
The overall average for weareroast.com
Across the whole site, the average crawl lag was 39.2 days across the 178 pages.
The trend over the month matters too. The site-wide lag went from ~36 days on 4 September to ~42 days by early October. On average, our pages are ageing faster than Google is coming back to them. This will be due to some of the pages being news articles, there is less need to recrawl these as they tend not to update. (Remember this is a fixed set of 178 URLs so no new articles added during the 30 days)
Using the rule of thumb above, a page recrawled every ~30 days would show a lag of ~15 days. So across the site as a whole, Google is coming back to our pages less often than its typical refresh time.
The spread is the interesting part. The homepage had a lag of 1.6 days, so Google is back almost every day. Our careers pages averaged 73.8 days.
Only 6 of our 14 folders came in under 20 days. All six are single pages linked from the navigation or footer on every page of the site.
Section by section
This is where the averages split apart. The sections below run from fastest to slowest.
Homepage
Crawl lag: 1.6 days (1 URL)
Google is back to the homepage almost every day. It changes the most and every other page on the site links to it.
Core pages
Section average: 20.3 days (7 URLs)
- About us: 8.2 days
- Contact us: 11.7 days
- ROAST Academy: 13.5 days
- ROAST Labs: 17.4 days
- Our People: 17.6 days
- B2B Marketing: 32.5 days
- Privacy policy: 41.3 days
The top five are all single pages linked from the navigation or footer on every page.
The privacy policy is linked from every page too, but it rarely changes. That is a good example of links not being enough on their own. If Google keeps finding the same page, it slows down.
B2B Marketing is the odd one out. It sits in the main navigation, so links don’t explain the 32.5 days. That is one we want to look at – how can we push more demand or add new content to the page.
Services
Section average: 25.9 days (39 URLs)
Our services pages are the fastest of the multi-page sections. They are linked from the main navigation, but change less often than news.
News
Section average: 38.6 days (60 URLs)
News is where we publish most, so I expected it to be one of the fastest. It isn’t – But it depends.
My guess is new articles get picked up quickly, then older ones slip down the news hub as new posts push them off the first page. Many of these are older so don’t expect to be recrawled as often. Also… we have a click to load more rather than traditional pagination on the main news page.
Author pages
Section average: 43.6 days (13 URLs)
Author pages only change when that person publishes something new, so there is little reason for Google to come back often.
Work
Section average: 43.7 days (18 URLs)
Once a case study is published, it rarely changes. Google treats it the same way.
Resources
Section average: 49.6 days (33 URLs)
Resources is the slowest content section on the site.
Careers
Section average: 73.8 days (7 URLs)
Careers was the slowest section on the site, almost 2.5 times the site average. The careers page is in the main navigation, so this is another one we want to dig into.
Crawl lag by folder / pages
| Folder | Section | URLs | Daily samples | Avg crawl lag (days) |
| / | Homepage | 1 | 30 | 1.6 |
| /about-us/ | Core pages | 1 | 30 | 8.2 |
| /contact-us/ | Core pages | 1 | 30 | 11.7 |
| /roast-academy/ | Core pages | 1 | 30 | 13.5 |
| /roast-labs/ | Core pages | 1 | 30 | 17.4 |
| /our-people/ | Core pages | 1 | 30 | 17.6 |
| /services/ | Services | 39 | 1,170 | 25.9 |
| /industry/ | Core pages | 1 | 30 | 32.5 |
| /news/ | News | 60 | 1,798 | 38.6 |
| /privacy-policy/ | Core pages | 1 | 30 | 41.3 |
| /author/ | Author pages | 13 | 389 | 43.6 |
| /work/ | Work | 18 | 520 | 43.7 |
| /resources/ | Resources | 33 | 979 | 49.6 |
| /careers/ | Careers | 7 | 210 | 73.8 |
| Whole site | 178 | 5,306 | 39.2 |
Data from ROAST Index Checker via the Search Console URL Inspection API, checked daily for 30 days. The whole-site figure is weighted by samples.
What changes how often a page gets recrawled
Your average isn’t fixed. It moves with how the site is built and how people use it.
The factors that move it are as follows:
- Domain age and history: an older site with a track record of useful content has built up crawl demand that a new site hasn’t.
- How often you update content: if Google sees a page change every time it visits, it learns to come back sooner. If nothing changes for months, it slows down. E.g. a news hub that updates daily vs a privacy policy that changes once a year.
- Publishing frequency: a site that adds new pages every week gives Googlebot a reason to check in more often.
- External links: pages with more links from other sites get discovered and revisited faster. This is one of the reasons digital PR has a technical benefit as well as a brand one.
- Internal links: a page linked from the main navigation gets crawled far more than one buried four clicks deep.
- Demand: pages people actually search for and click on get more attention.
- Server speed and reliability: Gary was clear that crawl capacity can drop in seconds if your server struggles, and takes 1 to 3 weeks to recover. Slow response times and 5xx errors both cause Google to back off.
- Sitemaps and lastmod: an accurate lastmod date in your sitemap tells Google what has changed. Google’s sitemap documentation says it uses lastmod when it is consistently accurate.
- Wasted crawl: redirect chains, duplicate URLs and parameter pages all use crawl time that could have gone on pages that matter.
- Quality: the word “never” came up a lot in Gary’s tables, and it was usually tied to quality. Fast technical fixes don’t help if Google doesn’t think the page is worth coming back to.
Why is crawl refresh important?
Getting Google to recrawl your pages more often won’t make you rank better on its own. Google has said crawling is not a ranking signal. It is a step every page has to go through before it can rank, not a reason it ranks higher.
What it does change is how long you wait. Google can only index what it has crawled, so every improvement you make sits in a queue until Googlebot comes back. On a section with a 40 day lag, that wait could be a month or more.
That matters most when
- You improve a page: better content, title tags or internal links do nothing in search until the page is recrawled and reindexed.
- You fix a problem: a noindex left on by mistake, a broken canonical or a redirect all stay as they were in Google until it comes back.
- Information goes out of date: prices, offers, stock and dates can show in search long after you have changed them.
- You remove or merge pages: old URLs keep showing until Google sees the 404, 410 or redirect.
- You are testing changes: if you don’t know when Google saw the change, you can’t tell whether a result is down to your change or something else.
- AI search picks up your content: AI Overviews and AI Mode run on the same crawling and indexing as Google Search, so an old version of a page is the version they use.
E.g. if you rewrite a services page on 1 March but Google doesn’t recrawl it until 10 April, you have lost nearly six weeks before the change can make any difference.
What about your domain?
Your numbers will be different to ours. Size, age, how often you publish, links and server setup all move them, so neither Google’s averages nor ours will tell you what is happening on your site.
The only way to know is to gather the data for your own domain.
To make that easier, we’ve built a free version of ROAST Index Checker that runs in Google Sheets and Apps Script. The Google Inspection API tool checks your URLs through the URL Inspection API on a daily schedule and logs the last crawl date every time, so you can build the same picture over time.
Setting it up takes about 15 minutes
- Make a copy of the Google Sheet template.
- Link a Google Cloud project with the Search Console API enabled.
- Run Setup, pick your Search Console property and import your sitemap.
- Leave the daily trigger running, then read the data back after 30 days.
Only caveat is the API limit of 2,000 URL checks per property per day. Under 2,000 pages you can check everything daily. Above that, split the site across the week or prioritise by section, e.g. the homepage and category pages daily, older blog posts weekly.
If you want help reading the results, or want us to run it as part of our technical SEO work, get in touch.
