What is Crawl budget? (Level up your SEO)

SEO

Crawl budget is important for your SEO-knowledge, but is often misunderstood.

You’ll hear people say things like ‘be careful of your crawl budget!‘ or ‘don’t publish too many articles, you might screw up your crawl budget!‘. ‘

But truth be told, most of these opinions are outdated and don’t reflect modern-day SEO.

Don’t believe me? Just ask John Mueller – Senior Webmaster Trends Analyst at Google.

BUT while crawl budget is (in the words oh @Johnmu) ‘over-rated’, it’s still an important part of SEO to understand, especially for people running new sites, or enormous sites (like eCommerce sites).

Here’s why…

So, what IS my crawl budget?

CRAWL!!!

Crawl budget refers to the number of pages the Googlebot crawls and indexes on your website during a set period of time.

Let’s break it down.

The ‘crawl’ part of crawl budget

In essence – everyday, Google sends out an army of robots (commonly referred to as ‘spiders’) who scour the internet looking for new or updated content.

When they do this, it is called ‘crawling’. To rank on search engines, your pages need to be crawled and indexed.

Crawling is how websites get found and then ranked on Google 🕷

The ‘budget’ part of crawl budget

As with any budget, your crawl budget is about resources. Because while Google is mighty and powerful, it still has its limitations (aka – the spiders can only crawl so much so fast!).

Crawl budget is directly linked with the resource a crawler uses on your site and your server capacity.

Usually crawl budget is nothing to worry about. However, in a few specific cases, it is essential to look at it.

When should you be concerned about crawl budget?

Popular sites don’t have to worry about crawl budget. This is because Google is familiar with them, and crawls them often.

The two sites that DO need to be worried about crawl budgets?

#1 – New websites that are not yet ranking on Google

Why?

That’s because even if a server has the capacity to support more crawling, when your site is new, search engine sites won’t usually crawl it. That’s simply because Google hasn’t assessed it as a reputable source just yet.

#2 – Enormous websites with hundreds of thousands or millions of pages

Why?

Because there might actually be so many pages, that Google actually misses some (or at least they take a while to get picked up).

Ways to verify crawl activity

So if you are a new website, an enormous, or just want to double check if your site is being crawled properly, the process is super simple.

To have an overview of Google crawl activity, head to Google’s Crawl Stats report. It is an authentic google report that helps you identify the crawling behavior, issues, and changes in crawling. Moreover, it also tells you how the Google search engine crawls your site.

If your site has a complex setup, you will have to access and store data from raw files and potentially use specific software such as Elasticsearch, Logstash, or Splunk.

Very helpful stuff!

How does Google adjust the site crawl?

No crawl budget is the same and various inputs have to be considered.

Crawl demand

Crawl demand is how much Google crawls your pages. It is based on the popularity of your pages and the performance of your content against Google’s index.

Optimized pages with links get a better index than the others.

Keep in mind that Google is indexing all the pages of all the websites of the internet.

To stand out you must make sure your pages are search-engine optimized and regularly updated.

Crawl rate limit

Crawl rate limit refers to the maximum of crawling your website can keep up with before it alters the server stability.

Google will automatically adjust the crawling limit based on your website’s abilities to handle it.

How to make Google crawl your website faster

To improve your site crawl’s demand and limit, follow the steps below:

Increase the number of resources and speed up your server

To increase the resources Google downloads when crawling your site and impact your crawl budget, make it easy for Google to connect and download resources related to the server.

Crawl demand depends on your page popularity or the quality of the links. To increase your crawl budget, increase the number of both internal and external links on your website’s pages.

If you don’t know what links to add, head to Site Audits to find the best links opportunities.

Having restricted and broken links on your website impacts your crawl budget negatively. Do an audit and clear all the issues relevant to linking and replace them with high-priority links. It will boost the crawl budget of your website.

Use the “All issues” report in Site Audits to identify broken and redirected links.

Use GET instead of POST where you can

This one involves HTTP Request methods. It consists of using GET requests as much as possible. That is because GET (pull) requests are cached and POST (push) requests are not.

Use the Indexing API

Check if your pages are eligible for the Google Indexing API. This will let Google know automatically when pages are deleted or added to your website. This technique is only valid for job posting, broadcast events, and live videos type services.

How to slow down Google crawl?

There are limited but practical ways to make Google crawl slower. But, first, you need to make a few adjustments to your website.

Below are our recommended adjustments:

  • Use the rate limiter tool in Google Search Console for slow but guaranteed results.
  • Use Google’s crawl rate by creating ‘503 Service Unavailable’ or ‘429 Too Many Requests’ status codes on your pages for fast but risky results.

Final thoughts

If you’re having an issue with pages indexation or crawling, be sure to reach out to us to speak with a digital strategist. They can walk you through the entire process, and make a suggestion for how to improve your crawl budget (and rankings) moving forward.