Illustration What is crawling?
Jesse Commandeur

Jesse Commandeur

Content & Authority Specialist
26-7-2022 Google,

What is crawling? 

A website crawl is a process by which search engines such as Google, Bing and Yahoo! onder search the content of your website and index it in search results. A crawl can be performed manually or automatically. The latter happens when you create a robot.txt-file on your site and the crawler knows not to open certain directories. When crawling a website, the crawler prioritises certain pages, such as home, category or contact pages. More on crawling a website in this article. 

What is a website crawl?

A website crawl is a process in which a bot/spider discovers what your website has to offer. It looks at your site's code and content to find out what URLs your website has. The search engine then indexes this information, causing your website to appear in search results. Read more about the different types of crawlers in our article on that.

HTML crawl

A website crawl is also called an HTML crawl because it is performed by software that reads your site's HTML code to find out what the site is about. It is important to keep in mind that the website crawl happens before your site appears in search results. It can take several days - up to several months - for a new website to appear in search engine results. Do you think it takes too long for a page to be indexed? Then you can add your new pages to Google Search Console, to prioritise URLs for indexing in the Google SERPs. . 

Why is a website crawl important for SEO?

A website crawl is important because it allows search engines to discover and index your site. If your website unable to find is for search engines, you won't get visitors from organic searches. That is why it is important that your website is crawlable. If your website is easily 'crawlable', it will be easier for search engines to find your website and index new pages. Want to know more? Then also read the knowledge base article: Understanding how Google indexes your website, or our blog: Index coverage in Search Console: your website's SEO control room.

Bringing problems to light 

A website crawl can also reveal problems with your site. If the crawler does not notice your images or content, it could be a sign that they are not optimised. This can have a negative impact on SEO. You get insight into technical flaws around your website with a website crawl. In this case, you perform a website crawl yourself with, for example Deepcrawl or ScreamingFrog and it's not the search engines crawling your website. Especially with SEO migrations we constantly crawl websites for new insights, for example to see if all the 301 redirects are set properly and there are no 404 pages be more. 

How to make your website crawl better?

The first thing you should do is create a sitemap. This will help the crawler easily discover your site and its content. When it comes to content, you need to make sure it is both useful for your visitors and optimised for the search engines. You can do this onder other things by making sure your content is unique. So copying and pasting content from other websites is not a good idea and can cause problems. You are then talking about duplicate content. This is detrimental to SEO. 

Avoid language errors

Besides the fact that your content should be unique, it is also important that your website has as few language errors as possible. The site also had to be easy to read. This makes it easier for the Googlebot to see what your website is about. Also, a minimum length of 300 words per page is recommended, so that Google has enough context when onderwerp your webpage. Otherwise, you will have to deal with thin pages. Want to know how to write good SEO text? Then check out our blog: Writing for SEO in 6 steps.

Discovery and indexing 

A website crawl is a very important aspect of any website. It allows search engines to discover and index your site, which is necessary for getting organic traffic. Want your website to be indexed quickly and properly? Then create a sitemap, optimise your page and make sure your website is a reliable source of information. At SEO agency Onder we are happy to help you optimise your website and give you SEO advice, so you will rank better and get more organic search traffic. 

Need SEO advice? Get in touch with us!

Would you like us to give you SEO advice, for example for a website migration? Then take a quick look at our SEO services page. Besides technical SEO knowledge, we also employ experienced content and link-building specialists at SEO agency Onder. Wondering what we can do for you? Then do a live Live SEO analysis

As a SEO agency, we are experts in optimising websites for search engines and offer comprehensive training. Are you interested in a live SEO analysis To get instant insight into the performance of your website? Our specialists can help you with a technical SEO analysis, paying attention to issues such as load time, URL structure and mobile-friendliness.



Any questions?

Get in touch! :) we are happy to help you with all your SEO questions

SEO question

Don't miss it

Subscribe to our newsletter and don't miss a single SEO tip

"*" indicates required fields

This field is for validation purposes and should be left unchanged.

Any questions?

Do you want to know more about SEO? Could your website use a new boost?
Get in touch

This is us!

A team of SEO specialists

Our team consists of enthusiastic SEO specialists who are happy to work with you for the best results.

Meet our team

These customers are supupside down from us!

Renzo van Dongen

Renzo van Dongen

Digital content manager Peugeot

Whether it's about content or technical adjustments, it doesn't matter. Martijn briefs this in a very clear way so that it can be implemented locally or by our head office in France. [...] Since the start of our collaboration, we have achieved great results. Keep it up!

Marien van Stegeren

Eigenaar Kitcentrum

When your online onder venture grows, expectations often grow too, in every area. To a large extent, we were in control of SEO and its interpretation ourselves. We found out that our own "general view" was no longer sufficient, so we engaged Bureau Onder. Surprising and clear. We have built up a fine collaboration, partly because Martijn also regularly visits our office with Eline or Mariska (depending on the case), which is very valuable to us!

Marten Stellingwerf


Martijn and Mariska make the SEO story understandable for the layman and are happy to think along with us. It is nice that there is a personal contact moment every month with an extensive report and the opportunity to talk everything through. After just a few months, Onder has already managed to achieve number 1 positions on Google with some of our webshop pages. We are very satisfied with Onder.

Mark Prummel

SEO Specialist Stella Fietsen

For several years, Onder has been a valued partner of Stella. Martijn is my sparring partner for complex SEO issues and the extra pair of critical eyes that keeps us on our toes. Together, we have achieved nice growth in online visibility, visitor numbers and leads. Onder onder stands out for its short lines of communication and professionalism. They speak our language: don't nag, get down to business. That is what we love!


Don't miss a single SEO insight
  • Insights on SEO
  • For beginners and advanced specialists
  • From experiments to developments at Google and from
    best practices to our own cases
  • A look behind the scenes at Onder

"*" indicates required fields

This field is for validation purposes and should be left unchanged.