Knowledge base header crawlers

Hinke Oevering

SEO Specialist
17-10-2022 Google,

What types of crawlers are there?

When it comes to search engine crawlers and bots, you may think they are all the same and there is not much difference between them. However, there is much more to crawlers than meets the eye. In fact, there are several types of crawlers with different purposes and functions. They can be used for different purposes. Therefore, it is important to know them if you want your SEO strategy to be successful. Let's take a look at the different crawlers you may encounter.

What is a crawler?

A crawler is a computer programme that scours the Internet to find and index content. Such a programme can be started by a user, like a search engine, or run autonomously, like a web spider. A crawler can be a web crawler, a site crawler, an internet crawler or a bot. The internet crawler is a programme that scours the internet to find information. The site crawler is a programme that searches a website to find information. In our article on crawl read more on this.

Google crawlers

Google Crawlers are the bots that crawl and index your website in the search engine's results pages. They do this by reading and analysing your website's source code. In the process, they create a rich snippet in SERPs of your content and user-generated content. In other words, Google Crawlers are responsible for compiling your search engine result pages and represent the first step of your SEO. If your website is not properly indexed, the crawlers cannot find and rank it. For this, your SEO strategy does not matter otherwise. One of the most important things to keep in mind when it comes to Google Crawlers, is that there are more than one of them. Before you start optimising your website, you need to know which of these crawlers to target and why.

SEO bone

First on our list is the SEO Bot. This crawler is responsible for analysing your content and checking for grammar, spelling and readability issues. It also checks for plagiarism and verifies whether your content is original or copied from another source. This is a very important onder part of the overall SEO process, as search engines penalise websites with copied or plagiarised content. The SEO Bot also checks the structure of your website and your internal link structure. In addition, it checks the loading speed of your website. This is crucial, as Google crawlers automatically penalise websites with slow loading times.

RSS crawler

The RSS crawler goes through your blog feed and creates an index of your latest posts. This includes their metadata, such as the published date and titles. The crawler also finds images and rich snippets of your posts, as well as videos, podcasts and other media included in your blog feed. It is important to keep in mind that the RSS feed is not a substitute for a sitemap. A sitemap is designed to show crawlers the whole picture of your website, while a feed contains only your latest posts. The RSS feed can be found in the source code of your website, usually onderaan your homepage.


The Robo crawler is responsible for scanning your website's source code for broken links, broken images and other errors. It helps you fix these errors before the Google crawler can visit your website. This is important because broken links can seriously hurt your rankings. The Robo crawler also analyses the loading speed of your website. Remember well that the slower your website's loading speed is, the lower your rankings will be.


Probot crawls your website and analyses it to determine how often it is updated. Some people think that a website should be updated once a week, but this is not always the case. It all depends on the type of website you have and your goals. For example, if you sell products, you can update your website daily with new products. If you have a content website that regularly publishes a new article, you might want to update your website once a week. The purpose of the Probot crawler is to analyse your website and determine the best update schedule.

Want to know more about crawlers?

Crawlers are important tools in the SEO process, as they are the first to discover your website and decide whether it is worthy of listing in Google. With that said, you need to ensure that these crawlers find your website and that your website's source code is optimised for them. This can be done by using the right SEO techniques for your website. For example, use our technical SEO checklist or take contact on with us for tailored help.

As a SEO agency, we are experts in optimising websites for search engines and offer comprehensive training. Are you interested in a live SEO analysis To get instant insight into the performance of your website? Our specialists can help you with a technical SEO analysis, paying attention to issues such as load time, URL structure and mobile-friendliness.



Any questions?

Get in touch! :) we are happy to help you with all your SEO questions

SEO question

Don't miss it

Subscribe to our newsletter and don't miss a single SEO tip

"*" indicates required fields

This field is for validation purposes and should be left unchanged.

Any questions?

Do you want to know more about SEO? Could your website use a new boost?
Get in touch

This is us!

A team of SEO specialists

Our team consists of enthusiastic SEO specialists who are happy to work with you for the best results.

Meet our team

These customers are supupside down from us!

Renzo van Dongen

Renzo van Dongen

Digital content manager Peugeot

Whether it's about content or technical adjustments, it doesn't matter. Martijn briefs this in a very clear way so that it can be implemented locally or by our head office in France. [...] Since the start of our collaboration, we have achieved great results. Keep it up!

Marien van Stegeren

Eigenaar Kitcentrum

When your online onder venture grows, expectations often grow too, in every area. To a large extent, we were in control of SEO and its interpretation ourselves. We found out that our own "general view" was no longer sufficient, so we engaged Bureau Onder. Surprising and clear. We have built up a fine collaboration, partly because Martijn also regularly visits our office with Eline or Mariska (depending on the case), which is very valuable to us!

Marten Stellingwerf


Martijn and Mariska make the SEO story understandable for the layman and are happy to think along with us. It is nice that there is a personal contact moment every month with an extensive report and the opportunity to talk everything through. After just a few months, Onder has already managed to achieve number 1 positions on Google with some of our webshop pages. We are very satisfied with Onder.

Mark Prummel

SEO Specialist Stella Fietsen

For several years, Onder has been a valued partner of Stella. Martijn is my sparring partner for complex SEO issues and the extra pair of critical eyes that keeps us on our toes. Together, we have achieved nice growth in online visibility, visitor numbers and leads. Onder onder stands out for its short lines of communication and professionalism. They speak our language: don't nag, get down to business. That is what we love!


Don't miss a single SEO insight
  • Insights on SEO
  • For beginners and advanced specialists
  • From experiments to developments at Google and from
    best practices to our own cases
  • A look behind the scenes at Onder

"*" indicates required fields

This field is for validation purposes and should be left unchanged.