MageHost - Crawler Bot Control

Gewijzigd op Wo, 26 Jun om 4:45 PM

Crawlers, also known as bots or spiders, navigate websites to index their contents. They read content much faster than human users, which can significantly impact server load. To manage this, you can instruct crawlers on which URLs to avoid.


Using a robots.txt file, you can specify a list of URLs or URL paths that crawlers should not access. 

Most reputable crawlers respect these instructions.


User-Agent: *
Disallow: */search/

This tells the crawler to skip any URLs containing /search/.

For more information, visit:


In HTML, you can use the rel="nofollow" attribute in anchor elements to indicate that certain links should not be followed by crawlers. This is especially useful for dynamically generated links, which can't always be pre-defined in a robots.txt file.


<a rel="nofollow" href="">

On e-commerce sites, for example, category filters create multiple URLs that lead to the same content. 

These URLs can generate an excessive number of pages for crawlers to process, as each filtered view results in a new unique URL. 

These excessive number of pages increases server load and can't usually be mitigated through caching.

Headers / Meta Tags

You can also use response headers or meta tags to instruct crawlers to skip all links on the page.


X-Robots-Tag: nofollow
<meta name="robots" content="nofollow">

By implementing these strategies, you can help crawlers navigate your website more efficiently. 

This ensures that all relevant information is indexed while reducing the time crawlers spend on your site. 

Consequently, server load decreases as crawlers only visit pertinent URLs, and more URLs can be cached, leading to improved response times.

Was dit artikel nuttig?

Dat is fantastisch!

Hartelijk dank voor uw beoordeling

Sorry dat we u niet konden helpen

Hartelijk dank voor uw beoordeling

Laat ons weten hoe we dit artikel kunnen verbeteren!

Selecteer tenminste een van de redenen
CAPTCHA-verificatie is vereist.

Feedback verzonden

We stellen uw moeite op prijs en zullen proberen het artikel te verbeteren