Search engines use bots to find content they can feature as search results. You want these bots to crawl your website because it improves your SEO, however, you don’t want them to crawl your website too frequently. If the bots crawl your website too frequently, it increases your server load and increases the number of requests Distil inspects. Distil inspects each request to your website, so it will inspect the search engine bot’s request as it does its crawling.
It’s best to find a happy medium for search engines that crawl your site. Fortunately, you have control over how frequently search engines crawl your site. Go to the search engine’s webmaster tools section and modify the crawl settings until you achieve a good balance between SEO responsiveness and server load.
Here’s how you can control how frequently Google’s bot, Googlebot, crawls your site.
- On the Webmaster Tools Home page, click your site.
- Click the gear icon, then click Site Settings.
- In the Crawl rate section, select the option you want.
Here are instructions for configuring Bing’s bot, Bingbot.
- Log into Bing Webmaster Tools.
- Add your site to Bing Webmaster Tools if you have not already done so.
- Click the Crawl tab in the top navigation.
- Click Crawl Settings in the left navigation.
- Select your website from the drop-down list (if not already selected).
- Select the radio button Use the selected crawl rate below to set a custom crawl rate and pattern.
- Use the graph to specify the crawl rate and crawl pattern by dragging the graph areas to the desired settings. You can click on individual vertical columns to set the crawl rate individually by hour.
- If your website is built in AJAX, select the check box labeled Configure Bing for Ajax site crawling so that Bing handles your website properly.
- Click Save to save your settings.