Follow

Disabling good bot crawl for Distil Pages

Google and other good bots may end up crawling the Distil captcha, block, or drop page. When this happens the request is recorded as an Error, as the HTTP status code served back on these pages is a 405, 416, or 456 respectively.

An example of how Google webmaster tools records a request for the Distil threat response pages is shown below:

Search_Console_.png

 

To avoid errors such as these you can modify your robots.txt with the directive to Disallow for the path /distil. Testing this with robots.txt Tester on the Google webmasters tool is a quick and easy way to find out how the Status changes.

Search_Console_-_robots_txt_Tester.png

The status shows Blocked as seen from the screenshot above. With the robots.txt change live on your site Google bot/other good bots will no longer try to crawl the page. If you test via the option Fetch as Google, you will see a status of Blocked and it will no longer be reported as an error.

 

Search_Console_-_Fetch_as_Google_.png

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments