When you onboard to Distil, it's important to whitelist all the third-party monitoring services you're using. By default, Distil automatically whitelists a set of verified "Good bots," such as Googlebot, Bingbot, and Yahoo Slurp.
Most other bots would fall into the category of bad bots, including automated tools that you use to monitor your site, perform testing, or to audit for SEO purposes. For those types of automation, we need to select a method for excluding those known tools from Distil's protection.
Distil supports the following basic methods of whitelisting monitoring services and third-party tools that you want to continue to access your site.
You can whitelist a single IP address or a range of IP addresses directly in the Distil Portal. To configure a whitelist in the Distil Portal, click your domain, click Settings > IP access list. For more information, see this tutorial.
HTTP Request Header
If it's not possible to whitelist by source IP, another option is to whitelist based on HTTP Request Header. Most monitoring or SEO tools provide either a unique User Agent string or offer you the ability to add custom HTTP Request header/value pairs. For example, the SEO tool Screaming Frog uses the User Agent "Screaming Frog SEO Spider" and monitoring tools Pingdom and Catchpoint allow users to create custom HTTP Request Headers to send with each request.
Whitelisting by URL is useful in cases where an entire portion of the site is to be excluded from Distil protection or when a specific low value URL is used for an uptime check.
In addition to HTTP Request Header/Value pair whitelisting, Distil also supports Cookie/Value pair whitelisting. This option is useful for sharing a key with remote workers who may not be identifiable by IP address, and ensuring they are not subject to content protection settings.
For more information on whitelisting, or to implement a whitelisting option other than IP, please contact Distil support.