The Protect Your Content section organizes various settings used to guard your site resources.
Under the Manage Your Threat Responses heading, configure automated responses to thwart attacks against your site and its content. Responses are broken out into seven categories.
Distil maintains a shared access control list (ACL) of prior threats that have already been detected across our network. For example, if we have detected a known violator on another site, your own site is automatically protected from that threat.
Known Violator Data Centers (KVDC)
Distil also maintains a list of troublesome data centers that commonly host malicious requests. The list blocks any traffic from data centers on the first request and includes both common cloud and managed hosting providers, such as Rackspace. Distil is constantly curating and updating the list of known violator data centers.
Requests from Identities
Distil verifies the identity of incoming requests. Malicious bots can easily spoof user agents. These nefarious actors spoof their user agent to masquerade as a good bot, e.g., Googlebot. Distil forces a two-factor authentication for all good bots and verifies that they’re coming from their correct user agents. Then we confirm that each request maps to one of the IPs within the range of the corresponding bot. If not, the request is flagged as a malicious attempt.
Requests from Aggregator User Agents
Next, Distil checks a homegrown list of known malicious aggregator user agents. These provide zero value to your site. They can also crawl certain parts of your site in a harsh manner, potentially impacting performance or reliability. Unless you need complete and open access to such tools as RSS or Atom feeds, Distil recommends blocking these request types.
Requests from Automated Browsers
You can activate multiple threat responses for Distil to use in automatically mitigating such threats.
NOTE: All of these settings default to monitor-only mode for new customers and newly onboarded URLs.
Identify bots without taking any action. Distil automatically runs the entire suite of detection, but does not take action. However, Distil does embed an X-Distil bot header that identifies the type of bot and the different threats that it failed, if applicable.
Present a form where an end user can submit a request to be unblocked. The Distil Support team handles unblock requests will investigate and unblock validate requests. Unblock requests are rarely completed by a human user who was improperly flagged as a bad bot. Instead, unblock requests are completed by a bot designed to spam forms. When necessary, Distil will unblock legitimate users.
Present a CAPTCHA form to verify incoming questionable requests. CAPTCHA forms are less aggressive than a block page but provide an effective Turing test against malicious bots and host valuable information for Distil’s data science team to research.
The Manage Referrers & Proxies section provides control of particular URLs or proxy servers driving traffic to your site. A referrer is any domain, subdomain, or specific URL path that is sending costly traffic or hot-linking to your site. You have the ability to block anonymous proxies if that makes sense for your organization. If it does, toggle this option to Yes to have Distil stop any inbound request coming from those locations.
Manage Rate Limiting
The Manage Rate Limiting section lets you configure automated responses that react when a request amount or user behavior is above normal human rates. It comes in handy for blocking either bots or nefarious users who run automated scripts to quickly evaluate links and scrub your site’s content.
Once you’ve onboarded a domain with Distil, our system automatically machine-learns, creating behavioral models of what normal human patterns look like for your site. After we have collected traffic for about four to five days, the system makes automatic recommendations as to where you should set thresholds. Alternately, you can manually set them to any limit at any time.
These rate limits are broken out by:
- When Pages per Minute Exceeds – Limits the number of pages visited in one minute
- When Pages per Session Exceeds – Limits the number of pages visited during a single session
- When Session Length Exceeds – Limits the amount of time spent on the domain during a single session
Test your site latency after implementing a rate limiting change by clicking the See How Your Traffic WIll be Affected link.
Review our Creating Custom Threat Response Pages support article for more information regarding custom pages.
IP Access List
The IP Access List lets you keep an extensive catalog of all IP addresses that visit your site. Additionally, you can configure access rights to allow or block IPs, set expiration dates for access rights, and maintain notes and comments for each address.
The IP Access List provides a toolbar to help you manage IP address activity. It includes:
- Checkbox – Select one or multiple IPs from the Access List to take further action.
- Filter By – Display allowed, blocked, or all IP addresses.
- Edit – Edit access rights, set expiration windows, and update notes for selected IPs.
- Delete – Delete selected IPs from your Access List.
- Delete All – Deletes all IPs from your Access List.
- Search by IP or Note – Search for specific IPs in your Access List by IP address (or range of addresses) and notes.
- Export – Export a .TSV file of the current Access List.
- Import – Import a .TSV file of known IPs to integrate and manage within your Access List.
- Add IP Address – Add a single IP or multiple IPs to your Access List and configure access rights, expiration windows, and notes about the IPs.
- Jump to Page – Jump to additional pages in your Access List.
NOTE: You will need to contact Distil Support for assistance with whitelisting and blacklisting IPs by User Agent, Path, HTTP Header, and Country.
Country Block List
Another way to protect your content is to block out requests from specific countries. Manage access by country by selecting a record in the Manage Your Country Access List table and add a country to your block list by searching for the country (by name or code) and clicking Block Country.