Bad bots are classified as one of the following:
- Known Violator User Agents – Generally associated with programming languages and other tools designed to programmatically harvest data from a page. Examples include cURL, wget, LWP::Simple, and PHP.
- Bad User Agents – Bad actors (often generic crawlers) that access websites, but aren’t necessarily interested in content. They cause high volumes of traffic while providing no benefit to the original site. Examples include CatchBot and NextGenSearchBot.
- Aggregator User Agents – User agents belonging to tools designed to access website RSS/Atom feeds. Examples include FeedFetcher, FeedBurner, and WordPress' internal feed aggregator.
- Unclassified User Agents – User agents that have not yet been classified in any of the above categories. Often these are tied to the “Reporting As” title; that is, they appear to be normal browser user agents, but Distil believes the browser may not actually be making the request.
- Uncategorized Bot – User agent not matching any of the patterns Distil has defined as known bots. Further research is required before it can be categorized.
Use this information when reviewing the Bad Bots report to better understand the bots targeting your site.