RT Cunningham

Bot List – My Continuously Updated Web Robots Project

Bot List

This a list of every web robot, or bot, I’ve investigated. I usually update the list as soon as I encounter a new one in the access log for this website.

I originally created a website to list each one and then describe them independently. The website didn’t get many visitors, so I decided to consolidate everything here.

I’m not going to tell you which bots are good or bad, but I’ll tell you which ones I’m blocking. Another list follows this one. It lists fake and malformed user agents you should always block.

360 Spider 
Type:
    Search Engine (China)
User Agent String:
    Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
Known IP Addresses:
    Random IP addresses in China


adscanner
Type:
    Advertising
User Agent String:
    Mozilla/5.0 (compatible; adscanner/)
Known IP Addresses:
    85.25.210.234
Note: See https://adscanner.tv/ - I block this user agent.


AdsrvrBot
Type:
    Unknown
User Agent String:
    Mozilla/5.0 (compatible; AdsrvrBot)
Known IP addresses:
    Amazon ranges
Note:
    Since I can't find any information on this bot, I block the user agent.


AhrefsBot
Type:
    Search Marketing
User Agent string:
    Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/) 
Known IP Addresses:
    OVH ranges
Note:
    Preparing to become a search engine.


aiohttp
Type:
    Client and HTTP Server
User Agent String:
    Python/3.6 aiohttp/3.5.4
Known IP Addresses:
    Any
Note:
    Asynchronous HTTP Client/Server for asyncio and Python. I block this user agent.


AmazonAdBot
Type:
    Advertising
User Agent String:
    Mozilla/5.0 (compatible; AmazonAdBot/1.0; +https://adbot.amazon.com)
Known IP Addresses:
    Amazon ranges


Apache-HttpAsyncClient
Type:
    Client
User Agent String:
    Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
Known IP Addresses:
    Any
Note:
    See https://hc.apache.org/httpcomponents-asyncclient-dev/ - I block this user agent.


Apache-HttpClient
Type:
    Client
User Agent String:
    Apache-HttpClient/4.5.2 (Java/1.8.0_161)
Known IP Addresses:
    Any
Note:
    See https://hc.apache.org/httpcomponents-client-4.5.x/ - I block this user agent.


Applebot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Known IP Addresses:
    17.0.0.0 - 17.255.255.255 (17.0.0.0/8)


Aspider
Type:
    Search Engine (enterprise)
User Agent String:
    Aspider/3.2.1, The Aspire Web Crawler (http://www.searchtechnologies.com/aspire) TestCrawl
Known IP Addresses:
    24.107.125.211
Note:
    I block this user agent.


Baiduspider
Type:
    Search Engine (China)
User Agent String:
    Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Known IP Addresses:
    3.243.252.0 - 63.243.252.255 (63.243.252.0/24)
    103.235.44.0 - 103.235.47.255 (103.235.44.0/22)
    104.193.88.0 - 104.193.91.255 (104.193.88.0/22)
    106.12.0.0 - 106.13.255.255 (106.12.0.0/15)
    119.63.192.0 - 119.63.199.255 (119.63.192.0/21)
    180.76.0.0 - 180.76.255.255 (180.76.0.0/16)
    182.61.0.0 - 182.61.255.255 (182.61.0.0/16)
    185.10.104.0 - 185.10.107.255 (185.10.104.0/22)
    123.125.71.0 - 123.125.71.255 (123.125.71.0/24) (YBV)
    220.181.0.0 - 220.181.255.255 (220.181.0.0/16) (CHINANET Beijing province network)


Barkrowler
Type:
    Data Collection
User Agent String:
    Barkrowler/0.9 (+http://www.exensa.com/crawl)
Known IP Addresses:
    195.154.104.33
Note:
    This is a fork of the BUbiNG (below) open-source Java crawler. I block this user agent.


bidswitchbot
Type:
    Advertising (ads.txt crawler)
User Agent String:
    bidswitchbot/1.0
Known IP Addresses:
    Google Cloud ranges


bingbot
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
    Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Known IP addresses:
    13.64.0.0 - 13.107.255.255 (13.64.0.0/11, 13.96.0.0/13, 13.104.0.0/14)
    40.74.0.0 - 40.125.127.255 (40.74.0.0/15, 40.76.0.0/14, 40.80.0.0/12, 40.96.0.0/12, 40.112.0.0/13, 40.120.0.0/14, 40.124.0.0/16, 40.125.0.0/17)
    52.145.0.0 - 52.191.255.255 (52.145.0.0/16, 52.146.0.0/15, 52.148.0.0/14, 52.152.0.0/13, 52.160.0.0/11)
    52.224.0.0 - 52.255.255.255 (52.224.0.0/11)
    157.54.0.0 - 157.60.255.255 (157.54.0.0/15, 157.56.0.0/14, 157.60.0.0/16)
    207.46.0.0 - 207.46.255.255 (207.46.0.0/16)


BingPreview
Type: 
    Search Engine
User Agent String:
    Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
    Mozilla/5.0 (Windows: Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b
Known IP Addresses:
    40.74.0.0 - 40.125.127.255 (40.74.0.0/15, 40.76.0.0/14, 40.80.0.0/12, 40.96.0.0/12, 40.112.0.0/13, 40.120.0.0/14, 40.124.0.0/16, 40.125.0.0/17)


Bleriot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
    Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w
Known IP Addresses:
    91.242.162.0 - 91.242.162.255 (91.242.162.0/24)
Note:
    Formerly known as Quantify


BUbiNG
Type:
    Data Collection
User Agent String:
    BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
Known IP Addresses:
    90.147.171.0 - 90.147.171.255 (90.147.171.0/24)
    159.149.0.0 - 159.149.255.255 (159.149.0.0/16)
Note:
    This is an open-source Java fully distributed crawler (no central coordination) developed at the University of Milan’s (Italy) science department. I block this user agent.


CCBot
Type:
    Search Engine Data
User Agent String:
    CCBot/2.0 (https://commoncrawl.org/faq/)
Known IP Addresses:
    Amazon ranges


Cliqzbot
Type: 
    Search Engine and Web Browser
User Agent String:
    Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)
Known IP Addresses:
    Amazon ranges


coccocbot-image
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
Known IP Addresses:
    103.131.68.0 - 103.131.71.255 (103.131.68.0/22)


coccocbot-web
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
Known IP Addresses:
    103.131.68.0 - 103.131.71.255 (103.131.68.0/22)


colly
Type:
    Scraper
User Agent String:
    colly - https://github.com/gocolly/colly
Known IP Addresses:
    Any
Note:
    I block this user agent.


contxbot
Type:
    Advertising (Amazon)

User Agent String:
    Mozilla/5.0 (compatible;contxbot/1.0)
Known IP Addresses:
    Amazon ranges


cURL
Type:
    Command Line Utility
User Agent Strings:
    curl/7.29.0
    curl/7.58.0
    curl/7.65.1
Known IP Addresses:
    Any
Note:
    Used for downloading files and web pages. I block this user agent.


Daum
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
    Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
    Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Known IP Addresses:
    203.133.160.0 - 203.133.191.255 (203.133.160.0/19)
Note:
    Formerly known as Daumoa.


Disqus
Type:
    Service (Comments)
User Agent String:
    Disqus/1.0
Known IP Addresses:
    75.126.0.0 - 75.126.255.255 (75.126.0.0/16)
    108.168.151.0 - 108.168.151.31 (108.168.151.0/27)
    173.192.0.0 - 173.193.255.255 (173.192.0.0/15)
    184.172.0.0 - 184.173.255.255 (184.172.0.0/15)
    208.101.0.0 - 208.101.63.255 (208.101.0.0/18)


domaincrawler.com
Type:
    Service (retrieves information about your domain name)
User Agent String:
    (info@domaincrawler.com; http://www.domaincrawler.com/rtcx.net)
Known IP Addresses:
    185.6.8.3
Note:
    There is no robot name for this bot. Notice how my domain name was added to the user agent string. I block this user agent.


DomainStatsBot
Type:
    Service
User Agent String:
    DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)
Known IP Addresses
    148.251.121.91


DotBot
Type:
    Search Marketing
User Agent String:
    Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Known IP Addresses:
    216.244.66.244
Note:  I block this user agent.


DuckDuckBot-Https
DuckDuckGo-Favicons-Bot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
    Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
Known IP Addresses:
    23.21.226.191
    54.208.102.37
    107.20.237.51
    107.21.1.8


e.ventures Investment Crawler
Type:
    Unknown
User Agent String:
    e.ventures Investment Crawler (eventures.vc)
Known IP Addresses:
    38.99.62.94
Note:
    I block this user agent.


EasyBib AutoCite
Type:
    Citations
User Agent String:
    EasyBib AutoCite (http://autocite-info.citation-api.com/)
Known IP Addresses:
    Amazon ranges


Embedly
Type:
    Service (allows other sources to embed parts of your website)
User Agent String:
    Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


evc-batch
Type:
    Advertising (ads.txt crawler)
User Agent String:
    Mozilla/5.0 (compatible; evc-batch/2.0)
Known IP Addresses:
    38.99.62.94


facebookexternalhit
Type:
    Social Media
User Agent Strings:
    facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    facebookexternalhit/1.1;line-poker/1.0
Known IP Addresses:
    31.13.64.0 - 31.13.127.255 (31.13.64.0/18)
    31.13.115.0 - 31.13.115.255 (31.13.115.0/24)
    66.220.144.0 - 66.220.159.255 (66.220.144.0/20)
    69.171.224.0 - 69.171.255.255 (69.171.224.0/19)
    147.92.128.0 - 147.92.255.255 (147.92.128.0/17) (JP)
    173.252.64.0 - 173.252.127.255 (173.252.64.0/18) 


FeedBurner
Type:
    Feed Distribution
User Agent String:
    FeedBurner/1.0 (http://www.FeedBurner.com)
Known IP Addresses
    74.125.0.0/16


fetch libfetch
Type:
    Command Line Utility
User Agent String:
    fetch libfetch/2.0
Known IP Addresses:
    Any
Note:
    FreeBSD download tool. I block this user agent.


GigablastOpenSource
Type:
    Search Engine (internal)
User Agent Strings:
    GigablastOpenSource/1.0
    Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
Known IP Addresses:
    Any
Note:
    Used by businesses as an internal search engine. I block this user agent.


Gluten Free Crawler
Type:
    Research
User Agent String:
    Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
Known IP Addresses:
    104.131.147.112
Note:
    I block this user agent.


Go-http-client
Type:
    Client
User Agent String:
    Go-http-client/2.0
Known IP Addresses:
    Any
Note:
    I block this user agent.


gocrawl
Type:
    Crawler (not a search engine)
User Agent String:
    Mozilla/5.0 (Windows NT 6.1; rv:15.0) gocrawl/0.4 Gecko/20120716 Firefox/15.0a2
Known IP Addresses:
    Any
Note:
    I block this user agent.


Google
Type:
    Developer
User Agent String:
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
Known IP Addresses:
    66.102.0.0 - 66.102.15.255 (66.102.0.0/20)


Googlebot
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Known IP Addresses:
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)


Google-Site-Verification
Type:
    Search Engine (webmaster tools)
User Agent String:
    Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
Known IP Addresses:
    72.14.192.0 - 72.14.255.255 (72.14.192.0/18)
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)


Googleweblight
Google Favicon
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Known IP Addresses:
    66.102.0.0 - 66.102.15.255 (66.102.0.0/20)
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)
Note:
    Serves optimized web pages for slow clients (less than 3G).


Grammarly
Type:
    Service
User Agent String:
    Grammarly/1.0 (http://www.grammarly.com)
Known IP Addresses:
    Amazon ranges
Note:
    User agent spam. I block this user agent.


GrapeshotCrawler
Type:
    Advertising
User Agent Strings:
    Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
    Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1. 4 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Known IP Addresses:
    148.64.56.0 - 148.64.56.255 (148.64.56.0/24)


GroupHigh
Type: Search Marketing
User Agent Strings:
    Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
    Mozilla/5.0 (compatible; GroupHigh/1.1; +http://www.grouphigh.com/)
Known IP Addresses:
    Random ARIN IP ranges.
Note: I block this user agent.


HeadlessChrome
Type:
    Web Browser
User Agent String:
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/67.0.3396.99 Safari/537.36
Known IP Addresses:
    Any
Note: It's for developers, but it can easily be abused. I block this user agent.


HTTP
Type:
    Unknown
User Agent String:
    HTTP
Known IP Addresses:
    113.35.251.98 (JP)
Note:
    I block this user agent.


HTTPie
Type:
    Command Line Utility
User Agent String:
    HTTPie/1.0.2
Known IP Addresses:
    Any
Note:
    Similar to cURL and wget. I block this user agent.


ips-agent
Type:
    Service (monitoring and security)
User Agent String:
    Mozilla/5.0 (compatible; ips-agent)
Known IP Addresses:

    69.58.176.0 - 69.58.191.255 (69.58.176.0/20)
Note:
    I block this user agent.


Java
Type:
    Programming Language
User Agent String:
    Java/1.8.0_212
Known IP Addresses
    Any
Note:
    I block this user agent.


Jersey
Type:
    Client
User Agent String:
    Jersey/2.25.1 (Apache HttpClient 4.5)
Known IP Addresses:
    Any
Note:
    A variation of Apache-HttpClient version 4.5. I block this user agent.


linkdexbot
Type:
    Search Marketing
User Agent String:
    Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Known IP Addresses:
    185.184.157.0 - 185.184.157.255 (185.184.156.0/22)
Note:
    I block this user agent.


LivelapBot
Type:
    Social Media
User Agent String:
    LivelapBot/0.2 (http://site.livelap.com/crawler)
Known IP Addresses:
    192.99.1.145
    198.27.82.109
Note: site.livelap.com works but livelap.com and www.livelap.com both indicate the domain is not set up. I cannot register at site.livelap.com. I block this user agent.


ltx71
Type:
    Research
User Agent String:
    ltx71
Known IP Addresses:
    Unknown in Google Cloud (deleted from access logs)
Note: I block this user agent.


lua-resty-http
Type:
    Nginx Scripting Module
User Agent String:
    ua-resty-http/0.10 (Lua) ngx_lua/10000
Known IP Addresses:
    Any
Note:
    This is an Nginx web server module. I block this user agent.


Mail.RU_Bot

Type:
    Search Engine (Russia)
User Agent Strings:
    Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
    Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
Known IP Addresses:
    95.163.248.0 - 95.163.255.255 (95.163.248.0/21)
    217.69.128.0 - 217.69.143.255 (217.69.128.0/20)


MauiBot
Type:
    Research
User Agent String:
    MauiBot (crawler.feedback+dc@gmail.com)
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


MBCrawler
Type:
    Service (monitors backlinks)
User Agent String:
    MBCrawler/1.0 (https://monitorbacklinks.com/robot)
Known IP Addresses:
    Amazon ranges
Note:
    This is a pay service. I block this user agent.


Mediapartners-Google
Type:
    Advertising
User Agent String:
    Mediapartners-Google
Known IP Addresses
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)
    72.14.192.0 - 72.14.255.255 (72.14.192.0/18)


MixnodeCache
Type:
    Data Collection
User Agent String:
    MixnodeCache/1.8(+https://cache.mixnode.com/)
Known IP Addresses
    Amazon ranges
Note:
    I block this user agent.


MJ12bot
Type:
    Search Engine (decentralized)
User Agent:
    Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/
Known IP Addresses:
    Random datacenter IP addresses worldwide
Note:
    Majestic-12 offers a lot of information about your website, but only if you let the MJ12bot access your website.


MojeekBot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Known IP Addresses
    5.102.173.64 - 5.102.173.79 (5.102.173.64/28)


nadesiko
Type:
    Unknown
User Agent String:
    nadesiko
Known IP Addresses:
    113.35.251.98 (JP)
Note:
    I block this user agent.


NetcraftSurveyAgent
Type:
    Analysis
User Agent String:
    Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Known IP Addresses:
    134.209.36.109 (Digital Ocean)
    167.71.174.74 (Digital Ocean)
    167.71.182.71 (Digital Ocean)
    167.71.191.243 (Digital Ocean)


newspaper
Type:
    Unknown
User Agent String:
    newspaper/0.2.2
Known IP Addresses:
    Any
Note:
    Searching for this user agent returns too many results. I block this user agent.


oBot
Type:
    Research and Development
User Agent String:
    Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
Known IP Addresses:
    194.153.113.0 - 194.153.113.255 (194.153.113.0/24)
    206.253.224.0 - 206.253.225.255 (206.253.224.0/23)
    206.253.226.0 - 206.253.226.255 (206.253.226.0/24)
Note:
    This is the IBM Germany Research and Development Content Security Division. They use website data to build content filters.


omgili
Type:
    Data Extraction
User Agent String:
    omgili/0.5 +http://omgili.com
Known IP Addresses:
    82.166.195.64
Note:
    It used to be a search engine, but now it feeds data to webhose.io. I now block this user agent.


Pandalytics
Type:
    Service (retrieves information about your domain name)
User Agent String:
    Pandalytics/1.0 (https://domainsbot.com/pandalytics/)
Known IP Addresses:
    Amazon ranges
Note:
    Maintains a domain name database. I block this user agent.

Pinterestbot
Type:
    Social Media
User Agent String:
    Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
Known IP Addresses:
    154.236.1.0 - 54.236.1.255 (54.236.1.0/24)


PocketParser
Type:
    Service and Web Browser Extension (saves pages)
User Agent String:
    PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
Known IP Addresses:

    Amazon ranges


proximic
Type:
    Advertising
User Agent String:
    Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
Known IP Addresses:
    Amazon ranges


python-requests
Type:
    Scraper
User Agent String:
    python-requests/2.21.0
Known IP Addresses:
    Any
Note:
    A URL fetcher. I block this user agent.


Python-urllib
Type:
    Scraper
User Agent Strings:
    Python-urllib/1.17
    Python-urllib/2.7
Known IP Addresses:
    Any
Note:
    A URL fetcher. I block this user agent.


RankingBot2
Type:
    Research
User Agent String:
    RankingBot2 -- https://varocarbas.com/bot_ranking2/
Known IP Addresses:
    81.43.124.145
Note:
    Controlled by an individual in Spain. I block this user agent.


Ruby
Type:
    Programming Language
User Agent Strings:
    Ruby
    Ruby, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
Known IP Addresses:
    Any
Note:
    I block this user agent.


Scrapy
Type:
    Scraper
User Agent Strings:
    Scrapy/1.5.1 (+https://scrapy.org)
    Scrapy/1.5.2 (+https://scrapy.org)
Known IP Addresses:
    Any
Note:
    This is a Python script. I block this user agent.


Screaming Frog SEO Spider
Type:
    Multi Platform Software (Windows, Macintosh, Linux)
User Agent String:
    Screaming Frog SEO Spider
Known IP Addresses:
    Any
Note:
    I block this user agent.


Seekport Crawler
Type:
    Search Engine (Germany)
User Agent String:
    Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)
Known IP Addresses:
    159.69.189.215 (Hetzner Online)


SemrushBot
Type:
    Search Marketing
User Agent Strings:
    Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
    Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)
Known IP Addresses:
    46.229.168.129
    213.174.152.181
Note:
    I block this user agent.


SEOkicks
Type:
    Service (backlink monitor)
User Agent String:
    Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)
Known IP Addresses:
    95.216.0.0 - 95.217.255.255 (95.216.0.0/16) (Hetzner Online)


serpstatbot
Type:
    Search Marketing
User Agent String:
    serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)
Known IP Addresses:
    136.243.155.105 (Hetzner Online)
Note:
    I block this user agent.


SeznamBot
Type:
    Search Engine (Czech Republic)
User Agent Strings:
    Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
    Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
 Known IP Addresses:
    77.75.72.0 - 77.75.79.255 (77.75.72.0/21)
    185.66.188.0 - 185.66.191.255 (185.66.188.0/22)


SiteChecker
Type:
    Unknown
User Agent String:
    SiteChecker/1.0
Known IP Addresses:
    Amazon ranges (KR area)
Note:
    I block this user agent.


Slackbot-LinkExpanding
Type:
    Meta Content Extractor
User Agent String:
    Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Known IP Addresses:
    Amazon Ranges


slick-proxy
Type:

    VPN Proxy
User Agent String:
    AppEngine-Google; (+http://code.google.com/appengine; appid: p~slick-proxy)
Known IP addresses:
    107.178.193.197 (Google Cloud)
Note:
    This isn't a bot. It's an application.


SMTBot
Type:
    Analysis
User Agent Strings:
    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
    Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)                 Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


snapchat-proxy
Type:
    Social Media Proxy
User Agent String
    AppEngine-Google; (+http://code.google.com/appengine; appid: s~snapchat-proxy)
Known IP Addresses:
    35.187.132.135 (Google Cloud)
    107.178.194.101 (Google Cloud)
Note:
    This isn't a bot. It's an application.


Sogou web spider
Type:
    Search Engine (China)
User Agent String:
    Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Known IP Addresses:
    Random IP addresses in China


startmebot
Type:
    Bookmarking Service
User Agent Strings:
    Mozilla/5.0 (compatible; startmebot/1.0; +https://start.me/bot)
    Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
Known IP Addresses:
    Amazon ranges


Streamline3Bot
Type:
    Web Directory
User Agent Strings:
    Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) Streamline3Bot/1.0
    Mozilla/5.0 (Windows NT 6.1; Win64; x64; +https://www.ubtsupport.com/legal/Streamline3Bot.php) Streamline3Bot/1.0
Known IP Addresses:
    OVH ranges
Note:
    I block this user agent.


SurdotlyBot
Type:
    Link Scanner
User Agent String:
    Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Known IP Addresses:
    Amazon ranges
Note:
    User Agent Spam. I block this user agent.


TelegramBot
Type:
    Messaging Application
User Agent String:
    TelegramBot (like TwitterBot)
Known IP Addresses:
    149.154.160.0 - 149.154.163.255 (149.154.160.0/22)


The Knowledge AI
Type:
    Unknown
User Agent String:
    The Knowledge AI
Known IP Addresses:
    64.62.128.0 - 64.62.255.255 (64.62.128.0/17)
Note:
    I block this user agent.


TrendsmapResolver
Type:
    Social Media
User Agent String:
    Mozilla/5.0 (compatible; TrendsmapResolver/0.1)
Known IP Addresses:
    88.99.145.42  (Hetzner Online)
Note:
    Gathers information for Twitter trends. I block this user agent.


TweetmemeBot
Type:
    Link Scanner and Data Collection
User Agent String:
    Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
Known IP Addresses:
    46.236.0.0 - 46.236.63.255 (46.236.0.0/19)
Note:
    Gathers information from Twitter. I block this user agent.


Twitterbot
Type:
    Social Media
User Agent String:
    Twitterbot/1.0
Known IP Addresses:
    199.16.156.0 - 199.16.159.255 (199.16.156.0/22)
    199.59.148.0 - 199.59.151.255 (199.59.148.0/22)


Uptimebot
Type:
    Service (monitors site uptime)
User Agent String:
    Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
Known IP Addresses:
    109.201.138.228
Note:
    User agent spam. It's a pay service. I block this user agent.


vuhuvBot
Type:
    Search Engine (Turkey)
User Agent String:
    Mozilla/5.0 (compatible; vuhuvBot/1.0; +http://vuhuv.com/bot.html)
Known IP Addresses:
    185.93.54.0 - 185.93.54.255 (185.93.54.0/24)


WbSrch
Type:
    Search Engine (independent)
User Agent String:
    Mozilla/5.0 (compatible; WbSrch/1.1 +https://wbsrch.com)
Known IP Addresses:
    216.151.3.108


Wget
Type:
    Command Line Utility
User Agent Strings:
    Wget/1.15 (linux-gnu)
    Wget/1.19.4 (linux-gnu)
Known IP Addresses:
    Any
Note:
    Used for downloading files and web pages. I block this user agent.


Widget Server
Type:
    Unknown
User Agent String:
    Mozilla/5.0 (en-US) AppleWebKit/537.36 (KHTML, like Gecko; Widget Server) Chrome/41.0.2272.118 Safari/537.36
Known IP Addresses:
    66.102.0.0 - 66.102.15.255 (66.102.0.0/20)
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)
Note:
    These are Google IP addresses.


www.ru
Type:
    Domain Reseller
User Agent String:
    www.ru
Known IP Addresses:
    31.172.81.30 (DE)
Note:
    User agent spam. I block this user agent.


Xenu Link Sleuth
Type:
    Windows Software
User Agent String:
    Xenu Link Sleuth/1.3.8
Known IP Addresses:
    Any
Note:
    I block this user agent.


yacybot
Type:
    Search Engine (peer-to-peer)
User Agent String:
    yacybot (/global; amd64 Windows Server 2016 10.0; java 1.8.0_211; America/en) http://yacy.net/bot.html
Known IP Addresses:
    Any


Yahoo! Slurp
Type:
    Search Engine
User Agent Strings:
     Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
     Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
Known IP Addresses
    72.30.0.0 - 72.30.255.255 (72.30.0.0/16)
    74.6.0.0 - 74.6.255.255 (74.6.0.0/16)
    217.146.176.0 - 217.146.178.255 (217.146.176.0/21)


YandexBot
YandexImages
Type: Search Engine (Russia)
User Agent Strings:
    Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
    Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Known IP Addresses:
    5.45.192.0 - 5.45.255.255 (5.45.192.0/18)
    5.255.192.0 - 5.255.255.255 (5.255.192.0/18)
    37.9.64.0 - 37.9.127.255 (37.9.64.0/18)
    37.140.128.0 - 37.140.191.255 (37.140.128.0/18)
    77.88.0.0 - 77.88.63.255 (77.88.0.0/18)
    84.201.128.0 - 84.201.191.255 (84.201.128.0/18)
    87.250.224.0 - 87.250.255.255 (87.250.224.0/19)
    93.158.128.0 - 93.158.191.255 (93.158.128.0/18)
    95.108.128.0 - 95.108.255.255 (95.108.128.0/17)
    100.43.64.0 - 100.43.95.255 (100.43.64.0/19)
    130.193.32.0 - 130-193.63.255 (130.193.32.0/19)
    141.8.128.0 - 141.8.191.255 (141.8.128.0/18)
    178.154.128.0 - 178.154.255.255 (178.154.128.0/17)
    199.21.96.0 - 199.21.99.255 (199.21.96.0/22)
    213.180.192.0 - 213.180.223.255 (213.180.192.0/19)


Yeti
Type:
    Search Engine (Korea)
User Agent String:
    Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/spd)
Known IP Addresses:
    125.209.192.0 - 125.209.255.255 (125.209.192.0/18)


zgrab
Type:
    Research
User Agent String:
    Mozilla/5.0 zgrab/0.x (compatible; Researchscan/t13rl; http://researchscan.comsys.rwth-aachen.de)
Known IP Addresses:
    137.226.0.0 - 137.226.255.255 (137.226.0.0/16)
Note:
    I block this user agent.

Some of these are exact user agent strings and some are partial. Exact and partial notes in parentheses are not included in the strings. I block each bot without question.

Chrome (exact)
facebookexternalhit/1.1 (compatible;) (exact)
Firefox (exact)
Internet Explorer (exact)
mozilla (exact)
Mozilla/5.0 (exact)
Mozilla / 5 (partial, note the added spaces)
Mozilla/5.0(compatible (partial, note the missing spaces)
Mozilla/5.0 Firefox/26.0 (exact)
Mozilla/5.0 (Linux; rv:26.0) Gecko/20100101 Firefox/26.0 (exact)
NameOfAgent (partial)
spider (exact)
User-Agent (partial)
userAgent (exact)
WEBKIT_VERSION (partial)

If you discover an error in either list, please let me know using the contact page.

Last Update: July 12, 2019