RT Cunningham

Bot List - My Continuously Updated Web Robots Project

Bot List This a list of every web robot, or bot, I’ve investigated. I usually update the list as soon as I encounter a new one in the access log for this website (last updated August 7, 2019).

I originally created a website to list each one and then describe them independently. The website didn’t get many visitors, so I decided to consolidate everything here.

I’m not going to tell you which bots are good or bad, but I’ll tell you which ones I’m blocking.

360 Spider 
Type:
    Search Engine (China)
User Agent String:
    Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider
Known IP Addresses:
    Random IP addresses in China


ACHE
Type:
    Crawler
User Agent String:
    Mozilla/5.0 (compatible; ACHE/0.8; +https://github.com/ViDA-NYU/ache)
Known IP Addresses:
    Any
Note:
    I block this user agent.


adbeat_bot
Type:
    Advertising
User Agent String:
    +https://www.adbeat.com/policy adbeat_bot
Known IP Addresses:
    Amazon ranges



admantx-adform
Type:
    Advertising
User Agent String:
    admantx-adform/3.1 (+http://www.admantx.com/service-fetcher.html)
Known IP Addresses:
    85.93.93.56


adscanner
Type:
    Advertising
User Agent String:
    Mozilla/5.0 (compatible; adscanner/)
Known IP Addresses:
    85.25.210.234
Note: See https://adscanner.tv/ - I block this user agent.


AdsrvrBot
Type:
    Unknown
User Agent String:
    Mozilla/5.0 (compatible; AdsrvrBot)
Known IP addresses:
    Amazon ranges
Note:
    Since I can't find any information on this bot, I block the user agent.


AhrefsBot
Type:
    Search Marketing
User Agent string:
    Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/) 
Known IP Addresses:
    OVH ranges
Note:
    Preparing to become a search engine.


aiohttp
Type:
    Client and HTTP Server
User Agent String:
    Python/3.6 aiohttp/3.5.4
Known IP Addresses:
    Any
Note:
    Asynchronous HTTP Client/Server for asyncio and Python. I block this user agent.


AmazonAdBot
Type:
    Advertising
User Agent String:
    Mozilla/5.0 (compatible; AmazonAdBot/1.0; +https://adbot.amazon.com)
Known IP Addresses:
    Amazon ranges


Apache-HttpAsyncClient
Type:
    Client
User Agent String:
    Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
Known IP Addresses:
    Any
Note:
    See https://hc.apache.org/httpcomponents-asyncclient-dev/ - I block this user agent.


Apache-HttpClient
Type:
    Client
User Agent String:
    Apache-HttpClient/4.5.2 (Java/1.8.0_161)
Known IP Addresses:
    Any
Note:
    See https://hc.apache.org/httpcomponents-client-4.5.x/ - I block this user agent.


Applebot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Known IP Addresses:
    17.0.0.0 - 17.255.255.255 (17.0.0.0/8)


archive.org_bot
Type:
    Scraper, Archiver
User Agent String:
    Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Known IP Addresses:
    207.241.224.0 - 207.241.239.255 (207.241.224.0/20)
Note:
    I block this user agent.


Aspider
Type:
    Search Engine (enterprise)
User Agent String:
    Aspider/3.2.1, The Aspire Web Crawler (http://www.searchtechnologies.com/aspire) TestCrawl
Known IP Addresses:
    24.107.125.211
Note:
    I block this user agent.


BacklinkCrawler
Type:
    SEO
User Agent String:
    BacklinkCrawler (http://www.backlinktest.com/crawler.html)
Known IP Addresses:
    5.9.65.19 (Hetzner Online)
Note:
    I block this user agent.

Baiduspider
Type:
    Search Engine (China)
User Agent String:
    Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Known IP Addresses:
    3.243.252.0 - 63.243.252.255 (63.243.252.0/24)
    103.235.44.0 - 103.235.47.255 (103.235.44.0/22)
    104.193.88.0 - 104.193.91.255 (104.193.88.0/22)
    106.12.0.0 - 106.13.255.255 (106.12.0.0/15)
    119.63.192.0 - 119.63.199.255 (119.63.192.0/21)
    180.76.0.0 - 180.76.255.255 (180.76.0.0/16)
    182.61.0.0 - 182.61.255.255 (182.61.0.0/16)
    185.10.104.0 - 185.10.107.255 (185.10.104.0/22)
    123.125.71.0 - 123.125.71.255 (123.125.71.0/24) (YBV)
    220.181.0.0 - 220.181.255.255 (220.181.0.0/16) (CHINANET Beijing province network)


Barkrowler
Type:
    Data Collection
User Agent String:
    Barkrowler/0.9 (+http://www.exensa.com/crawl)
Known IP Addresses:
    195.154.104.33
Note:
    This is a fork of the BUbiNG (below) open-source Java crawler. I block this user agent.


bidswitchbot
Type:
    Advertising (ads.txt crawler)
User Agent String:
    bidswitchbot/1.0
Known IP Addresses:
    Google Cloud ranges


bingbot
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
    Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Known IP addresses:
    13.64.0.0 - 13.107.255.255 (13.64.0.0/11, 13.96.0.0/13, 13.104.0.0/14)
    40.74.0.0 - 40.125.127.255 (40.74.0.0/15, 40.76.0.0/14, 40.80.0.0/12, 40.96.0.0/12, 40.112.0.0/13, 40.120.0.0/14, 40.124.0.0/16, 40.125.0.0/17)
    52.145.0.0 - 52.191.255.255 (52.145.0.0/16, 52.146.0.0/15, 52.148.0.0/14, 52.152.0.0/13, 52.160.0.0/11)
    52.224.0.0 - 52.255.255.255 (52.224.0.0/11)
    157.54.0.0 - 157.60.255.255 (157.54.0.0/15, 157.56.0.0/14, 157.60.0.0/16)
    207.46.0.0 - 207.46.255.255 (207.46.0.0/16)


BingPreview
Type: 
    Search Engine
User Agent String:
    Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
    Mozilla/5.0 (Windows: Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b
Known IP Addresses:
    40.74.0.0 - 40.125.127.255 (40.74.0.0/15, 40.76.0.0/14, 40.80.0.0/12, 40.96.0.0/12, 40.112.0.0/13, 40.120.0.0/14, 40.124.0.0/16, 40.125.0.0/17)


bitlybot
Type:
    URL shortener
User Agent Strings:
    bitlybot
    bitlybot/3.0 (+http://bit.ly/)
Known IP Addresses:
    Amazon ranges


Bleriot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)
    Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w
Known IP Addresses:
    91.242.162.0 - 91.242.162.255 (91.242.162.0/24)
Note:
    Formerly known as Quantify


BUbiNG
Type:
    Data Collection
User Agent String:
    BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
Known IP Addresses:
    90.147.171.0 - 90.147.171.255 (90.147.171.0/24)
    159.149.0.0 - 159.149.255.255 (159.149.0.0/16)
Note:
    This is an open-source Java fully distributed crawler (no central coordination) developed at the University of Milan’s (Italy) science department. I block this user agent.


CCBot
Type:
    Search Engine Data
User Agent String:
    CCBot/2.0 (https://commoncrawl.org/faq/)
Known IP Addresses:
    Amazon ranges


Centro Ads.txt Crawler
Type:
    Advertising
User Agent String:
    Centro Ads.txt Crawler/1.0
Known IP Addresses:
    216.152.140.59


Clickagy Intelligence Bot
Type:
    Data Mining
User Agent String:
    Clickagy Intelligence Bot v2
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


Cliqzbot
Type: 
    Search Engine and Web Browser
User Agent String:
    Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)
Known IP Addresses:
    Amazon ranges


CMSCrawler
Type: 
    Search Engine and Web Browser
User Agent String:
    Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)
Known IP Addresses:
    83.233.207.74


coccocbot-image
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)
Known IP Addresses:
    103.131.68.0 - 103.131.71.255 (103.131.68.0/22)
    123.16.0.0 - 123.31.255.255 (123.16.0.0/12)


coccocbot-web
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)
Known IP Addresses:
    103.131.68.0 - 103.131.71.255 (103.131.68.0/22)


colly
Type:
    Scraper
User Agent String:
    colly - https://github.com/gocolly/colly
Known IP Addresses:
    Any
Note:
    I block this user agent.


contxbot
Type:
    Advertising (Amazon)

User Agent String:
    Mozilla/5.0 (compatible;contxbot/1.0)
Known IP Addresses:
    Amazon ranges


cURL
Type:
    Command Line Utility
User Agent Strings:
    curl/7.29.0
    curl/7.58.0
    curl/7.65.1
Known IP Addresses:
    Any
Note:
    Used for downloading files and web pages. I block this user agent.


Daum
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1
    Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daum 4.1
    Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Known IP Addresses:
    203.133.160.0 - 203.133.191.255 (203.133.160.0/19)
Note:
    Formerly known as Daumoa.


Disqus
Type:
    Service (Comments)
User Agent String:
    Disqus/1.0
Known IP Addresses:
    75.126.0.0 - 75.126.255.255 (75.126.0.0/16)
    108.168.151.0 - 108.168.151.31 (108.168.151.0/27)
    173.192.0.0 - 173.193.255.255 (173.192.0.0/15)
    184.172.0.0 - 184.173.255.255 (184.172.0.0/15)
    208.101.0.0 - 208.101.63.255 (208.101.0.0/18)


domaincrawler.com
Type:
    Service (retrieves information about your domain name)
User Agent String:
    (info@domaincrawler.com; http://www.domaincrawler.com/rtcx.net)
Known IP Addresses:
    185.6.8.3
Note:
    There is no robot name for this bot. Notice how my domain name was added to the user agent string. I block this user agent.


DomainStatsBot
Type:
    Service
User Agent String:
    DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)
Known IP Addresses
    148.251.121.91


DotBot
Type:
    Search Marketing
User Agent String:
    Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Known IP Addresses:
    216.244.66.244


DuckDuckBot-Https
DuckDuckGo-Favicons-Bot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
    Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
Known IP Addresses:
    23.21.226.191
    54.208.102.37
    107.20.237.51
    107.21.1.8


e.ventures Investment Crawler
Type:
    Unknown
User Agent String:
    e.ventures Investment Crawler (eventures.vc)
Known IP Addresses:
    38.99.62.94
Note:
    I block this user agent.


EasyBib AutoCite
Type:
    Citations
User Agent String:
    EasyBib AutoCite (http://autocite-info.citation-api.com/)
Known IP Addresses:
    Amazon ranges


Embedly
Type:
    Service (allows other sources to embed parts of your website)
User Agent String:
    Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


evc-batch
Type:
    Advertising (ads.txt crawler)
User Agent String:
    Mozilla/5.0 (compatible; evc-batch/2.0)
Known IP Addresses:
    38.99.62.94


facebookexternalhit
Type:
    Social Media
User Agent Strings:
    facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    facebookexternalhit/1.1;line-poker/1.0
Known IP Addresses:
    31.13.64.0 - 31.13.127.255 (31.13.64.0/18)
    31.13.115.0 - 31.13.115.255 (31.13.115.0/24)
    66.220.144.0 - 66.220.159.255 (66.220.144.0/20)
    69.171.224.0 - 69.171.255.255 (69.171.224.0/19)
    147.92.128.0 - 147.92.255.255 (147.92.128.0/17) (JP)
    173.252.64.0 - 173.252.127.255 (173.252.64.0/18) 


FeedBurner
Type:
    Feed Distribution
User Agent String:
    FeedBurner/1.0 (http://www.FeedBurner.com)
Known IP Addresses
    74.125.0.0/16


fetch libfetch
Type:
    Command Line Utility
User Agent String:
    fetch libfetch/2.0
Known IP Addresses:
    Any
Note:
    FreeBSD download tool. I block this user agent.


GigablastOpenSource
Type:
    Search Engine (internal)
User Agent Strings:
    GigablastOpenSource/1.0
    Mozilla/5.0 (compatible; GigablastOpenSource/1.0)
Known IP Addresses:
    Any
Note:
    Used by businesses as an internal search engine. I block this user agent.


Gluten Free Crawler
Type:
    Research
User Agent String:
    Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
Known IP Addresses:
    104.131.147.112
Note:
    I block this user agent.


Go-http-client
Type:
    Client
User Agent String:
    Go-http-client/2.0
Known IP Addresses:
    Any
Note:
    I block this user agent.


gocrawl
Type:
    Crawler (not a search engine)
User Agent String:
    Mozilla/5.0 (Windows NT 6.1; rv:15.0) gocrawl/0.4 Gecko/20120716 Firefox/15.0a2
Known IP Addresses:
    Any
Note:
    I block this user agent.


Google
Type:
    Developer
User Agent String:
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
Known IP Addresses:
    66.102.0.0 - 66.102.15.255 (66.102.0.0/20)


Googlebot
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Known IP Addresses:
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)


Google-Site-Verification
Type:
    Search Engine (webmaster tools)
User Agent String:
    Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
Known IP Addresses:
    72.14.192.0 - 72.14.255.255 (72.14.192.0/18)
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)


Google-Structured-Data-Testing-Tool
Type:
    Search Engine (webmaster tools)
User Agent String:
    Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)
Known IP Addresses:
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)


Googleweblight
Google Favicon
Type:
    Search Engine
User Agent Strings:
    Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Known IP Addresses:
    66.102.0.0 - 66.102.15.255 (66.102.0.0/20)
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)
Note:
    Serves optimized web pages for slow clients (less than 3G).


Goose
Type:
    Scraper
User Agent String:
    Goose/3.1.6
Known IP Addresses:
    Google Cloud
Note:
    I block this bot.


Grammarly
Type:
    Service
User Agent String:
    Grammarly/1.0 (http://www.grammarly.com)
Known IP Addresses:
    Amazon ranges
Note:
    User agent spam. I block this user agent.


GrapeshotCrawler
Type:
    Advertising
User Agent Strings:
    Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
    Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1. 4 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Known IP Addresses:
    148.64.56.0 - 148.64.56.255 (148.64.56.0/24)


GroupHigh
Type: Search Marketing
User Agent Strings:
    Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
    Mozilla/5.0 (compatible; GroupHigh/1.1; +http://www.grouphigh.com/)
Known IP Addresses:
    Random ARIN IP ranges.
Note: I block this user agent.


HeadlessChrome
Type:
    Web Browser
User Agent String:
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/67.0.3396.99 Safari/537.36
Known IP Addresses:
    Any
Note: It's for developers, but it can easily be abused. I block this user agent.


HTTP
Type:
    Unknown
User Agent String:
    HTTP
Known IP Addresses:
    113.35.251.98 (JP)
Note:
    I block this user agent.


HTTPie
Type:
    Command Line Utility
User Agent String:
    HTTPie/1.0.2
Known IP Addresses:
    Any
Note:
    Similar to cURL and wget. I block this user agent.


ia_archiver
Type:
    Analytics
User Agent String:
    Mozilla/5.0 (compatible; ia_archiver/1.0; +http://www.alexa.com/help/webmasters; crawler@alexa.com)
Known IP Addresses:
    Amazon ranges


IAS crawler
Type:
    Advertising
User Agent String:
    IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)
Known IP Addresses:
    198.148.15.0 - 198.148.15.255 (198.148.15.0/24)


ips-agent
Type:
    Service (monitoring and security)
User Agent String:
    Mozilla/5.0 (compatible; ips-agent)
Known IP Addresses:
    69.58.176.0 - 69.58.191.255 (69.58.176.0/20)
Note:
    I block this user agent.


Java
Type:
    Programming Language
User Agent String:
    Java/1.8.0_212
Known IP Addresses
    Any
Note:
    I block this user agent.


Jersey
Type:
    Client
User Agent String:
    Jersey/2.25.1 (Apache HttpClient 4.5)
Known IP Addresses:
    Any
Note:
    A variation of Apache-HttpClient version 4.5. I block this user agent.


libwww-perl
Type:
    Client
User Agent String:
    (deleted by mistake)
Known IP Addresses:
    Any
Note:
    A collection of perl modules. I block this user agent.


Linguee Bot
Type:
    Service (translation)
User Agent String:
    Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
Known IP Addresses:
    85.93.93.147


linkdexbot
Type:
    Search Marketing
User Agent String:
    libwww-perl/6.05
Known IP Addresses:
    Any
Note:
    I block this user agent.


LivelapBot
Type:
    Social Media
User Agent String:
    LivelapBot/0.2 (http://site.livelap.com/crawler)
Known IP Addresses:
    192.99.1.145
    198.27.82.109
Note: site.livelap.com works but livelap.com and www.livelap.com both indicate the domain is not set up. I cannot register at site.livelap.com. I block this user agent.


looid.com
Type:
    Search Engine
User Agent String:
    looid.com Search/0.1
Known IP Addresses:
    85.214.196.80


ltx71
Type:
    Research
User Agent String:
    ltx71
Known IP Addresses:
    Unknown in Google Cloud (deleted from access logs)
Note: I block this user agent.


lua-resty-http
Type:
    Nginx Scripting Module
User Agent String:
    ua-resty-http/0.10 (Lua) ngx_lua/10000
Known IP Addresses:
    Any
Note:
    This is an Nginx web server module. I block this user agent.


Mail.RU_Bot
Type:
    Search Engine (Russia)
User Agent Strings:
    Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
    Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
    Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
Known IP Addresses:
    95.163.248.0 - 95.163.255.255 (95.163.248.0/21)
    217.69.128.0 - 217.69.143.255 (217.69.128.0/20)


MauiBot
Type:
    Research
User Agent String:
    MauiBot (crawler.feedback+dc@gmail.com)
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


MBCrawler
Type:
    Service (monitors backlinks)
User Agent String:
    MBCrawler/1.0 (https://monitorbacklinks.com/robot)
Known IP Addresses:
    Amazon ranges
Note:
    This is a pay service. I block this user agent.


Mediapartners-Google
Type:
    Advertising
User Agent String:
    Mediapartners-Google
Known IP Addresses
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)
    72.14.192.0 - 72.14.255.255 (72.14.192.0/18)


MixnodeCache
Type:
    Data Collection
User Agent String:
    MixnodeCache/1.8(+https://cache.mixnode.com/)
Known IP Addresses
    Amazon ranges
Note:
    I block this user agent.


MJ12bot
Type:
    Search Engine (decentralized)
User Agent:
    Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/
Known IP Addresses:
    Random datacenter IP addresses worldwide
Note:
    Majestic-12 offers a lot of information about your website, but only if you let the MJ12bot access your website.


MojeekBot
Type:
    Search Engine
User Agent String:
    Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Known IP Addresses
    5.102.173.64 - 5.102.173.79 (5.102.173.64/28)


nadesiko
Type:
    Unknown
User Agent String:
    nadesiko
Known IP Addresses:
    113.35.251.98 (JP)
Note:
    I block this user agent.


NetcraftSurveyAgent
Type:
    Analysis
User Agent String:
    Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Known IP Addresses:
    DigitalOcean ranges within 134.209.0.0 - 134.209.255.255 (134.209.0.0/16)
    DigitalOcean ranges within 167.71.0.0 - 167.71.255.255 (167.71.0.0/16)
    DigitalOcean ranges within 167.99.0.0 - 167.99.255.255 (167.99.0.0/16)


newspaper
Type:
    Unknown
User Agent String:
    newspaper/0.2.2
Known IP Addresses:
    Any
Note:
    Searching for this user agent returns too many results. I block this user agent.


NinjaBot
Type:
    Search Marketing
User Agent String:
    NinjaBot/2.0 (+http://www.imninjas.com)
Known IP Addresses:
    72.10.193.64 - 72.10.193.127 (72.10.193.64/26)
Note:
    Redirects to internetmarketingninjas.com. I block this user agent.


oBot
Type:
    Research and Development
User Agent String:
    Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)
Known IP Addresses:
    194.153.113.0 - 194.153.113.255 (194.153.113.0/24)
    206.253.224.0 - 206.253.225.255 (206.253.224.0/23)
    206.253.226.0 - 206.253.226.255 (206.253.226.0/24)
Note:
    This is the IBM Germany Research and Development Content Security Division. They use website data to build content filters.


omgili
Type:
    Data Extraction
User Agent String:
    omgili/0.5 +http://omgili.com
Known IP Addresses:
    82.166.195.64
Note:
    It used to be a search engine, but now it feeds data to webhose.io. I now block this user agent.


Pandalytics
Type:
    Service (retrieves information about your domain name)
User Agent String:
    Pandalytics/1.0 (https://domainsbot.com/pandalytics/)
Known IP Addresses:
    Amazon ranges
Note:
    Maintains a domain name database. I block this user agent.

Pinterestbot
Type:
    Social Media
User Agent String:
    Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
Known IP Addresses:
    154.236.1.0 - 54.236.1.255 (54.236.1.0/24)


PocketParser
Type:
    Service and Web Browser Extension (saves pages)
User Agent String:
    PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
Known IP Addresses:

    Amazon ranges


proximic
Type:
    Advertising
User Agent String:
    Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
Known IP Addresses:
    Amazon ranges


python-requests
Type:
    Scraper
User Agent Strings:
    python-requests/2.18.4
    python-requests/2.21.0
Known IP Addresses:
    Any
Note:
    A URL fetcher. I block this user agent.


Python-urllib
Type:
    Scraper
User Agent Strings:
    Python-urllib/1.17
    Python-urllib/2.7
Known IP Addresses:
    Any
Note:
    A URL fetcher. I block this user agent.


RankingBot2
Type:
    Research
User Agent String:
    RankingBot2 -- https://varocarbas.com/bot_ranking2/
Known IP Addresses:
    81.43.124.145
Note:
    Controlled by an individual in Spain. I block this user agent.


Ruby
Type:
    Programming Language
User Agent Strings:
    Ruby
    Ruby, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
Known IP Addresses:
    Any
Note:
    I block this user agent.


Scrapy
Type:
    Scraper
User Agent Strings:
    Scrapy/1.5.1 (+https://scrapy.org)
    Scrapy/1.5.2 (+https://scrapy.org)
Known IP Addresses:
    Any
Note:
    This is a Python script. I block this user agent.


Screaming Frog SEO Spider
Type:
    Multi Platform Software (Windows, Macintosh, Linux)
User Agent String:
    Screaming Frog SEO Spider
Known IP Addresses:
    Any
Note:
    I block this user agent.


Seekport Crawler
Type:
    Search Engine (Germany)
User Agent String:
    Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)
Known IP Addresses:
    159.69.189.215 (Hetzner Online)


SemrushBot
Type:
    Search Marketing
User Agent Strings:
    Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)
    Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
    Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
Known IP Addresses:
    85.208.96.0 - 85.208.99.255 (85.208.96.0/22)
    IP addresses within the 46.229.168.0 - 46.229.168.255 range
    IP addresses within the 213.174.152.0 - 213.174.152.255 range


SEOkicks
Type:
    Service (backlink monitor)
User Agent String:
    Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)
Known IP Addresses:
    95.216.0.0 - 95.217.255.255 (95.216.0.0/16) (Hetzner Online)


serpstatbot
Type:
    Search Marketing
User Agent String:
    serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)
Known IP Addresses:
    136.243.155.105 (Hetzner Online)
Note:
    I block this user agent.


SeznamBot
Type:
    Search Engine (Czech Republic)
User Agent Strings:
    Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)
    Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
 Known IP Addresses:
    77.75.72.0 - 77.75.79.255 (77.75.72.0/21)
    185.66.188.0 - 185.66.191.255 (185.66.188.0/22)


SiteChecker
Type:
    Unknown
User Agent String:
    SiteChecker/1.0
Known IP Addresses:
    Amazon ranges (KR area)
Note:
    I block this user agent.


Slackbot-LinkExpanding
Type:
    Meta Content Extractor
User Agent String:
    Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Known IP Addresses:
    Amazon Ranges


slick-proxy
Type:

    VPN Proxy
User Agent String:
    AppEngine-Google; (+http://code.google.com/appengine; appid: p~slick-proxy)
Known IP addresses:
    107.178.193.197 (Google Cloud)
Note:
    This isn't a bot. It's an application.


SMTBot
Type:
    Analysis
User Agent Strings:
    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)
    Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko)                 Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


snapchat-proxy
Type:
    Social Media Proxy
User Agent String
    AppEngine-Google; (+http://code.google.com/appengine; appid: s~snapchat-proxy)
Known IP Addresses:
    35.187.132.135 (Google Cloud)
    107.178.194.101 (Google Cloud)
Note:
    This isn't a bot. It's an application.


Sogou web spider
Type:
    Search Engine (China)
User Agent String:
    Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Known IP Addresses:
    Random IP addresses in China


startmebot
Type:
    Bookmarking Service
User Agent Strings:
    Mozilla/5.0 (compatible; startmebot/1.0; +https://start.me/bot)
    Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
Known IP Addresses:
    Amazon ranges


Streamline3Bot
Type:
    Web Directory
User Agent Strings:
    Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) Streamline3Bot/1.0
    Mozilla/5.0 (Windows NT 6.1; Win64; x64; +https://www.ubtsupport.com/legal/Streamline3Bot.php) Streamline3Bot/1.0
Known IP Addresses:
    OVH ranges
Note:
    I block this user agent.


SurdotlyBot
Type:
    Link Scanner
User Agent String:
    Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Known IP Addresses:
    Amazon ranges
Note:
    User Agent Spam. I block this user agent.


TelegramBot
Type:
    Messaging Application
User Agent String:
    TelegramBot (like TwitterBot)
Known IP Addresses:
    149.154.160.0 - 149.154.163.255 (149.154.160.0/22)


The Knowledge AI
Type:
    Unknown
User Agent String:
    The Knowledge AI
Known IP Addresses:
    64.62.128.0 - 64.62.255.255 (64.62.128.0/17)
Note:
    I block this user agent.


TrendsmapResolver
Type:
    Social Media
User Agent String:
    Mozilla/5.0 (compatible; TrendsmapResolver/0.1)
Known IP Addresses:
    88.99.145.42  (Hetzner Online)
Note:
    Gathers information for Twitter trends. I block this user agent.


TweetmemeBot
Type:
    Link Scanner and Data Collection
User Agent String:
    Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
Known IP Addresses:
    46.236.0.0 - 46.236.63.255 (46.236.0.0/19)
Note:
    Gathers information from Twitter. I block this user agent.


Twitterbot
Type:
    Social Media
User Agent String:
    Twitterbot/1.0
Known IP Addresses:
    199.16.156.0 - 199.16.159.255 (199.16.156.0/22)
    199.59.148.0 - 199.59.151.255 (199.59.148.0/22)


Uptimebot
Type:
    Service (monitors site uptime)
User Agent String:
    Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)
Known IP Addresses:
    109.201.138.228


vuhuvBot
Type:
    Search Engine (Turkey)
User Agent String:
    Mozilla/5.0 (compatible; vuhuvBot/1.0; +http://vuhuv.com/bot.html)
Known IP Addresses:
    185.93.54.0 - 185.93.54.255 (185.93.54.0/24)


WbSrch
Type:
    Search Engine (independent)
User Agent String:
    Mozilla/5.0 (compatible; WbSrch/1.1 +https://wbsrch.com)
Known IP Addresses:
    216.151.3.108


Wget
Type:
    Command Line Utility
User Agent Strings:
    Wget/1.15 (linux-gnu)
    Wget/1.19.4 (linux-gnu)
Known IP Addresses:
    Any
Note:
    Used for downloading files and web pages. I block this user agent.


Widget Server
Type:
    Unknown
User Agent String:
    Mozilla/5.0 (en-US) AppleWebKit/537.36 (KHTML, like Gecko; Widget Server) Chrome/41.0.2272.118 Safari/537.36
Known IP Addresses:
    66.102.0.0 - 66.102.15.255 (66.102.0.0/20)
    66.249.64.0 - 66.249.95.255 (66.249.64.0/19)
Note:
    These are Google IP addresses.


www.ru
Type:
    Domain Reseller
User Agent String:
    www.ru
Known IP Addresses:
    31.172.81.30 (DE)
Note:
    User agent spam. I block this user agent.


Xenu Link Sleuth
Type:
    Windows Software
User Agent String:
    Xenu Link Sleuth/1.3.8
Known IP Addresses:
    Any
Note:
    I block this user agent.


yacybot
Type:
    Search Engine (peer-to-peer)
User Agent String:
    yacybot (/global; amd64 Windows Server 2016 10.0; java 1.8.0_211; America/en) http://yacy.net/bot.html
Known IP Addresses:
    Any


Yahoo! Slurp
Type:
    Search Engine
User Agent Strings:
     Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
     Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)
Known IP Addresses
    72.30.0.0 - 72.30.255.255 (72.30.0.0/16)
    74.6.0.0 - 74.6.255.255 (74.6.0.0/16)
    217.146.176.0 - 217.146.178.255 (217.146.176.0/21)


YandexBot
YandexImages
Type: Search Engine (Russia)
User Agent Strings:
    Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
    Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Known IP Addresses:
    5.45.192.0 - 5.45.255.255 (5.45.192.0/18)
    5.255.192.0 - 5.255.255.255 (5.255.192.0/18)
    37.9.64.0 - 37.9.127.255 (37.9.64.0/18)
    37.140.128.0 - 37.140.191.255 (37.140.128.0/18)
    77.88.0.0 - 77.88.63.255 (77.88.0.0/18)
    84.201.128.0 - 84.201.191.255 (84.201.128.0/18)
    87.250.224.0 - 87.250.255.255 (87.250.224.0/19)
    93.158.128.0 - 93.158.191.255 (93.158.128.0/18)
    95.108.128.0 - 95.108.255.255 (95.108.128.0/17)
    100.43.64.0 - 100.43.95.255 (100.43.64.0/19)
    130.193.32.0 - 130-193.63.255 (130.193.32.0/19)
    141.8.128.0 - 141.8.191.255 (141.8.128.0/18)
    178.154.128.0 - 178.154.255.255 (178.154.128.0/17)
    199.21.96.0 - 199.21.99.255 (199.21.96.0/22)
    213.180.192.0 - 213.180.223.255 (213.180.192.0/19)


Yeti
Type:
    Search Engine (Korea)
User Agent String:
    Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/spd)
Known IP Addresses:
    125.209.192.0 - 125.209.255.255 (125.209.192.0/18)


Zoombot
Type:
    SEO
User Agent String:
    ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)
Known IP Addresses:
    Amazon ranges
Note:
    I block this user agent.


zgrab
Type:
    Research
User Agent String:
    Mozilla/5.0 zgrab/0.x (compatible; Researchscan/t13rl; http://researchscan.comsys.rwth-aachen.de)
Known IP Addresses:
    137.226.0.0 - 137.226.255.255 (137.226.0.0/16)
Note:
    I block this user agent.