3x detected (2x unique)
SetLinks bot,SetLinks.ru Crawler
Example user agent
Mozilla/5.0 (compatible; SetLinks bot 1.2) 3x detected (1x unique)
proximic
Example user agent
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.53 3x detected (1x unique)
diffbot
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15 2x detected (1x unique)
Smartsite HTTPClient
Example user agent
Smartsite HTTPClient - Mozilla/4.0 (compatible; MSIE 6.0) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.820 8x detected (3x unique)
Googlebot,Google Bot,Google
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 4x detected (1x unique)
Wget
Example user agent
Wget/1.10.2 2x detected (1x unique)
Adbeat Bot
Example user agent
Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.808 2x detected (1x unique)
BCKLINKS
Example user agent
BCKLINKS 1.0 4x detected (1x unique)
Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1) 3x detected (1x unique)
MarketwireBot
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com) 2x detected (1x unique)
Sitemap Generator
Example user agent
Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php) 2x detected (1x unique)
Digg Feed Fetcher
Example user agent
Digg Feed Fetcher 1.0 (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.48 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.50 2x detected (1x unique)
kgbody
Example user agent
Mozilla/5.0 (compatible; kgbody/2.0; +http://kids.goo.ne.jp/tool/kgbody.php) 5x detected (2x unique)
TweetmemeBot,Tweetmeme Bot
Example user agent
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0 2x detected (1x unique)
RMSnapKit
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) RMSnapKit/1.0 8x detected (2x unique)
Bing,bingbot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm) 3x detected (1x unique)
GLBot
Example user agent
Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.803 3x detected (1x unique)
LinkStats Bot
Example user agent
LinkStats Bot 2x detected (1x unique)
Web Glance
Example user agent
Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1 4x detected (2x unique)
PHP,CakePHP
Example user agent
CakePHP 5x detected (3x unique)
Daumoa,Daum,Daumoa/3.0
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0 2x detected (1x unique)
Rival IQ Bot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com) 6x detected (1x unique)
360spider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider 3x detected (1x unique)
Curl
Example user agent
curl/7.35.0 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.75 3x detected (1x unique)
memorybot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB) 4x detected (2x unique)
XML Sitemaps Generator,XML Sitemaps Generator/1.0
Example user agent
Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0 7x detected (3x unique)
Googlebot,Google PageSpeed Insights,Google
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1 4x detected (1x unique)
Google Wireless Transcoder
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14 2x detected (1x unique)
Google Image Proxy
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com) 3x detected (1x unique)
Google Keyword Suggestion
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16 2x detected (1x unique)
Second Live Client
Example user agent
Mozilla/5.0 (Windows; U; ; en-US) AppleWebKit/533.3 (KHTML, like Gecko) SecondLife/3.7.23.297296 (Second Life Release; default skin) Safari/533.3 3x detected (1x unique)
metager2-verification-bot
Example user agent
Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php) 7x detected (5x unique)
Google Ads Bot,AdsBot-Google,AdsBot Google-Mobile,Googlebot,AdsBot Google
Example user agent
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari 3x detected (1x unique)
LinksCrawler
Example user agent
LinksCrawler 0.1beta 2x detected (1x unique)
Incoming Links - WordPress Plugin
Example user agent
MonitorBacklinksWP (+http://monitorbacklinks.com/blog/incoming-links/) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/6.05 6x detected (3x unique)
MJ12bot,MJ12 Bot,MJ12bot/v1.4.3
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+) 4x detected (3x unique)
Bing Preview,BingPreview,BingBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML; like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 BingPreview/1.0b 4x detected (2x unique)
New-Sogou-Spider,Sogou Spider
Example user agent
New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98) 3x detected (1x unique)
XML Sitemaps Generator
Example user agent
Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0 3x detected (1x unique)
CrawlRobot
Example user agent
Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php) 2x detected (1x unique)
E-Mail Siphon
Example user agent
EmailSiphon 6x detected (3x unique)
googlebot,Google Bot,Google
Example user agent
mozilla/5.0 (iphone; cpu iphone os 8_3 like mac os x) applewebkit/600.1.4 (khtml, like gecko) version/8.0 mobile/12f70 safari/600.1.4 (compatible; googlebot/2.1; +http://www.google.com/bot.html) 2x detected (1x unique)
PycURL
Example user agent
PycURL/7.22.0 3x detected (1x unique)
Curl
Example user agent
curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.63 2x detected (1x unique)
vBulletin Bot
Example user agent
vBulletin via PHP 3x detected (2x unique)
com_bot,LinksManager
Example user agent
Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html) 3x detected (1x unique)
HRCrawler
Example user agent
HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko) 4x detected (3x unique)
com_bot,LinksManager,linksmanager_bot
Example user agent
Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html) 4x detected (1x unique)
SimplePie
Example user agent
SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111015034325 7x detected (5x unique)
Google Mediapartners,Mediapartners-Google,AdSense Bot,Googlebot,MediaPartners Google
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) 2x detected (1x unique)
Google Adwords DisplayAds WebRender
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36 4x detected (2x unique)
PHP,Guzzle Http Client
Example user agent
GuzzleHttp/6.1.0 curl/7.45.0 PHP/5.6.14 2x detected (1x unique)
PagePeeker Screenshot Maker
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) domain.com Safari/538.1 2x detected (1x unique)
Daumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) 2x detected (1x unique)
Shrook
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/434l 4x detected (2x unique)
ArchiveBot,ArchiveTeam ArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36 5x detected (2x unique)
Daumoa,Daum
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0 3x detected (2x unique)
Google Image Proxy,Gmail Image Proxy
Example user agent
Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.805 2x detected (1x unique)
RSS2HTML
Example user agent
FeedForAll rss2html.php v2 3x detected (1x unique)
Curl
Example user agent
curl/7.29.0 5x detected (4x unique)
GooglePlusBot,Google Web Snippet,Googlebot,Googlebot snippet
Example user agent
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/) 3x detected (1x unique)
FyberSpider
Example user agent
FyberSpider (+http://www.fybersearch.com/fyberspider.php) 4x detected (1x unique)
YioopBot
Example user agent
Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php) 4x detected (2x unique)
GrapeshotCrawler,GrapeshotCrawler/2.0
Example user agent
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) 3x detected (1x unique)
memorybot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB) 5x detected (3x unique)
Butterfly,Butterfly Robot,topsy Butterfly
Example user agent
Mozilla/5.0 (Macintosh; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8 2x detected (1x unique)
libcurl
Example user agent
libcurl-agent/1.0 3x detected (2x unique)
Webcrawler,HubSpot Webcrawler
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler 4x detected (2x unique)
Google Page Speed,Google PageSpeed Insights
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 4x detected (3x unique)
Bing Preview,BingPreview,BingBot
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b) 4x detected (3x unique)
Bing Preview,BingPreview,BingBot
Example user agent
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0) BingPreview/1.0b 2x detected (1x unique)
Second Live Commerce Client
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0 ; .NET CLR 2.0.50215; SL Commerce Client v1.0; Tablet PC 2.0 3x detected (2x unique)
Google Image Proxy,Gmail Image Proxy
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com GoogleImageProxy) 4x detected (2x unique)
PHP,Guzzle Http Client
Example user agent
Guzzle/4.2.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.11 5x detected (1x unique)
360spider
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.69 5x detected (2x unique)
Google Web Preview,Googlebot
Example user agent
Mozilla/5.0 (iPad; CPU OS 5_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/5.1 Mobile/9A334 Safari/7534.48.3 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.76 8x detected (5x unique)
Googlebot Mobile,Googlebot-Mobile,Google Bot Mobile,Googlebot,Google
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 4x detected (2x unique)
GomezAgent,Gomez Site Monitor
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1 2x detected (1x unique)
open-webkit-sharp
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) OpenWebKitSharp 8x detected (5x unique)
Facebook,FacebookBot,FacebookExternalHit,Facebook External Hit,FacebookExternalHit/1.1
Example user agent
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) 2x detected (1x unique)
SiteCon
Example user agent
Mozilla/5.0 (Linux; U; en-gb) AppleWebKit/418.9.1 (KHTML, like Gecko) SiteCon/8.10.22 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.814 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.36 2x detected (1x unique)
dead link checker
Example user agent
www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 4x detected (2x unique)
Google Page Speed,Google PageSpeed Insights
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 4x detected (1x unique)
Wget
Example user agent
Wget/1.12 (linux-gnu) 7x detected (3x unique)
Yahoo! Slurp,Inktomi Slurp,Yahoo
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) 2x detected (1x unique)
vSentry
Example user agent
Mozilla/5.0 (Windows; chromeframe/2.4.8.5746) AppleWebKit/1.0 (KHTML, like Gecko) Bromium Safari/1.0 4x detected (2x unique)
Daumoa,Daum
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 2x detected (1x unique)
Adbeat Bot
Example user agent
Mozilla/5.0 (X11; U; Linux x86; de-de) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4 6x detected (3x unique)
MJ12bot,MJ12 Bot,MJ12bot/v1.4.4
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) 2x detected (1x unique)
profiller
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) profiller Safari/537.21 2x detected (1x unique)
WebThumb
Example user agent
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0 4x detected (3x unique)
MetaURI Bot,MetaURI API/2.0,MetaURI
Example user agent
MetaURI API/2.0 +metauri.com 2x detected (1x unique)
Web Glance
Example user agent
Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1 3x detected (1x unique)
GIDBot
Example user agent
GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php) 5x detected (3x unique)
Spinn3r,Spinn3r RSS Aggregator,Spinn3r 3.1
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19 2x detected (1x unique)
Embed PHP Library
Example user agent
Embed PHP Library 2x detected (1x unique)
Anonymizied
Example user agent
Mozilla/5.0 (Randomized by FreeSafeIP.com/upgrade-to-remove; compatible; MSIE 9.0; Windows NT 5.0WOW64) Chrome/18.0.1229.79 2x detected (1x unique)
Adbeat Bot
Example user agent
Mozilla/5.0 (X11; U; Linux x86; nl-NL) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.800 (+http://passoire.afraid.org/mylittlewebsurvey/index.html) 4x detected (1x unique)
Wget
Example user agent
Wget/1.14 (linux-gnu) 7x detected (4x unique)
Bing,MsnBot,msnbot-media,BingBot
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b 4x detected (2x unique)
JamesBOT,WebCrawler
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.821 2x detected (1x unique)
Lightspeed Systems Crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0;Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727;.NET CLR 3.0.04506.30) Lightspeedsystems 4x detected (3x unique)
Baiduspider,Baidu Mobile Search,Baidu Spider
Example user agent
Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.816 2x detected (1x unique)
JobBoerse Bot
Example user agent
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0 2x detected (1x unique)
Shrook
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/1290y 6x detected (4x unique)
Bing,msnbot-media,BingBot,MSNBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/534+ (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 MsnBot-Media /1.0b 4x detected (2x unique)
PHP,Guzzle Http Client
Example user agent
GuzzleHttp/6.1.0 curl/7.38.0 PHP/7.0.0RC7 4x detected (1x unique)
Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot) 3x detected (1x unique)
SMTBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.64 3x detected (2x unique)
SetLinks bot,SetLinks.ru Crawler
Example user agent
SetLinks bot 1.0 4x detected (2x unique)
PHP,Guzzle Http Client
Example user agent
GuzzleHttp/6.1.0 curl/7.35.0 PHP/5.5.9-1ubuntu4.14 2x detected (1x unique)
akregator
Example user agent
Mozilla/5.0 (X11;) AppleWebKit/534.34 (KHTML, like Gecko) akregator/4.14.1 Safari/534.34 5x detected (1x unique)
360spider
Example user agent
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider 5x detected (2x unique)
Google Web Preview,Googlebot
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36 2x detected (1x unique)
Baidu Mobile Search
Example user agent
Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spi_der.html) 5x detected (1x unique)
360spider
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider 4x detected (3x unique)
Bing Preview,BingPreview,BingBot
Example user agent
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko 4x detected (1x unique)
PHP
Example user agent
PHP/5.3.77 4x detected (2x unique)
SurveyBot,Survey Bot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools) 5x detected (2x unique)
Facebook,facebookscraper
Example user agent
facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php) 2x detected (1x unique)
Google Toolbar
Example user agent
Mozilla/4.0 (compatible; GoogleToolbar 7.5.5111.1712; Windows XP 5.1; MSIE 8.0.6001.18702) 2x detected (1x unique)
FlipboardProxy
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.65 3x detected (2x unique)
NewsBlur,NewsBlur Feed Fetcher
Example user agent
NewsBlur Feed Fetcher - 0 subscribers - http://www.newsblur.com/site/1719739/nouvelles-cinema (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3) 3x detected (2x unique)
Perl,libwww
Example user agent
libwww-perl/5.800 8x detected (4x unique)
Facebook,FacebookBot,FacebookExternalHit,Facebook External Hit
Example user agent
facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php) 4x detected (2x unique)
Linkdex Bot,linkdexbot
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/) 1x detected (1x unique)
PHP Server Monitor
Example user agent
Mozilla/5.0 (compatible; phpservermon/3.1.1; +http://www.phpservermonitor.org) 9x detected (3x unique)
Yahoo! Slurp,Inktomi Slurp,Yahoo
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5 7x detected (4x unique)
Exabot Thumbnails,Exabot,Exalead,Exabot-Thumbnails
Example user agent
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) 1x detected (1x unique)
Daum
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1 2x detected (2x unique)
BountiiBot,Bountii Bot
Example user agent
Mozilla/5.0 (compatible; BountiiBot/1.1; +http://bountii.com/contact.php) 4x detected (2x unique)
Daumoa,Daum
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0; 4x detected (3x unique)
NewsBlur,Safari,NewsBlur Favicon Fetcher
Example user agent
NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3) 1x detected (1x unique)
Baidu Spider
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; baidu Transcoder;) 3x detected (2x unique)
FeedWrangler,Feed Wrangler
Example user agent
Feed Wrangler/1.0 (3 subscribers; feed-id=248559; http://feedwrangler.net; Allow like Gecko) 1x detected (1x unique)
Googlebot
Example user agent
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36 3x detected (2x unique)
NewsBlur,NewsBlur Feed Fetcher
Example user agent
NewsBlur Feed Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3) 3x detected (2x unique)
websitepulse checker,WebSitePulse
Example user agent
websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux) 5x detected (3x unique)
Apache Nutch,Nutch,Nutch-based Bot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7 4x detected (2x unique)
Google Page Speed,Google PageSpeed Insights
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/4.0.5 Mobile/8A306 Safari/6531.22.7 1x detected (1x unique)
Google Partner Monitoring
Example user agent
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1 google_partner_monitoring FWSzVTDDBz14547302713138T 5x detected (3x unique)
Bing Preview,BingPreview,BingBot
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b 3x detected (2x unique)
NewsBlur,NewsBlur Favicon Fetcher
Example user agent
NewsBlur Page Fetcher - 7 subscribers - http://www.newsblur.com/site/3966817/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3) 2x detected (1x unique)
NewsBlur
Example user agent
NewsBlur Feed Finder (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3) 4x detected (1x unique)
Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot) 4x detected (1x unique)
SimplePie
Example user agent
SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20130514092120 2x detected (2x unique)
YRSpider,Yunyun Bot
Example user agent
Mozilla/5.0 (compatible; YRSpider; +http://www.yunyun.com/SiteInfo.php?r=about) 5x detected (1x unique)
360spider
Example user agent
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider 6x detected (4x unique)
Butterfly,Butterfly Robot,Butterfly/1.0 a,topsy Butterfly
Example user agent
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8 3x detected (1x unique)
Fever
Example user agent
Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko) 5x detected (4x unique)
Yandex Newslinks,YandexNewslinks,Yandex Bot,YandexBot
Example user agent
Mozilla/5.0 (compatible; YandexNewslinks; +http://yandex.com/bots) 1x detected (1x unique)
Baidu Spider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8;baidu Transcoder) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729) 4x detected (1x unique)
SimplePie
Example user agent
SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20121030175911 8x detected (3x unique)
Googlebot,Google Bot,Google
Example user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36 5x detected (3x unique)
Bing Preview,BingPreview,BingBot
Example user agent
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/538.14 (KHTML, like Gecko) BingPreview/1.0b 4x detected (1x unique)
Wget
Example user agent
Wget/1.6 8x detected (2x unique)
Bing,bingbot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) 3x detected (3x unique)
Google App Engine,AppEngine-Google,Google AppEngine
Example user agent
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~chengzhoukun) 4x detected (1x unique)
Wget
Example user agent
Wget/1.15 (linux-gnu) 3x detected (3x unique)
Google App Engine,AppEngine-Google,Google AppEngine
Example user agent
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko AppEngine-Google; (+http://code.google.com/appengine; appid: s~oowsxs) 7x detected (5x unique)
Google Ads Bot,AdsBot-Google,AdsBot Google-Mobile,Googlebot,AdsBot Google
Example user agent
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 2x detected (2x unique)
Google App Engine,Google AppEngine
Example user agent
Mozilla/5.0 (Android; Mobile; rv:29.0) Gecko/29.0 Firefox/29.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~oldecreator) 3x detected (3x unique)
Google App Engine,AppEngine-Google,Google AppEngine
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1626.1 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~hex-plus) 8x detected (3x unique)
Googlebot,Google Bot,Google
Example user agent
Mozilla/5.0 AppleWebKit/537.4 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.4 3x detected (2x unique)
OrangeBot,Orange Bot
Example user agent
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com) 2x detected (2x unique)
PowerMapper,CrawlerProcess
Example user agent
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.10.736.0 2x detected (2x unique)
Bing Preview,BingBot
Example user agent
Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Nokia; Lumia 520) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Mobile Safari/537.36 Edge/12.0 BingPreview/1.0b 7x detected (3x unique)
Googlebot,Google PageSpeed Insights,Google
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1 8x detected (3x unique)
Googlebot,Google Web Preview,Google
Example user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 2x detected (2x unique)
FAST Web Crawler,FAST-WebCrawler
Example user agent
Opera/30.0.1835.59 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/29.16 AppleWebKit/599.0+ (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 FAST-WebCrawler/3.8 3x detected (2x unique)
Google Web Preview,Googlebot
Example user agent
Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.4 4x detected (1x unique)
Wget
Example user agent
Wget/1.16.1 (darwin14.0.0) 2x detected (2x unique)
Bing Preview,BingBot
Example user agent
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b) 4x detected (1x unique)
Wget
Example user agent
Wget/1.11.4 6x detected (3x unique)
Google Web Preview,Google,Googlebot
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13 3x detected (1x unique)
Curl
Example user agent
curl/7.16.0 (i386-unknown-openbsd4.1) libcurl/7.16.0 OpenSSL/0.9.7j zlib/1.2.3 2x detected (2x unique)
Bing Preview,BingBot
Example user agent
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.11067 BingPreview/1.0b 2x detected (1x unique)
GomezAgent
Example user agent
Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT) 3x detected (1x unique)
Curl
Example user agent
curl/7.14.1 (i386-portbld-freebsd4.7) libcurl/7.14.1 OpenSSL/0.9.8 zlib/1.1.3 4x detected (2x unique)
ArchiveBot,ArchiveTeam ArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36 7x detected (5x unique)
Bing,FacebookBot,BingBot,Googlebot,Google
Example user agent
(Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.4.5.3000 Firefox/38 7x detected (5x unique)
Bing,FacebookBot,BingBot,Googlebot,Google
Example user agent
(Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.0 1x detected (1x unique)
HeartRails Capture
Example user agent
Mozilla/5.0 (X11; U; Linux i686; xx; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18 8x detected (4x unique)
Yahoo! Japan,YFSBJ crawler,Y!J-AGENT,Yahoo
Example user agent
Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80) 6x detected (3x unique)
MJ12bot,MJ12 Bot,MJ12bot/v1.4.0
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+) 3x detected (2x unique)
Yahoo,Yahoo! Japan
Example user agent
Mozilla/5.0 (Linux; U; Android 2.3.3; ja-jp; sdk Build/GRI34; Y!J-BRZ/YATSHA crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 4x detected (1x unique)
SimplePie
Example user agent
SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111118194710 8x detected (4x unique)
Yahoo! Japan,YFSJ crawler,Y!J-AGENT,Yahoo
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248)
Comparison created
2016-05-17 07:49:28 | by
ThaDafinser