0x detected (0x unique)
Example user agent
Http Connector Spider, contact Alcatel-Lucent IDOL Search 2x detected (2x unique)
Crawllybot,Crawlboy
Example user agent
Crawllybot/0.1/0.1 (Crawllybot/0.1; http://www.crawlly.com; crawler@crawlly.com) 3x detected (3x unique)
Almaden,Fake Browser,www.almaden.ibm.com
Example user agent
http://www.almaden.ibm.com/cs/crawler 2x detected (2x unique)
NameProtect,abot
Example user agent
abot/0.1 (abot; http://www.abot.com; abot@abot.com) 4x detected (4x unique)
Googlebot,Google Bot Mobile,Googlebot Mobile,KDDI-Googlebot-Mobile
Example user agent
KDDI-CA34 UP.Browser/6.2.0.10.2.2(GUI)MMP/2.0 (compatible; KDDI-Googlebot-Mobile/2.1; http://www.google.com/bot.html) 2x detected (1x unique)
RedCarpet
Example user agent
RedCarpet/1.0 (http://www.redcarpet-inc.com/robots.html) 1x detected (1x unique)
AdnormCrawler
Example user agent
AdnormCrawler www.adnorm.com/crawler 2x detected (1x unique)
DataparkSearch
Example user agent
DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/) 2x detected (1x unique)
MetaGeneratorCrawler
Example user agent
MetaGeneratorCrawler/1.3.4 (www.metagenerator.info) 1x detected (1x unique)
FatBot
Example user agent
Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler) 1x detected (1x unique)
InfuzApp
Example user agent
Mozilla/5.0 (compatible; InfuzApp/1.0; +http://www.infuz.com/bot.html) 1x detected (1x unique)
wume_crawler
Example user agent
wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) 3x detected (1x unique)
NewsGator
Example user agent
NewsGator/2.0 Bot (http://www.newsgator.com) 1x detected (1x unique)
Cloudview Crawler
Example user agent
Exalead Cloudview Crawler,gzip(gfe),gzip(gfe),gzip(gfe) 2x detected (2x unique)
Cydral Web Image Search,CydralSpider
Example user agent
CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com) 3x detected (3x unique)
80bot,80legs/0.71,80Legs
Example user agent
Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620 1x detected (1x unique)
Diffbot
Example user agent
Diffbot/0.1 1x detected (1x unique)
RSS-SPIDER
Example user agent
RSS-SPIDER (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php) 1x detected (1x unique)
VM-Crawler
Example user agent
Mozilla/4.0, VM-Crawler/cs version info ofni noisrev sc 1x detected (1x unique)
RDF crawler
Example user agent
OpenLink Virtuoso RDF crawler 2x detected (2x unique)
Bot Blocker Crawler,bot
Example user agent
Bot Blocker Crawler 1.0 (btw, IncrediBILL says "HI!") 1x detected (1x unique)
SRCCN!Spider
Example user agent
Mozilla/5.0 (compatible; SRCCN!Spider/1.1; +http://site.srccn.com/spider.html) 1x detected (1x unique)
Beehive Crawler
Example user agent
ZoomInfo::Beehive Crawler 1x detected (1x unique)
50.nu
Example user agent
50.nu/0.01 ( http://50.nu/bot.html ) 4x detected (3x unique)
YodaoBot,YodaoBot-Image,YodaoBot-Image/1.0
Example user agent
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;) 3x detected (2x unique)
Nusearch Spider,nuSearch
Example user agent
Nusearch Spider (compatible; MSIE 6.0) 3x detected (1x unique)
Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 2x detected (2x unique)
WinWebBot,WinWebBot/1.0
Example user agent
WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;) 2x detected (1x unique)
SMTBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot) 3x detected (2x unique)
FyberSpider,FyberSpider/1.3
Example user agent
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) 1x detected (1x unique)
enicura crawler
Example user agent
enicura crawler 1.0,gzip(gfe),gzip(gfe) 3x detected (1x unique)
NetSeer crawler
Example user agent
Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com) 5x detected (3x unique)
Yahoo! Japan,YFSBJ crawler,Y!J-AGENT
Example user agent
Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80) 2x detected (2x unique)
Infohelfer,Infohelfer/1.2.0
Example user agent
Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php) 2x detected (1x unique)
Findexa Crawler
Example user agent
Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) 3x detected (1x unique)
Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/2.3; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 2x detected (2x unique)
com/bot,SEOdiver Bot
Example user agent
Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot) 1x detected (1x unique)
Crawler
Example user agent
Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/) 1x detected (1x unique)
123metaspider-Bot
Example user agent
123metaspider-Bot (Version: 1.04, powered by www.123metaspider.com) 0x detected (0x unique)
Example user agent
Mozilla/5.0 (Linux; Android 4.0.4; VORTEX Build/IMM76D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19 2x detected (1x unique)
AddSugarSpiderBot
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) AddSugarSpiderBot www.idealobserver.com 1x detected (1x unique)
it-Crawler
Example user agent
Liquida.it-Crawler/1.0 ( crawler@liquida.it +http://www.liquida.it ) 1x detected (1x unique)
CosmixCrawler
Example user agent
CosmixCrawler/0.1 1x detected (1x unique)
BurstFind Crawler
Example user agent
BurstFind Crawler 1.1 - www.burstfind.com 1x detected (1x unique)
Content Crawler
Example user agent
Content Crawler 1x detected (1x unique)
TridentSpider
Example user agent
Mozilla/5.0 (compatible; TridentSpider/3.1) 1x detected (1x unique)
lworld spider
Example user agent
Mozilla/4.0 (compatible; lworld spider 1.0; Windows NT 5.1) 2x detected (2x unique)
FAST PartnerSite,PartnerSite Crawler
Example user agent
Fast PartnerSite Crawler 1x detected (1x unique)
Door Spider
Example user agent
FFC Trap Door Spider 1x detected (1x unique)
Crawler
Example user agent
1NET/Crawler (compatible; 1NET; robot; http://www.1NET.gr) 1x detected (1x unique)
BOT for JCE
Example user agent
BOT/0.1 (BOT for JCE) 4x detected (1x unique)
thumbshots-de-bot
Example user agent
thumbshots-de-bot (+http://www.thumbshots.de/) 3x detected (2x unique)
WiseNut search engine crawler,ZyBorg
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) 2x detected (2x unique)
SpiderLing,Generic Research Crawler
Example user agent
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/) 3x detected (1x unique)
woriobot
Example user agent
Mozilla/5.0 (compatible; woriobot +http://worio.com) 2x detected (2x unique)
PlantyNet,PlantyNet_WebRobot
Example user agent
PlantyNet_WebRobot_V1.9 babo@plantynet.com 2x detected (1x unique)
GoGuidesBot
Example user agent
GoGuidesBot/0.0.1 (GoGuides Indexing Spider; http://www.goguides.org/spider.html) 1x detected (1x unique)
SSL-Crawler
Example user agent
Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.de 2x detected (1x unique)
GarlikCrawler
Example user agent
GarlikCrawler/1.2 (http://garlik.com/ 1x detected (1x unique)
ADJUSTbot
Example user agent
Mozilla/5.0 (compatible; ADJUSTbot/2.0; +http://www.ad-just.jp/) 1x detected (1x unique)
LikaholixCrawler
Example user agent
Mozilla/5.0 (compatible; LikaholixCrawler/1.0; +http://mylikes.com/about/crawler) 2x detected (1x unique)
AskPeterBot
Example user agent
Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html) 1x detected (1x unique)
GematchCrawler
Example user agent
GematchCrawler/2.1 (http://www.gematch.com/crawler.html) 2x detected (2x unique)
LSSRocketCrawler,Lightspeed Systems RocketCrawler
Example user agent
LSSRocketCrawler/1.0 LightspeedSystems 1x detected (1x unique)
MXit WebBot
Example user agent
Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.3.119) Opera Mini/3.1 2x detected (2x unique)
hitcrawler,hitcrawler_0
Example user agent
hitcrawler_0.1 (ringken@gmail.com) 2x detected (1x unique)
ZIBB Crawler
Example user agent
ZIBB Crawler (email address / WWW address) 2x detected (2x unique)
WIRE/0.11 b,WIRE
Example user agent
WIRE/0.11 (Linux; i686; Robot,Spider,Crawler,aromano@cli.di.unipi.it) 2x detected (2x unique)
Mobile Spider1,sogou spider
Example user agent
Sogou Mobile Spider1.0 (http://wap.sogou.com) 2x detected (1x unique)
HyperCrawl
Example user agent
Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html) 2x detected (2x unique)
WebCrawler,JamesBOT
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html 1x detected (1x unique)
Infocrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Infocrawler; Alexa Toolbar) 1x detected (1x unique)
HuaweiSymantecSpider
Example user agent
Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com) 1x detected (1x unique)
WangIDSpider
Example user agent
Mozilla/5.0 (compatible; WangIDSpider/1.0; +http://www.wangid.com/spider.html) 2x detected (1x unique)
USyd-NLP-Spider
Example user agent
USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) 2x detected (1x unique)
IssueCrawler
Example user agent
IssueCrawler 2x detected (2x unique)
WebTarantula,com Crawler
Example user agent
WebTarantula.com Crawler 1x detected (1x unique)
PeerFactor Crawler
Example user agent
PeerFactor Crawler 1x detected (1x unique)
SpeedySpider
Example user agent
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com) 2x detected (1x unique)
sReleaseBot
Example user agent
Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/) 2x detected (1x unique)
n4p_bot
Example user agent
n4p_bot (crawler@n4p.com) 1x detected (1x unique)
ReadPath_Spider
Example user agent
ReadPath_Spider 1x detected (1x unique)
Imperia-LinkSpider
Example user agent
Imperia-LinkSpider,gzip(gfe),gzip(gfe) 1x detected (1x unique)
XSpider
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 7.0) XSpider 7 3x detected (3x unique)
General Crawlers,SmartAndSimpleWebCrawler,Java
Example user agent
SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net),gzip(gfe),gzip(gfe) 1x detected (1x unique)
LookUpCrawler
Example user agent
LookUpCrawler - lookupcanada.ca [ZSEBOT] 1x detected (1x unique)
Overture-WebCrawler
Example user agent
Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 1x detected (1x unique)
MicrosoftPrototypeCrawler
Example user agent
MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com) 1x detected (1x unique)
UOLCrawler
Example user agent
UOLCrawler (soscrawler@uol.com.br) 2x detected (2x unique)
imrbot,Mignify Bot
Example user agent
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com) 1x detected (1x unique)
gonzo1
Example user agent
gonzo1[D] mailto:crawleradmin.t-info@telekom.de 1x detected (1x unique)
HttpSpider
Example user agent
HttpSpider/0.91 2x detected (2x unique)
Internet Archive,UA crawler
Example user agent
Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler) 2x detected (2x unique)
Web-sniffer,Perl
Example user agent
Mozilla/5.0 (Windows; Windows NT 6.1; WOW64; rv:2.0b8pre; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; SLCC1; SLCC2; Media Center PC 6.0; GTB6.4; InfoPath.2; en-US; FunWebProducts; Zango 10.1.181.0; SV1; PRTG Network Monitor (www.paessler.com)) Gecko/20101114 Firefox/4.0b8pre QuickTime/7.6.2 Songbird/1.1.2 Web-sniffer/1.0.36 lftp/3.7.4 libwww-perl/5.820 GSiteCrawler/v1.12 rev. 260 Snoopy v1.2 3x detected (2x unique)
mapion-news-bot,Mapian News Bot
Example user agent
mapion-news-bot/1.0 (http://www.mapion.co.jp/news/) 0x detected (0x unique)
Example user agent
SearQuBot/SearQuBot v1.0 1x detected (1x unique)
PAD-bot
Example user agent
Mozilla/5.0 (compatible; PAD-bot/9.0; +http://www.descargarprogramagratis.com/) 3x detected (1x unique)
MQBOT
Example user agent
MQbot http://metaquerier.cs.uiuc.edu/crawler 1x detected (1x unique)
Symfony Spider
Example user agent
Symfony Spider (http://symfony.com/spider) 2x detected (1x unique)
memorybot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB) 3x detected (2x unique)
discoverybot,discoverybot/2.0
Example user agent
Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html) 1x detected (1x unique)
it2media-domain-crawler
Example user agent
it2media-domain-crawler/1.0 on crawler-prod.it2media.de 1x detected (1x unique)
adspider
Example user agent
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.21 (KHTML, like Gecko) adspider Safari/537.21 2x detected (1x unique)
JobdiggerSpider
Example user agent
Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider) 1x detected (1x unique)
WebCrunch
Example user agent
Mozilla/5.0 (compatible; WebCrunch/1.2; +http://webcrunch.net/crawler) 1x detected (1x unique)
CamontSpider
Example user agent
CamontSpider/1.0 http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html 1x detected (1x unique)
NWSpider
Example user agent
NWSpider 0.9 2x detected (2x unique)
US-Crawler,Links4US-Crawler
Example user agent
Links4US-Crawler, ( http://links4us.com/) 2x detected (1x unique)
LoadTimeBot
Example user agent
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html) 1x detected (1x unique)
audioCrawlerBot
Example user agent
audioCrawlerBot/1.0 (http://www.audiocrawler.com/) 2x detected (1x unique)
Envolk
Example user agent
envolk/1.7 ( http://www.envolk.com/envolkspider.html) 3x detected (3x unique)
Amfibibot/0.07,Amfibibot,Amfibi
Example user agent
Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) 2x detected (2x unique)
Research Projects,CMS crawler
Example user agent
CMS crawler ( http://buytaert.net/crawler/) 2x detected (1x unique)
Socialradarbot
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com) 2x detected (1x unique)
MarketwireBot
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com) 2x detected (1x unique)
XoviBot
Example user agent
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/) 1x detected (1x unique)
Axtaris
Example user agent
Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; http://axtaris.org; crawler at axtaris dot org) 2x detected (2x unique)
mixi-crawler,mixi
Example user agent
mixi-crawler/2.00 (http://mixi.jp/) 2x detected (1x unique)
ShowyouBot
Example user agent
ShowyouBot (http://showyou.com/crawler) 1x detected (1x unique)
ProloCrawler
Example user agent
ProloCrawler/1.0 (http://www.prolo.com) 1x detected (1x unique)
HubSpot Crawler
Example user agent
HubSpot Crawler 1.0 http://www.hubspot.com/ 3x detected (2x unique)
Esribot,Esribot/1.0
Example user agent
Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/) 2x detected (2x unique)
Fake Browser,compatible crawler
Example user agent
http://www.Syntryx.com/ ANT Chassis 9.27; Mozilla/4.0 compatible crawler 1x detected (1x unique)
METASpider
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) METASpider 2x detected (1x unique)
PDFBot
Example user agent
PDFBot (crawler@pdfind.com) 2x detected (1x unique)
LinkpadBot
Example user agent
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru) 1x detected (1x unique)
KIT webcrawler
Example user agent
KIT webcrawler/0.2.4 1x detected (1x unique)
DotSpotsBot
Example user agent
DotSpotsBot/0.2 (crawler; support at dotspots.com) 2x detected (2x unique)
snap.com beta crawler v0,beta crawler
Example user agent
snap.com beta crawler v0 2x detected (1x unique)
GSiteCrawler
Example user agent
GSiteCrawler/v1.12 rev. 260 (http://gsitecrawler.com/) 1x detected (1x unique)
BravoBrian SpiderEngine
Example user agent
BravoBrian SpiderEngine MarcoPolo 1x detected (1x unique)
PsSpider
Example user agent
PsSpider/0.7 (spider@bildkiste.de) 1x detected (1x unique)
BT Crawler
Example user agent
BT Crawler 1.0,gzip(gfe),gzip(gfe),gzip(gfe) 1x detected (1x unique)
Gnam Spider
Example user agent
Gnam Gnam Spider 2x detected (2x unique)
QuickFinder Crawler,QuickFinder
Example user agent
QuickFinder Crawler 2x detected (2x unique)
Finderbots,bixo
Example user agent
Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com) 1x detected (1x unique)
CFG_SPIDER_USER_AGENT
Example user agent
CFG_SPIDER_USER_AGENT 1x detected (1x unique)
Axadine Crawler
Example user agent
axadine/ (Axadine Crawler; http://www.axada.de/; ) 2x detected (1x unique)
EveryoneSocialBot
Example user agent
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/) 2x detected (2x unique)
Jambot,Jambot/0.1.1
Example user agent
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com) 2x detected (1x unique)
BabalooSpider
Example user agent
BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) 1x detected (1x unique)
Llaut
Example user agent
Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) 2x detected (1x unique)
grub
Example user agent
Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org) 3x detected (2x unique)
TestCrawler,LDSpider
Example user agent
ldspider (BTC 2011 crawl, harth@kit.edu, http://code.google.com/p/ldspider/wiki/Robots) 2x detected (1x unique)
Comodo Spider
Example user agent
Comodo Spider 1.1 1x detected (1x unique)
AESOP_com_SpiderMan
Example user agent
AESOP_com_SpiderMan 1x detected (1x unique)
archive_crawler
Example user agent
Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 http://www.archive.org/details/archive_crawler) 2x detected (2x unique)
FAST Enterprise Crawler,Using Fast Enterprise Crawler
Example user agent
RB2B-bot v0.1 (Using Fast Enterprise Crawler 6 search@reedbusiness.com) 2x detected (2x unique)
com crawler,semantic-visions.com crawler
Example user agent
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) 3x detected (2x unique)
ArchiveBot,ArchiveTeam ArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36 2x detected (1x unique)
AjaxSnapBot
Example user agent
Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html) 2x detected (2x unique)
Slackbot,Slackbot-Link-Expanding
Example user agent
Slackbot-LinkExpanding+(+https://api.slack.com/robots) 2x detected (2x unique)
DNSGroup,DNS Group Crawler
Example user agent
DNSGroup/0.1 (DNS Group Crawler; http://www.dnsgroup.com/; crawler@dnsgroup.com) 2x detected (1x unique)
5Bot
Example user agent
Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net) 1x detected (1x unique)
Engage Crawler
Example user agent
Adknowledge Engage Crawler 1x detected (1x unique)
My Spider
Example user agent
My Spider 1.0,gzip(gfe),gzip(gfe) 2x detected (2x unique)
Fake Browser,GetMeLinked Spider
Example user agent
Mozilla 4.0 - GetMeLinked Spider www.GetMeLinked.com Web Directory 2x detected (1x unique)
Steeler
Example user agent
Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) 2x detected (1x unique)
m65bot
Example user agent
DoCoMo/2.0 P904i( m65bot/0.1; c; http://m65.jp/bot.html ) 2x detected (2x unique)
DuckDuckGo-Favicons-Bot,DuckDuck Favicons Bot
Example user agent
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com) 1x detected (1x unique)
mCrawler
Example user agent
Vodafone mCrawler (bergum@fast.no) 2x detected (2x unique)
SpokeSpider/1.0,SpokeSpider
Example user agent
SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really) 1x detected (1x unique)
Automattic Analytics Crawler
Example user agent
Automattic Analytics Crawler/0.2;+http://wordpress.com/crawler/ 3x detected (2x unique)
sproose,Sproose/1.0beta
Example user agent
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com) 2x detected (1x unique)
HRCrawler
Example user agent
HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko) 1x detected (1x unique)
Denodo IECrawler
Example user agent
Denodo IECrawler/4.5,gzip(gfe),gzip(gfe) 5x detected (4x unique)
Y!J-BRJ/YATS,Yahoo! Japan,YATS crawler,Y!J-AGENT
Example user agent
Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) 3x detected (2x unique)
HiddenMarket,HiddenMarket-1.0-beta
Example user agent
HiddenMarket-1.0-beta (www.hiddenmarket.net/crawler.php) 2x detected (1x unique)
NerdyBot
Example user agent
NerdyBot 1x detected (1x unique)
blackspider
Example user agent
blackspider 1x detected (1x unique)
Index crawler
Example user agent
IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org) 2x detected (2x unique)
Research spider,Generic Research Crawler
Example user agent
Research spider - ak1835@albany.edu 1x detected (1x unique)
FAST FreshCrawler
Example user agent
schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; Contact: webcrawl@schibstedsok.no;) 2x detected (2x unique)
gsa-crawler,Google Search Appliance
Example user agent
gsa-crawler 2x detected (2x unique)
woobot,WooRank
Example user agent
woobot/1.1 1x detected (1x unique)
boitho crawler
Example user agent
boitho crawler 1x detected (1x unique)
info Crawler
Example user agent
visaduhoc.info Crawler 3x detected (2x unique)
BlogPulse,ISSpider
Example user agent
BlogPulse (ISSpider-3.0) 2x detected (2x unique)
Crawlboy,Crawlzilla
Example user agent
Crawlzilla/1.0 (Crawlzilla; http://www.crawlzilla.com; crawler@crawlzilla.com) 3x detected (2x unique)
netEstate NE Crawler,NE Crawler
Example user agent
netEstate NE Crawler (+http://www.sengine.info/) 1x detected (1x unique)
Comodo-Webinspector-Crawler
Example user agent
Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.com 1x detected (1x unique)
DoubleVerify Crawler
Example user agent
DoubleVerify Crawler 1x detected (1x unique)
acquia-crawler
Example user agent
acquia-crawler (detected bad behaviour? please tell us at it@acquia.com),gzip(gfe),gzip(gfe) 2x detected (1x unique)
ArchitextSpider
Example user agent
ArchitextSpider 3x detected (2x unique)
Steeler,Steeler/3.5
Example user agent
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) 1x detected (1x unique)
Sosoblogspider
Example user agent
Sosoblogspider+(+http://help.soso.com/soso-blog-spider.htm) 2x detected (1x unique)
lbot
Example user agent
Mozilla/5.0 (lbot/2.6.4) lbot@mailme.not 3x detected (1x unique)
Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 1x detected (1x unique)
unchaos_crawler_2
Example user agent
unchaos_crawler_2.0.2 (search.engine@unchaos.com) 2x detected (1x unique)
Uipbot
Example user agent
uipbot/1.0 (uipbot@semasio.net) 1x detected (1x unique)
Fake Browser
Example user agent
Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html) 2x detected (1x unique)
AntBot
Example user agent
AntBot/1.0 (http://www.ant.com) 4x detected (4x unique)
R6_CommentReader,Radian 6,R6 CommentReader,salesforce radian6
Example user agent
R6_CommentReader(www.radian6.com/crawler) 3x detected (2x unique)
Sogou Orion spider,sogou spider
Example user agent
Sogou Orion spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07) 2x detected (2x unique)
SocialSpider,SocialSpider-Finder
Example user agent
SocialSpider-Finder/0.2 1x detected (1x unique)
ptd-crawler
Example user agent
Mozilla/5.0 (compatible; ptd-crawler; http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com) 1x detected (1x unique)
Fujiko Spider
Example user agent
Fujiko Spider (fujiko@inelegant.org) 3x detected (1x unique)
Google Wireless Transcoder
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14 3x detected (3x unique)
Ex-Crawler,Java,General Crawlers
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/v0.1.5ALPHA SVN rev58; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe) 2x detected (1x unique)
MixBot
Example user agent
MixBot (+http://t.co/GSRLLKex24) 1x detected (1x unique)
BlogRangerCrawler
Example user agent
BlogRangerCrawler/1.0 3x detected (3x unique)
Ex-Crawler,Java,General Crawlers
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/0.1.5a; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe) 2x detected (1x unique)
mozDex
Example user agent
mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com) 3x detected (2x unique)
boitho.com-dc,Boitho
Example user agent
boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html ) 4x detected (3x unique)
Boitho,boitho.com-dc,boitho.com-dc/0.86
Example user agent
boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html ) 1x detected (1x unique)
GoGuidesBot
Example user agent
Mozilla/5.0 (compatible; GoGuidesBot; http://www.goguides.org/spider.html) 1x detected (1x unique)
Jomjaibot
Example user agent
Jomjaibot/1.0 Crawl (+http://www.jomjaibot.com/) 3x detected (2x unique)
Semantic Discovery,semanticdiscovery
Example user agent
semanticdiscovery/0.2(http://www.semanticdiscovery.com/sd/robot.html) 2x detected (2x unique)
AppCodes crawler,TestCrawler
Example user agent
AppCodes crawler - looking for iOS app mentions. More info: support@appcodes.com. Robots.txt id: AppCodesCrawler 1x detected (1x unique)
Logict IPv6 Crawler
Example user agent
Logict IPv6 Crawler/1.0 (http://ipv6search.logict.net),gzip(gfe),gzip(gfe) 3x detected (1x unique)
Scrubby
Example user agent
Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html) 1x detected (1x unique)
fastlwspider
Example user agent
fastlwspider/1.0 1x detected (1x unique)
PhoenixWebBot
Example user agent
PhoenixWebBot Beta 2x detected (1x unique)
ca-crawler
Example user agent
Mozilla/5.0 (compatible; ca-crawler/1.0) 1x detected (1x unique)
449 Overture-WebCrawler
Example user agent
449 Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp 2x detected (2x unique)
NimbleCrawler/2.0.1,NimbleCrawler
Example user agent
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com 2x detected (1x unique)
ObjectsSearch
Example user agent
ObjectsSearch/0.2 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) 2x detected (2x unique)
PageFreezer,pagefreezer crawler
Example user agent
PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com) 2x detected (1x unique)
BOTW Spider
Example user agent
Mozilla/4.0 (compatible; BOTW Spider; http://botw.org) 1x detected (1x unique)
MedSpider
Example user agent
MedSpider v0.0.1 1x detected (1x unique)
Rakutenbot
Example user agent
Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html) 1x detected (1x unique)
kizasi-spider
Example user agent
kizasi-spider/1.0 (+http://kizasi.jp/) 1x detected (1x unique)
BeijingCrawler
Example user agent
BeijingCrawler 2x detected (1x unique)
SandCrawler
Example user agent
SandCrawler - Compatibility Testing 1x detected (1x unique)
IWE Spider
Example user agent
IWE Spider v.01 - www.pavka.com.au 2x detected (2x unique)
PagesInventory,PagesInventory Bot
Example user agent
PagesInventory (robot http://www.pagesinvenotry.com) 4x detected (3x unique)
PowerMapper,Crawlboy,CrawlerProcess
Example user agent
CrawlerProcess (http://www.PowerMapper.com) /5.5.678.0 1x detected (1x unique)
BejiBot
Example user agent
BejiBot Crawler (BNL Services; http://www.bejjan.net/crawler/) 2x detected (2x unique)
woobot,WooRank
Example user agent
woobot/2.0 1x detected (1x unique)
BuildCMS crawler
Example user agent
BuildCMS crawler (http://www.buildcms.com/crawler) 1x detected (1x unique)
X-Crawler
Example user agent
X-Crawler 2x detected (2x unique)
Nutch,GrowerIdeas Crawler
Example user agent
GrowerIdeas Crawler/GrowerIdeas-nutch-1.6 (Crawls URLs for indexing content for our new search startup which aims to provide simple and smart search across curated content. For more info please contact helloworld@growerideas.com. If you think 2x detected (2x unique)
DealGates Bot/1.1,DealGates Bot
Example user agent
DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html) 2x detected (1x unique)
WebAlta Crawler
Example user agent
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) 2x detected (1x unique)
ThumbShotsBot
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010091808 ThumbShotsBot (KFSW 3.0.6-3) 1x detected (1x unique)
AstraSpider
Example user agent
Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) 1x detected (1x unique)
Yahoo-VerticalCrawler-FormerWebCrawler
Example user agent
Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler 1x detected (1x unique)
Yahoo! Japan
Example user agent
Mozilla/5.0 (Linux; U; Android 2.3.3; ja-jp; sdk Build/GRI34; Y!J-BRZ/YATSHA crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 1x detected (1x unique)
dtSearchSpider
Example user agent
dtSearchSpider 1x detected (1x unique)
acedo crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0; acedo crawler extension) 2x detected (1x unique)
Patwebbot
Example user agent
Patwebbot (http://www.herz-power.de/technik.html) 1x detected (1x unique)
CrawlBot
Example user agent
Mozilla/5.0 (compatible; CrawlBot/1.0.0) 1x detected (1x unique)
rating crawler
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) Russian CMS rating crawler (itrack.ru/cmsrate, avlasov@itrack.ru) 2x detected (1x unique)
LOOQ
Example user agent
LOOQ/0.1 alfa (LOOQ Crawler for european sites; http://looq.eu; root (at) looq dot eu) 2x detected (2x unique)
grapeFX/0.9,grapeFX
Example user agent
Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk 2x detected (2x unique)
webceo Bot,online-webceo-bot
Example user agent
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com) 1x detected (1x unique)
Spider_Monkey
Example user agent
Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml) 1x detected (1x unique)
ByWebSite-Search-Spider
Example user agent
ByWebSite-Search-Spider,gzip(gfe),gzip(gfe) 1x detected (1x unique)
wminer
Example user agent
wminer/1.2 (North America Web Crawler; http://www.wminer.com/bot; info at wminer dot com) 1x detected (1x unique)
bendercrawler
Example user agent
Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler) 1x detected (1x unique)
haupia-crawler
Example user agent
haupia-crawler 1x detected (1x unique)
WSDL Crawler
Example user agent
WSDL Crawler 1x detected (1x unique)
semantics webbot
Example user agent
semantics webbot/1.0 1x detected (1x unique)
WASALive Bot
Example user agent
Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/) 1x detected (1x unique)
AcquiaCrawler
Example user agent
AcquiaCrawler,gzip(gfe),gzip(gfe) 4x detected (3x unique)
Accoona-AI-Agent,Accoona Bot,Accoona-AI-Agent/1.1.1
Example user agent
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) 4x detected (3x unique)
Yahoo-MMCrawler,Yahoo! Image,Yahoo-MMCrawler/4.0
Example user agent
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com) 1x detected (1x unique)
altavista
Example user agent
AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net 5x detected (2x unique)
FAST-WebCrawler,FAST Web Crawler
Example user agent
FAST-WebCrawler/2.2.11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 2x detected (1x unique)
WIRE
Example user agent
WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler) 2x detected (2x unique)
EventGuruBot/1.0,EventGuruBot
Example user agent
Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html) 3x detected (1x unique)
BitlyBot
Example user agent
bitlybot 2x detected (2x unique)
HubSpot Webcrawler,Webcrawler
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler 3x detected (1x unique)
FTRF: Friendly Robot
Example user agent
FTRF: Friendly robot/1.3 1x detected (1x unique)
None-Crawler
Example user agent
Mozilla/5.0 (compatible; .None-Crawler/0.1 +http://domains.ericbinek.None/) 1x detected (1x unique)
ScSpider
Example user agent
ScSpider/0.2 2x detected (2x unique)
Fake Browser,BotSeer
Example user agent
Mozilla 4.0(compatible; BotSeer/1.0; http://botseer.ist.psu.edu) 2x detected (1x unique)
DomainCrawler
Example user agent
DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/bot 2x detected (2x unique)
iaskspider2,iaskspider
Example user agent
iaskspider2 (iask@staff.sina.com.cn) 2x detected (1x unique)
MonoBot
Example user agent
Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name) 1x detected (1x unique)
OMGCrawler
Example user agent
OMGCrawler 1.0 2x detected (1x unique)
PluckItCrawler
Example user agent
PluckItCrawler/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com;) 1x detected (1x unique)
tCrawler
Example user agent
tCrawler/0.1,gzip(gfe),gzip(gfe),gzip(gfe) 3x detected (2x unique)
Sogou Head Spider,sogou spider
Example user agent
Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) 1x detected (1x unique)
TAMU_CS_IRL_CRAWLER
Example user agent
TAMU_CS_IRL_CRAWLER/1.0 4x detected (2x unique)
Grub,grub-client
Example user agent
grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org) 1x detected (1x unique)
Synthesio Crawler
Example user agent
Synthesio Crawler release MonaLisa (contact at synthesio dot fr) 2x detected (2x unique)
ownCloud Server Crawler,Server Crawler
Example user agent
ownCloud Server Crawler 2x detected (2x unique)
Generic Research Crawler,Runet-Research-Crawler
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru) 3x detected (2x unique)
Blogram,BlogramCrawler
Example user agent
BlogramCrawler/1.0.1(+http://blogram.jp/) 3x detected (2x unique)
envolk/1.7,Envolk
Example user agent
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) 2x detected (1x unique)
diffbot
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15 3x detected (2x unique)
Semantic Discovery,semanticdiscovery
Example user agent
semanticdiscovery/2.0(http://www.semanticdiscovery.com/robot.html) 2x detected (1x unique)
Slackbot
Example user agent
Slackbot (+https://api.slack.com/robots) 1x detected (1x unique)
FunkyCrawler
Example user agent
Mozilla/5.0 (compatible; FunkyCrawler; ) 2x detected (1x unique)
Pete-Spider
Example user agent
Pete-Spider/1.1 2x detected (1x unique)
Finderlein Research Crawler
Example user agent
Finderlein Research Crawler 1.0 2x detected (1x unique)
IconSurf
Example user agent
IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) 2x detected (1x unique)
noxtrumbot
Example user agent
noxtrumbot/1.0 (crawler@noxtrum.com) 2x detected (2x unique)
bl.uk_lddc_bot,uk_lddc_bot
Example user agent
bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html) 2x detected (1x unique)
Crawler
Example user agent
Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/) 2x detected (2x unique)
NetWhatCrawler,NetWhatCrawler/0.06-dev
Example user agent
NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com) 1x detected (1x unique)
Host-Spy Crawler
Example user agent
Mozilla/5.0 (compatible; Host-Spy Crawler/1.2; +http://www.host-spy.com/) 2x detected (1x unique)
SoftListBot
Example user agent
Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/) 2x detected (1x unique)
ScollSpider
Example user agent
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com) 3x detected (1x unique)
YioopBot
Example user agent
Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php) 2x detected (2x unique)
GroupHigh Bot,com/bot
Example user agent
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html) 2x detected (2x unique)
RedBot1.0,redbot
Example user agent
RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) 1x detected (1x unique)
WordSurfer Spider
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; WordSurfer Spider 2.2;)) 1x detected (1x unique)
TayaCrawler
Example user agent
TayaCrawler (Beta; v0.1; tayadev@nexdegree.com) 1x detected (1x unique)
Webspider
Example user agent
Webspider/1.0 (web spider; ) 1x detected (1x unique)
SMXCrawler
Example user agent
SMXCrawler/1.0 (www.socialmetrix.com) 5x detected (4x unique)
Yahoo! Japan,YATS crawler,Y!J-BRJ/YATS,Y!J-AGENT
Example user agent
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html) 2x detected (2x unique)
iaskspider,iAsk Spider
Example user agent
Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0) 1x detected (1x unique)
na-Webcrawler
Example user agent
na-Webcrawler (helpdesk@newsaktuell.de) 2x detected (2x unique)
bl.uk_lddc_bot,uk_lddc_bot
Example user agent
Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html) 2x detected (2x unique)
Flatland,flatlandbot
Example user agent
flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com) 5x detected (2x unique)
FAST Web Crawler,FAST-WebCrawler
Example user agent
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 1x detected (1x unique)
YebolBot
Example user agent
YebolBot (Email: yebolbot@gmail.com; If the web crawling affects your web service, or you don't like to be crawled by us, please email us. We'll stop crawling immediately.) 1x detected (1x unique)
CowCrawler
Example user agent
CowCrawler/CowCrawler-dev (+http://beta.cow.com),gzip(gfe),gzip(gfe) 1x detected (1x unique)
Sitespider
Example user agent
Mozilla/5.5 (compatible; MSIE 5.5; Windows NT 5.1) Sitespider+ b432.1 1x detected (1x unique)
Trends Crawler
Example user agent
Trends Crawler, Real time trends bot (info@trendscrawler.com) 1x detected (1x unique)
RSScrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; RSScrawler/4.0 (compatible; MSIE 6.0; Windows NT 5.0); .NET CLR 2.0.50727 1x detected (1x unique)
HuaweiSymantecSpider
Example user agent
Huaweisymantecspider (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727),gzip(gfe) 2x detected (2x unique)
socialbm_bot,socialbm_bot/1.0
Example user agent
Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net) 1x detected (1x unique)
RSSIncludeBot
Example user agent
RSSIncludeBot/1.0 (http://www.rssinclude.com/spider) 2x detected (1x unique)
hivaBot
Example user agent
hivaBot(http://gorgor.ir and info@gorgor.ir)/hivaBot-0.2 2x detected (1x unique)
BuiBui-Bot
Example user agent
BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com) 2x detected (1x unique)
hivaBot
Example user agent
hivaBot/hivaBot-0.2 (info@gorgor.ir) 2x detected (2x unique)
focuseekbot,seekbot
Example user agent
Mozilla/4.0 (compatible; focuseekbot) 2x detected (2x unique)
EnaBot,EnaBot/1.2
Example user agent
EnaBot/1.2 (http://www.enaball.com/crawler.html) 1x detected (1x unique)
Sangfor Spider
Example user agent
Sangfor Spider/0.7 4x detected (3x unique)
Yahoo! Japan,Y!J-AGENT,Y!J-BSC/1.0
Example user agent
Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/) 2x detected (2x unique)
fastbot crawler,fastbot
Example user agent
fastbot crawler beta 4.0 (+http://www.fastbot.de) 1x detected (1x unique)
TTop-Crawler
Example user agent
TTop-Crawler,gzip(gfe),gzip(gfe) 2x detected (2x unique)
SetLinks bot,SetLinks.ru Crawler
Example user agent
Mozilla/5.0 (compatible; SetLinks bot 1.2) 2x detected (1x unique)
JobRoboter
Example user agent
JobRoboter/1.1 1x detected (1x unique)
LNSpiderguy
Example user agent
LNSpiderguy 2x detected (2x unique)
Crawlboy,CrawlFire
Example user agent
CrawlFire - you can disable this Robot: http://pastebin.de/25277 3x detected (1x unique)
UnisterBot
Example user agent
Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de) 1x detected (1x unique)
EtaoSpider
Example user agent
Mozilla/5.0 (compatible; EtaoSpider/1.0; http://open.etao.com/dev/EtaoSpider) 2x detected (1x unique)
DBot
Example user agent
Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com) 1x detected (1x unique)
SeekOn Spider
Example user agent
SeekOn Spider 1.9(+http://www.seekon.com/spider.html) 2x detected (2x unique)
Nett.io bot,io bot
Example user agent
Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/) 1x detected (1x unique)
OpenX Spider
Example user agent
Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org) 2x detected (2x unique)
WEBB Crawler,TestCrawler
Example user agent
WEBB Crawler - see: http://badcheese.com/robots.html 2x detected (1x unique)
WNMbot
Example user agent
WNMbot/0.4.0 1x detected (1x unique)
Fluffy the spider
Example user agent
Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com) 2x detected (2x unique)
Cityreview Robot,cityreview
Example user agent
Cityreview Robot (+http://www.cityreview.org/crawler/) 2x detected (2x unique)
Python,pythonic-crawler
Example user agent
pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp) 5x detected (1x unique)
FAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/5.3.4 (crawler@fast.no) 1x detected (1x unique)
Microsoft MSN SocialStreams Bot
Example user agent
Microsoft MSN SocialStreams Bot,gzip(gfe),gzip(gfe) 2x detected (2x unique)
CyberPatrol,SiteCat Webbot
Example user agent
CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) 5x detected (4x unique)
Y!J-BRW/1.0,Yahoo! Japan,Y!J-BRW,Y!J-AGENT
Example user agent
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) 1x detected (1x unique)
jikespider
Example user agent
jikespider "Mozilla/5.0 3x detected (1x unique)
Crawler4j
Example user agent
crawler4j (http://code.google.com/p/crawler4j/) 1x detected (1x unique)
Sangfor Spider
Example user agent
Sangfor Spider 2x detected (1x unique)
Microsoft Bing Mobile SocialStreams Bot
Example user agent
Microsoft Bing Mobile SocialStreams Bot 1x detected (1x unique)
URLfilterDB-crawler
Example user agent
Mozilla/5.0 (compatible; Linux i686; en-US; URLfilterDB-crawler/1.0) ufdb/1.0 2x detected (2x unique)
robot,Werbefreie Deutsche Suchmaschine
Example user agent
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html) 1x detected (1x unique)
Lite Bot
Example user agent
Lite Bot0316B 3x detected (1x unique)
woriobot
Example user agent
Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com) 1x detected (1x unique)
Kyoto-Crawler
Example user agent
Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/) 1x detected (1x unique)
nyobot
Example user agent
nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com),gzip(gfe),gzip(gfe) 1x detected (1x unique)
tivraSpider
Example user agent
tivraSpider/1.0 (crawler@tivra.com) 1x detected (1x unique)
PicSpider
Example user agent
PicSpider/1.1 (spider@217-20-118-26.internetserviceteam.com; http://www.bildkiste.de) 1x detected (1x unique)
jrCrawler
Example user agent
jrCrawler/1.0b 2x detected (1x unique)
JoobleBot
Example user agent
JoobleBot (compatible; http://jooble.org/jooble-bot) 2x detected (1x unique)
MSR-ISRCCrawler
Example user agent
MSR-ISRCCrawler 1x detected (1x unique)
SiteCrawler
Example user agent
SiteCrawler,gzip(gfe),gzip(gfe) 3x detected (2x unique)
Shim-Crawler,Shim Crawler
Example user agent
Shim-Crawler 1x detected (1x unique)
HiveCrawler
Example user agent
Mozilla/5.0 (compatible; HiveCrawler/1.2 http://www.businessinsider.com) 3x detected (1x unique)
MetaJobBot
Example user agent
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler) 1x detected (1x unique)
BrightCrawler
Example user agent
BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) 3x detected (2x unique)
nuSearch Spider,nuSearch
Example user agent
nuSearch Spider www.nusearch.com (compatible; MSIE 4.01; Windows NT) 1x detected (1x unique)
cis455crawler
Example user agent
cis455crawler 1x detected (1x unique)
Asterias Crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.0 1x detected (1x unique)
PipeLine Spider
Example user agent
pipeLiner/0.10 (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) 2x detected (1x unique)
WebAlta Crawler
Example user agent
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) 2x detected (1x unique)
IstellaBot
Example user agent
Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/) 2x detected (1x unique)
memoryBot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB) 2x detected (2x unique)
Accoona Bot,Accoona-Biz-Agent
Example user agent
Accoona-Biz-Agent/1.1.1 crawler@accoona.com 2x detected (1x unique)
HooWWWer
Example user agent
HooWWWer/2.1.0 ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi) 1x detected (1x unique)
media-percbotspider
Example user agent
media-percbotspider 1x detected (1x unique)
Liquida Spider
Example user agent
Liquida Spider 1.0 +http://liquida.com/ 2x detected (2x unique)
Jaxified Crawler,Jaxified
Example user agent
Jaxified Crawler 1.0a (+http://www.jaxified.com/),gzip(gfe),gzip(gfe) 2x detected (2x unique)
Wayback Archive Bot,spider
Example user agent
Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org) 1x detected (1x unique)
ZooShot
Example user agent
ZooShot 0.42 2x detected (2x unique)
TestCrawler,robots
Example user agent
robots 2x detected (1x unique)
Linguee Bot
Example user agent
Linguee Bot (bot@linguee.com) 1x detected (1x unique)
LargeSmall Crawler
Example user agent
LargeSmall Crawler 3x detected (2x unique)
suggybot,suggybot/0.01a
Example user agent
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/) 3x detected (2x unique)
200PleaseBot,200PleaseBot/1.0
Example user agent
Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot) 1x detected (1x unique)
agbot
Example user agent
agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com) 1x detected (1x unique)
Feed Crawler
Example user agent
www2.apserver.net ASP-Ranker Feed Crawler 2x detected (1x unique)
lbot
Example user agent
Mozilla/5.0 (lbot/2.6.4) (lbot@mailme.not) 2x detected (2x unique)
Webbot.ru,Webbot
Example user agent
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html) 4x detected (3x unique)
Radian 6,salesforce radian6,R6 FeedFetcher
Example user agent
R6_FeedFetcher_(www.radian6.com/crawler) 3x detected (2x unique)
MetaWeb Crawler,FAST MetaWeb Crawler
Example user agent
FAST MetaWeb Crawler (helpdesk at fastsearch dot com) 2x detected (1x unique)
LinkStats Bot
Example user agent
LinkStats Bot 1x detected (1x unique)
Web-Robot
Example user agent
Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3 1x detected (1x unique)
spiderpig
Example user agent
spiderpig/0.1,gzip(gfe),gzip(gfe) 2x detected (1x unique)
ayna-crawler
Example user agent
Mozilla/5.0 (compatible; ayna-crawler http://www.ayna.com) 1x detected (1x unique)
Top10Ranking Spider
Example user agent
Top10Ranking Spider/3.1 ( http://www.top10Ranking.nl/, Top10ranking.nl heeft op een aantal woorden uw posities in Google gecheckt) 1x detected (1x unique)
SlugBug Spider
Example user agent
SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com) 4x detected (3x unique)
CazoodleBot,CazoodleBot b,MQBOT
Example user agent
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com) 1x detected (1x unique)
Bot.AraTurka.com
Example user agent
Bot.AraTurka.com/0.0.1 1x detected (1x unique)
Inspyder-Crawler
Example user agent
Mozilla/4.0 (compatible; Inspyder-Crawler; http://www.inspyder.com) 1x detected (1x unique)
lb-spider
Example user agent
lb-spider 1x detected (1x unique)
HPI FeedCrawler
Example user agent
HPI FeedCrawler/0.1 (+http://www.hpi.uni-potsdam.de/meinel/bross/feedcrawler) 3x detected (1x unique)
FindLinks
Example user agent
findlinks/2.6 ( http://wortschatz.uni-leipzig.de/findlinks/) 1x detected (1x unique)
Brekiri crawler
Example user agent
Brekiri crawler/1.0 2x detected (2x unique)
Jaxified,RSS Crawler
Example user agent
Jaxified Public RSS Crawler ( http://www.jaxified.com/) 2x detected (2x unique)
Ilse,INGRID
Example user agent
INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider) 1x detected (1x unique)
EcoGrader Crawler
Example user agent
EcoGrader Crawler: Beta 5x detected (3x unique)
Yahoo! Japan,Y!J-AGENT,YFSJ crawler
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248) 1x detected (1x unique)
PArchiveCrawler
Example user agent
PArchiveCrawler/1.0 1x detected (1x unique)
dCrawlBot
Example user agent
dCrawlBot/1.0.1120 2x detected (2x unique)
FAST Web Crawler,FAST-WebCrawler
Example user agent
Opera/30.0.1835.59 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/29.16 AppleWebKit/599.0+ (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 FAST-WebCrawler/3.8 1x detected (1x unique)
Searchspider
Example user agent
Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com) 2x detected (1x unique)
masidani_bot
Example user agent
masidani_bot_v0.3 (info@masidani.com) 2x detected (2x unique)
IBM Connections,IC OpenGraph Crawler
Example user agent
IC OpenGraph Crawler 4.5 (proprietary) 1x detected (1x unique)
Thesis_php_crawler
Example user agent
Thesis_php_crawler 2x detected (1x unique)
Vagabondo
Example user agent
Mozilla/5.0 (compatible; Vagabondo/Wapspider; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 1x detected (1x unique)
Medialab Spider
Example user agent
Mozilla/4.0 (compatible); MSIE 5.0; Medialab Spider 2x detected (2x unique)
Slackbot-Link-Expanding,Slackbot
Example user agent
Slackbot-LinkExpanding (+https://api.slack.com/robots) 2x detected (1x unique)
AOLbot
Example user agent
Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/) 1x detected (1x unique)
BarraHomeCrawler
Example user agent
BarraHomeCrawler (albertof@barrahome.org) 1x detected (1x unique)
favicon_archive
Example user agent
FaviconArchiver/1.0 (+http://moblur.org/workshop/favicon_archive/) 4x detected (2x unique)
YodaoBot,YodaoBot/1.0
Example user agent
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) 1x detected (1x unique)
Apexoo Spider
Example user agent
Apexoo Spider (http://www.apexoo.com/spider/) 0x detected (0x unique)
Example user agent
Mozilla/5.0 (Linux; U; Android 2.2; en-us; LG Vortex Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 1x detected (1x unique)
es-spider
Example user agent
Enterprise_Search/1.0 110 (http://www.innerprise.net/es-spider.asp) 1x detected (1x unique)
CSimpleSpider
Example user agent
CSimpleSpider Robot 3x detected (2x unique)
yoozBot,OOZBOT
Example user agent
Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir) 2x detected (1x unique)
nbot
Example user agent
Mozilla/5.0 (compatible; nbot/2.0) 3x detected (2x unique)
BabalooSpider,BabalooSpider/1.3
Example user agent
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) 2x detected (2x unique)
007AC9 Crawler,Crawler
Example user agent
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/) 2x detected (2x unique)
SNK Screenshot Bot,Save n Keep Screenshot Bot
Example user agent
SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com ) 3x detected (2x unique)
Mapian News Bot,mapion-news-bot
Example user agent
"mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)" 1x detected (1x unique)
Feed Seeker Bot
Example user agent
Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php) 3x detected (2x unique)
oBot/2.3.1,oBot
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/) 2x detected (1x unique)
Comodo Spider
Example user agent
Comodo Spider 1.2 1x detected (1x unique)
Kurzor
Example user agent
Kurzor/1.0 (Kurzor; http://adcenter.hu/docs/en/bot.html; cursor@easymail.hu) 2x detected (1x unique)
HuaweiSymantecSpider
Example user agent
HuaweiSymantecSpider/1.0 DSE-support@huaweisymantec.com (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider) 2x detected (2x unique)
Lipperhey SEO Service,Lipperhey Spider
Example user agent
Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/) 1x detected (1x unique)
SearchSpider
Example user agent
SearchSpider/1.2.10 1x detected (1x unique)
Noobot
Example user agent
noobot Spider/Noobot-1.2 (noobot-bot; http://www.noobot.fr) 2x detected (1x unique)
Kraken
Example user agent
Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net) 3x detected (2x unique)
StatusCake Error Detector,Detector Crawler
Example user agent
StatusCake Error Detector Crawler V0.01 2x detected (1x unique)
seebot
Example user agent
seebot/2.0 (+http://www.seegnify.com/bot) 3x detected (2x unique)
GrapeshotCrawler,GrapeshotCrawler/2.0
Example user agent
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) 1x detected (1x unique)
BlinkaCrawler
Example user agent
Mozilla/5.0 (compatible; BlinkaCrawler/1.0; http://www.blinka.jp/crawler/) 1x detected (1x unique)
Hailoobot
Example user agent
Mozilla/5.0 (compatible; Hailoobot/1.2; http://www.hailoo.com/spider.html) 2x detected (2x unique)
CloudServerMarketSpider,CloudServerMarketSpider/1.0
Example user agent
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html) 3x detected (3x unique)
WI Job Roboter,Job Roboter Spider 3,Job Roboter
Example user agent
Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at) 2x detected (2x unique)
URLfilterDB Crawler,URLfilterDB-crawler
Example user agent
Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0 1x detected (1x unique)
MPICrawler
Example user agent
MPICrawler/0.1 +http://kermit.news.cs.nyu.edu/crawler.html,gzip(gfe),gzip(gfe) 5x detected (3x unique)
80legs/0.83 b,008,80Legs
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620 1x detected (1x unique)
EverbeeCrawler
Example user agent
EverbeeCrawler 3x detected (2x unique)
Yahoo! Japan,Y!J-AGENT
Example user agent
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/) 2x detected (2x unique)
YowedoBot/1.0,Yowedo
Example user agent
YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com) 2x detected (2x unique)
EuripBot,Europe Internet Portal
Example user agent
EuripBot/0.5 (+http://www.eurip.com) PreCheck 1x detected (1x unique)
ExactSeekCrawler
Example user agent
ExactSeekCrawler/1.0 1x detected (1x unique)
8bo Crawler Bot
Example user agent
8bo Crawler Bot 2x detected (1x unique)
oBot
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/) 1x detected (1x unique)
SuperSpider
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; SuperSpider/139; Windows 98; Win 9x 4.90) 1x detected (1x unique)
BIXOCRAWLER
Example user agent
Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com) 2x detected (1x unique)
LinkAider
Example user agent
LinkAider (http://linkaider.com/crawler/) 1x detected (1x unique)
EDI
Example user agent
EDI/1.6.0 (Edacious & Intelligent Web Crawler) 3x detected (2x unique)
ht://Dig,htdig
Example user agent
htdig/3.1.6 (romieu@bastide-medical.fr) 2x detected (1x unique)
OmniExplorer_Bot
Example user agent
OmniExplorer_Bot/6.70 ( http://www.omni-explorer.com) WorldIndexer 3x detected (2x unique)
CazoodleBot-0.1,CazoodleBot
Example user agent
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) 2x detected (1x unique)
ConveraMultiMediaCrawler
Example user agent
ConveraMultiMediaCrawler/0.1 ( http://www.authoritativeweb.com/crawl) 2x detected (1x unique)
Crazywebcrawler
Example user agent
CRAZYWEBCRAWLER 0.9.0, http://www.crazywebcrawler.com 0x detected (0x unique)
Example user agent
Mozilla/5.0 (Linux; U; Android 4.2.1; en-us; MyPhone Agua Vortex Build/JOP40D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30 1x detected (1x unique)
BurstFindCrawler
Example user agent
BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com) 2x detected (2x unique)
Xerka,Xerka WebBot
Example user agent
Xerka WebBot v1.0.0 [AIDO_CREA_ab] 2x detected (1x unique)
zspider
Example user agent
Zspider (+http://www.zhanzhangsou.com/index.htm) 3x detected (2x unique)
SEOkicks Robot,SEOkicks-Robot
Example user agent
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html) 1x detected (1x unique)
deepcrawler
Example user agent
deepcrawler/3.1 (http://www.queusearch.com/whatis_deepcrawler.php),gzip(gfe),gzip(gfe) 2x detected (1x unique)
CMS Crawler
Example user agent
Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com) 3x detected (2x unique)
voyager/2.0,voyager
Example user agent
voyager/2.0 (http://www.kosmix.com/crawler.html) 1x detected (1x unique)
GeekTools Crawler
Example user agent
GeekTools Crawler - http://domains.geek-tools.org 1x detected (1x unique)
Media Crawler
Example user agent
Social Media Crawler using your Home URL on Twitter,Facebook,Myspace,Linkedin by ProfileCapture - contact profilecapture@gmail.com to report any problems with my crawling. http://profilecapture.com 2x detected (1x unique)
MerchantCentricBot
Example user agent
Mozilla/5.0 (compatible; MerchantCentricBot/1.0) 1x detected (1x unique)
persomm-spider
Example user agent
persomm-spider/v1.0 1x detected (1x unique)
Wocodi Web Crawler
Example user agent
WocBot/Mozilla/5.0 (Wocodi Web Crawler 1.0; http://www.wocodi.com/crawler; crawler@wocodi.com) 3x detected (2x unique)
NE Crawler,netEstate NE Crawler
Example user agent
netEstate NE Crawler (+http://www.website-datenbank.de/) 1x detected (1x unique)
mSEARCH Crawler
Example user agent
FAST-mSEARCH Crawler 0.1 (bergum@fast.no) 1x detected (1x unique)
SearchSpider
Example user agent
SearchSpider.com/1.1 2x detected (1x unique)
Spiderlytics
Example user agent
Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com) 1x detected (1x unique)
Comodo HTTP(S) Crawler
Example user agent
Comodo HTTP(S) Crawler - http://www.instantssl.com/crawler 2x detected (1x unique)
Orbiter
Example user agent
Orbiter ( http://www.dailyorbit.com/bot.htm) 2x detected (1x unique)
LinksCrawler
Example user agent
LinksCrawler 0.1beta 1x detected (1x unique)
lemurwebcrawler
Example user agent
Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/) 2x detected (2x unique)
com WebBot,EmeraldShield.com WebBot
Example user agent
EmeraldShield.com WebBot 2x detected (1x unique)
Jayde Crawler
Example user agent
Jayde Crawler. http://www.jayde.com 1x detected (1x unique)
Student Spider
Example user agent
PKU Student Spider 2x detected (2x unique)
INGRID,Ilse
Example user agent
INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) 2x detected (1x unique)
Socialradarbot
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com) 3x detected (2x unique)
Peew,Peew/1.0
Example user agent
Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/) 1x detected (1x unique)
Mozilla-Firefox-Spider
Example user agent
Mozilla-Firefox-Spider(Wenanry) 2x detected (1x unique)
GIDBot
Example user agent
GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php) 2x detected (1x unique)
Theme Spider
Example user agent
Theme Spider ( http://www.themespider.com/spider.html) 1x detected (1x unique)
PathDefender
Example user agent
Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler) 3x detected (2x unique)
Zao-Crawler,Zao
Example user agent
Zao-Crawler 1x detected (1x unique)
Twitmunin Crawler
Example user agent
Twitmunin Crawler http://www.twitmunin.com 1x detected (1x unique)
Tasapspider
Example user agent
Tasapspider/0.9 2x detected (2x unique)
Crawlboy,CrawlerBoy
Example user agent
CrawlerBoy Pinpoint.com 1x detected (1x unique)
SimpleCrawler
Example user agent
SimpleCrawler/0.1 2x detected (1x unique)
memoryBot
Example user agent
Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/) 2x detected (1x unique)
ContextAd Bot
Example user agent
ContextAd Bot 1.0 2x detected (1x unique)
redbot
Example user agent
RED/1 (http://redbot.org/) 1x detected (1x unique)
WebCrawler
Example user agent
WebCrawler v1.3 2x detected (1x unique)
DomainSigmaCrawler
Example user agent
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot) 3x detected (2x unique)
Sogou-Test-Spider,Sogou-Test-Spider/4.0
Example user agent
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98) 2x detected (1x unique)
Norbert the Spider
Example user agent
Norbert the Spider(Burf.com) 2x detected (1x unique)
GarlikCrawler
Example user agent
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com) 1x detected (1x unique)
SAS Crawler
Example user agent
Teragram/SAS Crawler 1x detected (1x unique)
UCMore Crawler
Example user agent
UCMore Crawler App 2x detected (1x unique)
ScollSpider
Example user agent
ScollSpider/2.0 ( http://www.webwobot.com/ScollSpider.php) 2x detected (2x unique)
DomainScan Server Monitoring,robot
Example user agent
Mozilla/5.0 (compatible; DomainSCAN/2.0; +http://domainscan.info/robot/) 1x detected (1x unique)
TECOMAC-Crawler
Example user agent
TECOMAC-Crawler/0.3 1x detected (1x unique)
1470.net crawler
Example user agent
1470.net crawler 2x detected (1x unique)
AISearchBot
Example user agent
AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.) 3x detected (1x unique)
PiplBot
Example user agent
Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/) 1x detected (1x unique)
Superarama.com - BOT
Example user agent
Superarama.com - BOT/v.0.1 2x detected (2x unique)
mixi-news-crawler,mixi
Example user agent
mixi-news-crawler/1.00 (http://mixi.jp/) 1x detected (1x unique)
Punk Spider
Example user agent
Punk Spider/PunkSPIDER-v0.1 2x detected (2x unique)
ru Screenshot Bot,Screenshot Bot
Example user agent
pr-cy.ru Screenshot Bot 2x detected (1x unique)
DBot
Example user agent
Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws) 2x detected (2x unique)
MetamojiCrawler/1.0,MetamojiCrawler
Example user agent
Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html 3x detected (2x unique)
Peeplo Screenshot Bot,Peeplo Screenshot Bot/0.20
Example user agent
Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com ) 2x detected (2x unique)
VSE Link Tester,VSE
Example user agent
VSE/1.0 (vsecrawler@hotmail.com) 2x detected (1x unique)
pub-crawler
Example user agent
Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com) 4x detected (2x unique)
iaskspider,iAsk Spider
Example user agent
iaskspider/2.0( http://iask.com/help/help_index.html) 3x detected (2x unique)
uMBot-FC/1.0,uMBot
Example user agent
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com) 4x detected (4x unique)
Yahoo! Japan,Y!J-BRI,Y!J-BRI/0.0.1,Y!J-AGENT
Example user agent
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html ) 2x detected (1x unique)
YisouSpider
Example user agent
YisouSpider 1x detected (1x unique)
AcquiloSpider
Example user agent
AcquiloSpider/1.0,gzip(gfe),gzip(gfe) 1x detected (1x unique)
blog_crawler
Example user agent
blog_crawler/1.0 2x detected (2x unique)
IlTrovatore,IlTrovatore/1.2
Example user agent
IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it) 2x detected (1x unique)
Scopia Crawler
Example user agent
Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co) 2x detected (2x unique)
SWEBot/1.0,SWEBot
Example user agent
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net) 2x detected (2x unique)
ZeBot_www.ze.bz,ZE.bz
Example user agent
ZeBot_www.ze.bz (ze.bz@hotmail.com) 3x detected (2x unique)
aggregator VocusBot,VocusBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; aggregator VocusBot 0.4; +http://www.vocus.com/vnhs.html) 2x detected (1x unique)
WebZIP
Example user agent
WebZIP/7.0 (http://www.spidersoft.com) 2x detected (2x unique)
LDSpider,lworldspider
Example user agent
lworldspider 1x detected (1x unique)
Altresium
Example user agent
Altresium/4.6 (+http://www.altresium.com/bot.html) 2x detected (2x unique)
Computer_and_Automation_Research_Institute_Crawler,Generic Research Crawler
Example user agent
Computer_and_Automation_Research_Institute_Crawler (crawler@info.ilab.sztaki.hu) 1x detected (1x unique)
T3census-Crawler
Example user agent
T3census-Crawler/1.0 3x detected (2x unique)
Flamingo_SearchEngine,Flamingo Search
Example user agent
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot) 1x detected (1x unique)
FlaxCrawler
Example user agent
FlaxCrawler/1.0,gzip(gfe),gzip(gfe) 2x detected (2x unique)
Orgbybot,OrgbyBot/1.2
Example user agent
Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine) 1x detected (1x unique)
Apexoo Spider
Example user agent
Apexoo Spider 1.1 1x detected (1x unique)
Lightningspider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2 1x detected (1x unique)
FastCrawler3
Example user agent
Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no) 1x detected (1x unique)
Crawly
Example user agent
Mozilla/5.0 (compatible; Crawly/1.9; http://92.51.162.40/crawler.html) 2x detected (2x unique)
Automattic Analytics Crawler/0.1,Automattic Analytics Crawler
Example user agent
Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/ 2x detected (2x unique)
VSE Link Tester,VSE
Example user agent
VSE/1.0 (testcrawler@hotmail.com) 1x detected (1x unique)
PaxleFramework
Example user agent
Mozilla/5.0 (compatible; PaxleFramework/0.1.1; http://www.paxle.net/en/bot) 1x detected (1x unique)
Inar_spider2
Example user agent
Inar_spider2 (tspyyp@tom.com) 1x detected (1x unique)
Feedjit Favicon Crawler
Example user agent
Feedjit Favicon Crawler 1.0 3x detected (2x unique)
spider,proximic
Example user agent
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php) 1x detected (1x unique)
ZoomSpider
Example user agent
ZoomSpider - wrensoft.com 1x detected (1x unique)
Yet-Another-Spider
Example user agent
Mozilla/4.0 (compatible; Yet-Another-Spider; ) 2x detected (1x unique)
Adaxas Spider
Example user agent
Adaxas Spider (http://www.adaxas.net/) 2x detected (2x unique)
VisBot,Visbot/2.0
Example user agent
VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) 0x detected (0x unique)
Example user agent
Mozilla/5.0 (Windows; U; Windows CE 4.21; rv:1.8.1a3) Gecko/20060610 Minimo/0.016 2x detected (2x unique)
Webspider,RankFlex
Example user agent
RankFlex.com Webspider 2x detected (2x unique)
PowerMapper,CrawlerProcess
Example user agent
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.10.736.0 2x detected (2x unique)
Lijit,Lijit Crawler
Example user agent
Lijit Crawler (+http://www.lijit.com/robot/crawler) 1x detected (1x unique)
WSDLSpider
Example user agent
WSDLSpider/1.0 (http://www.wsdlworld.com) 2x detected (2x unique)
BoardReader,TestCrawler
Example user agent
BoardReader/1.0 (http://boardreader.com/info/robots.htm)-Machine1 2x detected (2x unique)
CrawlWave,Crawlboy
Example user agent
CrawlWave/1.2 (crawlwave[at]circular.gr http://www.spiderwave.aueb.gr/ 1x detected (1x unique)
parallelContextFocusCrawler1
Example user agent
parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1 1x detected (1x unique)
VinjaVideoSpider
Example user agent
VinjaVideoSpider/1.1 1x detected (1x unique)
WebCompanyCrawler
Example user agent
WebCompanyCrawler 2x detected (1x unique)
Digincore bot
Example user agent
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions. 2x detected (2x unique)
stq_bot,Searchteq Bot
Example user agent
stq_bot (+http://www.searchteq.de) 2x detected (2x unique)
ExB Language Crawler 2.1.5,ExB Language Crawler
Example user agent
ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler) 5x detected (4x unique)
Yahoo! Japan,YFSJ crawler,Y!J-BRO/YFSJ,Y!J-AGENT
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0) 3x detected (2x unique)
robot,Apercite
Example user agent
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html) 1x detected (1x unique)
Votay bot
Example user agent
Mozilla/5.0 (Votay bot/4.0; http://www.votay.com/arts/comics/) 1x detected (1x unique)
NodejsSpider
Example user agent
NodejsSpider/1.0 1x detected (1x unique)
zcspider
Example user agent
IS Alpha/zcspider-0.1 3x detected (2x unique)
Sogou Develop Spider,sogou spider
Example user agent
Sogou develop spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07) 4x detected (4x unique)
salesforce radian6,R6_FeedFetcher,R6 FeedFetcher,Radian 6
Example user agent
R6_FeedFetcher(www.radian6.com/crawler) 1x detected (1x unique)
RSpider
Example user agent
RSpider/1.0 2x detected (1x unique)
Simpy
Example user agent
Simpy/1.1 (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com) 1x detected (1x unique)
JobSpider_BA
Example user agent
JobSpider_BA/1.1 5x detected (2x unique)
80Legs,008
Example user agent
Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620 2x detected (1x unique)
memoryBot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html) 1x detected (1x unique)
Wireless Crawler
Example user agent
EricssonR320/R1A (Fast Wireless Crawler) 3x detected (3x unique)
DomainsDB,MetaCrawler,DomainsDB.net MetaCrawler
Example user agent
DomainsDB.net MetaCrawler v.0.9.7b (http://domainsdb.net/) 2x detected (2x unique)
PagesInventory,PagesInventory Bot
Example user agent
PagesInventory (robot http://www.pagesinventory.com) 1x detected (1x unique)
amphetameme crawler
Example user agent
amphetameme crawler (crawler@amphetameme.com) 2x detected (2x unique)
Crawlboy,Crawler
Example user agent
Crawler 0.1,gzip(gfe),gzip(gfe) 2x detected (2x unique)
Kalooga,KaloogaBot
Example user agent
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler) 2x detected (2x unique)
AboutUsBot,AboutUs Bot
Example user agent
AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org) 3x detected (2x unique)
furlbot,FurlBot/Furl Search 2.0
Example user agent
Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net) 1x detected (1x unique)
Internet Spider
Example user agent
Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com) 1x detected (1x unique)
Iplexx Spider
Example user agent
Mozilla/4.0 (compatible; Iplexx Spider/1.1) 1x detected (1x unique)
DRKSpider
Example user agent
DRKSpider - Website link validator - http://www.drk.com.ar/spider/,gzip(gfe),gzip(gfe),gzip(gfe) 2x detected (2x unique)
Phantom.js bot,js bot
Example user agent
Phantom.js bot 1x detected (1x unique)
Yahoo! Japan
Example user agent
Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) 2x detected (1x unique)
Blogshares Spiders
Example user agent
Blogshares Spiders (Renewed V1.65) 1x detected (1x unique)
ZillaCrawler
Example user agent
ZillaCrawler 2x detected (2x unique)
TestCrawler,MultiCrawler
Example user agent
MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.html 2x detected (1x unique)
Google Wireless Transcoder
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;) 2x detected (2x unique)
Livelap Crawler,LivelapBot
Example user agent
LivelapBot/0.2 (http://site.livelap.com/crawler) 1x detected (1x unique)
NationalDirectory-WebSpider
Example user agent
NationalDirectory-WebSpider/1.3 1x detected (1x unique)
BebopBot
Example user agent
BebopBot/2.5.1 (compatible; media crawler V1; http://www.apassion4jazz.net/bebopbot.html;) 1x detected (1x unique)
Autonomy Spider
Example user agent
Autonomy Spider 3x detected (2x unique)
nuSearch,Nusearch Spider
Example user agent
NuSearch Spider www.nusearch.com 3x detected (2x unique)
pmoz.info ODP link checker,doc/botinfo
Example user agent
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm) 2x detected (2x unique)
GingerCrawler/1.0,GingerCrawler
Example user agent
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com) 1x detected (1x unique)
Analyzer Crawler
Example user agent
www.website-analyzer.net Website Analyzer Crawler 1x detected (1x unique)
TheUsefulbot
Example user agent
TheUsefulbot_2.3.62 (bot@theusefulnet.com) 3x detected (1x unique)
lmspider
Example user agent
lmspider (lmspider@scansoft.com) 2x detected (2x unique)
Arachnida Web Crawler,Web Crawler
Example user agent
Arachnida Web Crawler 1x detected (1x unique)
sgbot
Example user agent
Mozilla/5.0 (compatible; sgbot v0.01a, sgcrawlerbot@gmail.com) 1x detected (1x unique)
Spider
Example user agent
Spider/5.0 1x detected (1x unique)
Isara
Example user agent
Isara-Search/Isara-1.0 (A non-profit web crawler operated by a charity organization.; www.isara.org; webmaster@isara.org) 3x detected (2x unique)
Steeler,Steeler/3.3
Example user agent
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) 1x detected (1x unique)
NewzCrawler
Example user agent
NewzCrawler/1.9 (compatible; MSIE 6.00; Newz Crawler 1.9; http://www.newzcrawler.com/ ) 3x detected (3x unique)
ProjectWF-java-test-crawler,Java,General Crawlers
Example user agent
ProjectWF-java-test-crawler 2x detected (1x unique)
TestCrawler
Example user agent
Mozilla/5.0 (compatible; TestCrawler) 1x detected (1x unique)
SpiderView
Example user agent
Mozilla/4.0 (compatible; SpiderView 1.0;unix) 1x detected (1x unique)
YoonoCrawler
Example user agent
YoonoCrawler/1.0 (crawler@yoono.com) 2x detected (2x unique)
Zeusbot,Zeus
Example user agent
Zeusbot/0.8.1 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com) 1x detected (1x unique)
loc-crawler
Example user agent
Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://loc.gov),gzip(gfe),gzip(gfe),gzip(gfe) 1x detected (1x unique)
StoneSunSpider
Example user agent
Mozilla/5.0 (compatible; StoneSunSpider/1.1) 2x detected (2x unique)
PagesInventory Bot,PagesInventory
Example user agent
PagesInventory (robot +http://www.pagesinventory.com) 2x detected (1x unique)
Netseer crawler
Example user agent
compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com 2x detected (1x unique)
XSpider
Example user agent
XSpider 0.01;http://uncool.oicp.net/spider.html 1x detected (1x unique)
RavenCrawler
Example user agent
RavenCrawler 1x detected (1x unique)
exooba crawler
Example user agent
exooba/exooba crawler (exooba; exooba) 2x detected (2x unique)
Hatena Crawler,Hatena::Crawler
Example user agent
Hatena::Crawler/0.01 1x detected (1x unique)
Arikus_Spider
Example user agent
Arikus_Spider 2x detected (1x unique)
OpenHoseBot
Example user agent
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html) 0x detected (0x unique)
Example user agent
Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) ) 2x detected (1x unique)
uMBot
Example user agent
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com) 1x detected (1x unique)
mbot
Example user agent
mbot v.1.16 2x detected (1x unique)
SMTBot
Example user agent
Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot) 2x detected (1x unique)
KomodiaBot
Example user agent
Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0 2x detected (1x unique)
SMTBot
Example user agent
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot) 1x detected (1x unique)
ExactSeek_Spider
Example user agent
ExactSeek_Spider 1x detected (1x unique)
FAST EnterpriseCrawler
Example user agent
FAST EnterpriseCrawler 6 1x detected (1x unique)
LEIA
Example user agent
LEIA/3.01pr (LEIAcrawler; leia@gseek.com; http://www.gseek.com) 2x detected (2x unique)
Adbeat Bot,adbeat_bot
Example user agent
adbeat_bot 1x detected (1x unique)
TouTrix crawler
Example user agent
TouTrix crawler/1.0 1x detected (1x unique)
3 Robot
Example user agent
Abrave Spider v5.3 Robot 2 (http://robot.abrave.com) 1x detected (1x unique)
moba-crawler
Example user agent
DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/) 2x detected (2x unique)
SafeSearch microdata crawler,microdata crawler
Example user agent
SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com) 1x detected (1x unique)
Comodo-Webinspector-Crawler
Example user agent
Comodo-Webinspector-Crawler 2.1 2x detected (2x unique)
NikiBot,niki-bot
Example user agent
niki-bot 1x detected (1x unique)
ShopSalad
Example user agent
SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT shopsalad.com) 1x detected (1x unique)
Lucidworks Bot
Example user agent
Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com) 0x detected (0x unique)
Example user agent
Mozilla/5.0 (Linux; U; Android 2.2.2; en-us; LG vortex Build/FRG83G) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 1x detected (1x unique)
RyzeCrawler
Example user agent
RyzeCrawler/1.1.0 (+http://www.ryze.nl/crawler/) 2x detected (1x unique)
Xaldon WebSpider
Example user agent
Xaldon WebSpider 2.7.b6 4x detected (3x unique)
IRLbot/3.0,IRLbot,Research Projects
Example user agent
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) 3x detected (1x unique)
IRLbot
Example user agent
IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler) 2x detected (1x unique)
i1searchbot
Example user agent
i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com) 1x detected (1x unique)
SpiderKU
Example user agent
SpiderKU/0.9 1x detected (1x unique)
clumboot
Example user agent
Mozilla/5.0 (compatible; clumboot; +http://localhost/clumpit/bot.php) 2x detected (1x unique)
YRL_ODP_CRAWLER
Example user agent
YRL_ODP_CRAWLER 5x detected (1x unique)
FAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/6 (www.fastsearch.com) 2x detected (2x unique)
Research Crawler,Generic Research Crawler
Example user agent
LinkStar Research Crawler (http://linkstar.com/),gzip(gfe),gzip(gfe) 1x detected (1x unique)
search crawler
Example user agent
camoca/camoca-n.1.2 (super-agent; search crawler; info at does not exist dot com) 2x detected (1x unique)
PostPost
Example user agent
PostPost/1.0 ( http://postpo.st/crawlers) 1x detected (1x unique)
OpenCrawler
Example user agent
Mozilla/5.0 (compatible; OpenCrawler/0.1.6.6; http://code.google.com/p/opencrawler/),gzip(gfe),gzip(gfe) 1x detected (1x unique)
Impressumscrawler
Example user agent
netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler) 1x detected (1x unique)
bmdspider
Example user agent
Mozilla/4.0 (compatible; bmdspider; windows 5.1) 1x detected (1x unique)
Tailsweep
Example user agent
TailsweepBlogCrawler/Tailsweep-2.9-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com) 1x detected (1x unique)
Toms Spider
Example user agent
Toms Spider 0.3 1x detected (1x unique)
Yahoo-Newscrawler
Example user agent
Yahoo-Newscrawler/3.9 (news-search-crawler at yahoo-inc dot com) 1x detected (1x unique)
Engine Spider
Example user agent
ByWebSite Search Engine Spider 1x detected (1x unique)
ProoXiBot
Example user agent
ProoXiBot 1x detected (1x unique)
SWAT Crawler
Example user agent
SWAT Crawler. AGH University project. In case of problem contact: opal@tempus.metal.agh.edu.pl. Thanks. 1x detected (1x unique)
Kotoss Crawler
Example user agent
Kotoss Crawler http://hitokoto.kotoss.com,gzip(gfe),gzip(gfe) 1x detected (1x unique)
SpiderMan
Example user agent
SpiderMan 3.0.1-2-11-111 (CP/M;8-bit) 3x detected (2x unique)
ArchiveBot,ArchiveTeam ArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36 1x detected (1x unique)
FAST Crawler
Example user agent
Mozilla/5.0 (compatible; FAST Crawler 6.3) 1x detected (1x unique)
Comodo-Certificates-Spider
Example user agent
Comodo-Certificates-Spider 2x detected (1x unique)
FollowSite Bot
Example user agent
FollowSite Bot ( http://www.followsite.com/bot.html ) 4x detected (2x unique)
holmes/3.12.4 - morfeo,Holmes
Example user agent
holmes/3.12.4 (http://morfeo.centrum.cz/bot) 1x detected (1x unique)
pregnancycrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com) 3x detected (3x unique)
WASALive-Bot,WASALive Bot,WASALive-Bot
Example user agent
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/) 1x detected (1x unique)
TelemetrySpider2
Example user agent
TelemetrySpider2/0.1 linux 1x detected (1x unique)
LB-Crawler
Example user agent
LB-Crawler/1.0 1x detected (1x unique)
ERACrawler
Example user agent
ERACrawler/1.0 1x detected (1x unique)
KiwiStatus Spider
Example user agent
KiwiStatus (NZS.com)/0.2 (NZS.com KiwiStatus Spider, Local Search New Zealand; http://www.nzs.com; bot-at-nzs dot com) 1x detected (1x unique)
PortalBSpider
Example user agent
PortalBSpider/2.0 (spider@portalb.com) 2x detected (2x unique)
chlooe Bot,bot-pge.chlooe.com/1.0.0
Example user agent
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/) 2x detected (2x unique)
DocWeb Link Crawler,Link Crawler
Example user agent
DocWeb Link Crawler (http://doc.php.net) 3x detected (3x unique)
AboutUs Bot Johnny5,AboutUsBot Johnny5/2.0,AboutUsBot
Example user agent
Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/) 2x detected (1x unique)
zitebot
Example user agent
Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com) 2x detected (2x unique)
Experibot_v1,Experibot
Example user agent
Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html) 2x detected (1x unique)
SemanticBot
Example user agent
Mozilla/5.0 (compatible; SemanticBot/1.1) 2x detected (1x unique)
AdvBot
Example user agent
Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html) 2x detected (1x unique)
Netchart Adv Crawler
Example user agent
Netchart Adv Crawler 1.0 2x detected (2x unique)
CompSpyBot,CompSpyBot/1.0
Example user agent
Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html) 1x detected (1x unique)
zyklop-crawler
Example user agent
Mozilla/5.0 (compatible; zyklop; +http://www.seoratio.de/zyklop-crawler/) 2x detected (1x unique)
CrawlRobot
Example user agent
Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php) 1x detected (1x unique)
archive.bibalex.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.bibalex.org_bot; +http://www.test.com) 2x detected (2x unique)
DueDil Crawler,our-crawler
Example user agent
Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/) 2x detected (1x unique)
MetaGeneratorCrawler
Example user agent
MetaGeneratorCrawler/1.3.8 (www.metagenerator.info) 2x detected (2x unique)
robot,Werbefreie Deutsche Suchmaschine
Example user agent
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html) 4x detected (2x unique)
Yahoo-MMCrawler,Yahoo! Image
Example user agent
Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) 2x detected (1x unique)
Vagabondo
Example user agent
Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) 2x detected (1x unique)
ImplisenseBot
Example user agent
ImplisenseBot 1.0 1x detected (1x unique)
NimbleCrawler
Example user agent
Mozilla/4.0 (NimbleCrawler 0.1) obeys NimbleCrawler directive contact jpump@looksmart.net 3x detected (3x unique)
com Bot,Qualidator.com Bot,Qualidator.com Bot 1.0
Example user agent
Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;) 5x detected (2x unique)
FAST-WebCrawler,FAST Web Crawler
Example user agent
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) 2x detected (2x unique)
Fake IE,MSIECrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/4.0; FDM; MSIECrawler; Media Center PC 5.0) 1x detected (1x unique)
Aleksika Spider
Example user agent
Aleksika Spider/1.0 ( http://www.aleksika.com/) 3x detected (2x unique)
ConveraCrawler,Convera
Example user agent
ConveraCrawler/0.9c ( http://www.authoritativeweb.com/crawl) 3x detected (2x unique)
WBSearchBot,WBSearchBot/1.1
Example user agent
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html) 1x detected (1x unique)
NMG Spider
Example user agent
NMG Spider/0.3 (szukanko.com) 3x detected (1x unique)
MetaHeadersBot
Example user agent
MetaHeadersBot (+http://www.metaheaders.com/bot.html) 2x detected (1x unique)
DomainsBot
Example user agent
domainsbot (+http://www.domainsbot.com) 1x detected (1x unique)
MSIECrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 4.0; MSIECrawler; Windows 95) 1x detected (1x unique)
ly spider
Example user agent
trunk.ly spider contact@trunk.ly 1x detected (1x unique)
LesnikBot
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot) 1x detected (1x unique)
8webcrawler
Example user agent
Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8webcrawler) http://skateboarddirectory.com 2x detected (1x unique)
PWBot
Example user agent
Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw) 2x detected (2x unique)
Thinglink ImageBot,Elmer
Example user agent
Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot) 2x detected (1x unique)
Tutorial Crawler
Example user agent
Tutorial Crawler 1.4 2x detected (2x unique)
Fake IE,CrawlDaddy
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.1.1.0 http://code.google.com/p/abot) 3x detected (2x unique)
fr_bot,bnf.fr_bot
Example user agent
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html) 1x detected (1x unique)
JikeSpider
Example user agent
JikeSpider; +http://shoulu.jike.com/spider.html 2x detected (1x unique)
Thumbshots.ru
Example user agent
Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3 1x detected (1x unique)
PornSpider
Example user agent
Mozilla/5.0 (compatible; PornSpider/1.0; +http://www.pornspider.net) 1x detected (1x unique)
AnomaliesBot
Example user agent
AnomaliesBot/0.06-dev (The Anomalies Network Search Spider; http://www.anomalies.net; info@anomalies.net) 4x detected (4x unique)
updated,Updated!,updated crawler,updated/0.1-alpha
Example user agent
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com) 1x detected (1x unique)
bot
Example user agent
bot 2x detected (2x unique)
Gomez Site Monitor,GomezAgent
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1 2x detected (1x unique)
fr-crawler
Example user agent
Mozilla/5.0 (compatible; fr-crawler/1.1) 5x detected (2x unique)
80legs,008
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620 1x detected (1x unique)
SheenBot
Example user agent
SheenBot/SheenBot-1.0.4 (Sheen web crawler) 2x detected (1x unique)
JUST-CRAWLER
Example user agent
JUST-CRAWLER( http://www.justsystems.com/jp/tech/crawler/) 3x detected (2x unique)
Sogou Push Spider,sogou spider
Example user agent
Sogou Push Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) 2x detected (1x unique)
Checklinks
Example user agent
Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks) 2x detected (1x unique)
GG PeekBot
Example user agent
GG PeekBot 2.0 ( http://gg.pl/ http://info.gadu-gadu.pl/praca ) 3x detected (2x unique)
WinkBot,WinkBot/0.06
Example user agent
WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com) 1x detected (1x unique)
cloudforecastbot
Example user agent
cloudforecastbot 1x detected (1x unique)
eseek-crawler
Example user agent
eseek-crawler.0.5 (crawler@exactseek.com) 1x detected (1x unique)
Crayon Crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Crayon Crawler) 1x detected (1x unique)
TwitterCrawler
Example user agent
Mozilla/5.0 (compatible; TwitterCrawler) 2x detected (2x unique)
ANTIPLAG SPIDER,DAWINCI ANTIPLAG SPIDER
Example user agent
Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER) 2x detected (1x unique)
Facebot
Example user agent
Facebot/1.0 0x detected (0x unique)
Example user agent
Mozilla/5.0 (Windows; U; Windows CE 5.2; rv:1.8.1.4pre) Gecko/20070327 Minimo/0.020 1x detected (1x unique)
ip-web-crawler
Example user agent
ip-web-crawler.com 1x detected (1x unique)
Goodzer
Example user agent
Mozilla/5.0 (compatible; Goodzer/2.0; crawler@goodzer.com) 2x detected (2x unique)
ProCogBot,ProCogBot/1.0
Example user agent
Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html) 1x detected (1x unique)
VodpodCrawler
Example user agent
Mozilla/5.0 (compatible; VodpodCrawler/1.0; +http://vodpod.com/site/help) 3x detected (2x unique)
ssearch_bot,sSearch Crawler
Example user agent
ssearch_bot (sSearch Crawler; http://www.semantissimo.de) 1x detected (1x unique)
TwitSpider
Example user agent
TwitSpider 2x detected (1x unique)
metager2-verification-bot
Example user agent
Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php) 3x detected (2x unique)
envolk[ITS]spider,Envolk
Example user agent
envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html) 2x detected (1x unique)
NetzCheckBot
Example user agent
NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com) 1x detected (1x unique)
GomezAgent
Example user agent
Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT) 1x detected (1x unique)
XYZ Spider
Example user agent
XYZ Spider 1x detected (1x unique)
PercolateCrawler
Example user agent
PercolateCrawler/4 (ops@percolate.com) 1x detected (1x unique)
ZemlyaCrawl
Example user agent
Mozilla/5.0 (compatible; ZemlyaCrawl/1.0; +http://zemlyaozer.com/bot) 2x detected (1x unique)
GLBot
Example user agent
Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co) 1x detected (1x unique)
IXE Crawler
Example user agent
IXE Crawler
Comparison created
2016-02-13 13:25:41 | by
ThaDafinser