Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)Googlebot-richsnippets
Example user agent
Googlebot-richsnippetsCrawllybot
Example user agent
Crawllybot/0.1/0.1 (Crawllybot/0.1; http://www.crawlly.com; crawler@crawlly.com)www.almaden.ibm.com
Example user agent
http://www.almaden.ibm.com/cs/crawlerabot
Example user agent
abot/0.1 (abot; http://www.abot.com; abot@abot.com)spbot
Example user agent
Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )KDDI-Googlebot-Mobile
Example user agent
KDDI-CA34 UP.Browser/6.2.0.10.2.2(GUI)MMP/2.0 (compatible; KDDI-Googlebot-Mobile/2.1; http://www.google.com/bot.html)RedCarpet
Example user agent
RedCarpet/1.0 (http://www.redcarpet-inc.com/robots.html)AdnormCrawler
Example user agent
AdnormCrawler www.adnorm.com/crawlerDataparkSearch
Example user agent
DataparkSearch/4.35-02122005 ( http://www.dataparksearch.org/)MetaGeneratorCrawler
Example user agent
MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)FatBot
Example user agent
Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)Nutch
Example user agent
MySpider/Nutch-2.2InfuzApp
Example user agent
Mozilla/5.0 (compatible; InfuzApp/1.0; +http://www.infuz.com/bot.html)wume_crawler
Example user agent
wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)NewsGator
Example user agent
NewsGator/2.0 Bot (http://www.newsgator.com)Googlebot
Example user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36Ask Jeeves
Example user agent
Inne: Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html)Cloudview Crawler
Example user agent
Exalead Cloudview Crawler,gzip(gfe),gzip(gfe),gzip(gfe)Sogou web spider
Example user agent
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htmawesomebar_scraper
Example user agent
awesomebar_scraper/1.0ExactSeek Crawler
Example user agent
ExactSeek Crawler/0.1CydralSpider
Example user agent
CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com)Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2,22- Example user agent
Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5) 80bot
Example user agent
Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620Diffbot
Example user agent
Diffbot/0.1msnbot
Example user agent
msnbot/1.1 ( http://search.msn.com/msnbot.htm)RSS-SPIDER
Example user agent
RSS-SPIDER (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php)EasouSpider
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us; EasouSpider; +http://www.easou.com/search/spider.html)VM-Crawler
Example user agent
Mozilla/4.0, VM-Crawler/cs version info ofni noisrev scRDF crawler
Example user agent
OpenLink Virtuoso RDF crawlerBot Blocker Crawler
Example user agent
Bot Blocker Crawler 1.0 (btw, IncrediBILL says "HI!")SRCCN!Spider
Example user agent
Mozilla/5.0 (compatible; SRCCN!Spider/1.1; +http://site.srccn.com/spider.html)Beehive Crawler
Example user agent
ZoomInfo::Beehive Crawler50.nu
Example user agent
50.nu/0.01 ( http://50.nu/bot.html )Baiduspider-cpro
Example user agent
Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)YodaoBot-Image
Example user agent
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)GoogleBot
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1archiver
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)Nusearch Spider
Example user agent
Nusearch Spider (compatible; MSIE 6.0)SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)Googlebot
Example user agent
Googlebot/2.1 (+http://www.google.com/bot.html)WinWebBot
Example user agent
WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;)NewsGatorOnline
Example user agent
NewsGatorOnline/2.0 (http://www.newsgator.com)Sosospider
Example user agent
Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)SMTBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)Zookabot
Example user agent
Zookabot/2.4; +http://zookabot.comFyberSpider
Example user agent
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)enicura crawler
Example user agent
enicura crawler 1.0,gzip(gfe),gzip(gfe)rogerbot
Example user agent
rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+pr1-crawler-14@moz.com)NetSeer crawler
Example user agent
Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)msnbot
Example user agent
msnbot/2.0b v1394720947.6703YFSBJ crawler
Example user agent
Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80)Infohelfer
Example user agent
Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)Googlebot-Video
Example user agent
Googlebot-Video/1.0Findexa Crawler
Example user agent
Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece)dotbot
Example user agent
dotbotVagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/2.3; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)com/bot
Example user agent
Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)BoardReader Favicon Fetcher
Example user agent
BoardReader Favicon Fetcher /1.0 info@boardreader.comUptimeRobot
Example user agent
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)- Example user agent
Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/) 123metaspider-Bot
Example user agent
123metaspider-Bot (Version: 1.04, powered by www.123metaspider.com)YoudaoBot
Example user agent
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )AddSugarSpiderBot
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) AddSugarSpiderBot www.idealobserver.comWhatWeb
Example user agent
WhatWeb/0.4.8it-Crawler
Example user agent
Liquida.it-Crawler/1.0 ( crawler@liquida.it +http://www.liquida.it )CosmixCrawler
Example user agent
CosmixCrawler/0.1BurstFind Crawler
Example user agent
BurstFind Crawler 1.1 - www.burstfind.comContent Crawler
Example user agent
Content CrawlerNewsGatorOnline
Example user agent
NewsGatorOnline/2.0 (http://www.newsgator.com; 2 subscribers)TridentSpider
Example user agent
Mozilla/5.0 (compatible; TridentSpider/3.1)lworld spider
Example user agent
Mozilla/4.0 (compatible; lworld spider 1.0; Windows NT 5.1)heritrix
Example user agent
Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)PartnerSite Crawler
Example user agent
Fast PartnerSite CrawlerDoor Spider
Example user agent
FFC Trap Door Spiderloc-crawl
Example user agent
Mozilla/5.0 (compatible; loc-crawl/1.10.1 +http://www.google.com)- Example user agent
Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html) spbot
Example user agent
Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )Baiduspider
Example user agent
BaiduspiderCrawler
Example user agent
1NET/Crawler (compatible; 1NET; robot; http://www.1NET.gr)- Example user agent
BOT/0.1 (BOT for JCE) SputnikBot
Example user agent
Mozilla/5.0 (compatible; SputnikBot/2.2)thumbshots-de-bot
Example user agent
thumbshots-de-bot (+http://www.thumbshots.de/)ZyBorg
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)SpiderLing
Example user agent
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)woriobot
Example user agent
Mozilla/5.0 (compatible; woriobot +http://worio.com)ZyBorg
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0PlantyNet_WebRobot
Example user agent
PlantyNet_WebRobot_V1.9 babo@plantynet.comGoGuidesBot
Example user agent
GoGuidesBot/0.0.1 (GoGuides Indexing Spider; http://www.goguides.org/spider.html)SSL-Crawler
Example user agent
Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36; SSL-Crawler: http://crawler.dcsec.uni-hannover.deGarlikCrawler
Example user agent
GarlikCrawler/1.2 (http://garlik.com/AdsBot-Google
Example user agent
AdsBot-Google (+http://www.google.com/adsbot.html)ADJUSTbot
Example user agent
Mozilla/5.0 (compatible; ADJUSTbot/2.0; +http://www.ad-just.jp/)SiteCheck-sitecrawl
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) SiteCheck-sitecrawl by Siteimprove.comLikaholixCrawler
Example user agent
Mozilla/5.0 (compatible; LikaholixCrawler/1.0; +http://mylikes.com/about/crawler)AskPeterBot
Example user agent
Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html)heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130207.001528 +http://webarchiv.cz/kontakty/)GematchCrawler
Example user agent
GematchCrawler/2.1 (http://www.gematch.com/crawler.html)LSSRocketCrawler
Example user agent
LSSRocketCrawler/1.0 LightspeedSystemsMXit WebBot
Example user agent
Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.3.119) Opera Mini/3.1hitcrawler_0
Example user agent
hitcrawler_0.1 (ringken@gmail.com)ZIBB Crawler
Example user agent
ZIBB Crawler (email address / WWW address)WIRE
Example user agent
WIRE/0.11 (Linux; i686; Robot,Spider,Crawler,aromano@cli.di.unipi.it)heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.fPingdomBot
Example user agent
Pingdom.com_bot_version_1.4_(http://www.pingdom.com)Mobile Spider1
Example user agent
Sogou Mobile Spider1.0 (http://wap.sogou.com)HyperCrawl
Example user agent
Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)WebThumb
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)DotBot
Example user agent
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)WebCrawler
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.htmlInfocrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Infocrawler; Alexa Toolbar)Sosoimagespider
Example user agent
Sosoimagespider ( http://help.soso.com/soso-image-spider.htm)WangIDSpider
Example user agent
Mozilla/5.0 (compatible; WangIDSpider/1.0; +http://www.wangid.com/spider.html)USyd-NLP-Spider
Example user agent
USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)IssueCrawler
Example user agent
IssueCrawlercom Crawler
Example user agent
WebTarantula.com CrawlerPeerFactor Crawler
Example user agent
PeerFactor CrawlerSpeedySpider
Example user agent
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)Sosospider
Example user agent
Sosospider+(+help.soso.com/webspider.htm)Nutch
Example user agent
Innovazion Crawler/Nutch-1.7sReleaseBot
Example user agent
Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/)n4p_bot
Example user agent
n4p_bot (crawler@n4p.com)ReadPath_Spider
Example user agent
ReadPath_Spiderrobot
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19Imperia-LinkSpider
Example user agent
Imperia-LinkSpider,gzip(gfe),gzip(gfe)AppEngine-Google
Example user agent
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko AppEngine-Google; (+http://code.google.com/appengine; appid: s~oowsxs)XSpider
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 7.0) XSpider 7linkdexbot
Example user agent
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)SmartAndSimpleWebCrawler
Example user agent
SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net),gzip(gfe),gzip(gfe)LookUpCrawler
Example user agent
LookUpCrawler - lookupcanada.ca [ZSEBOT]Icarus6j
Example user agent
Icarus6j - (contact: phil@icarus6.com)Overture-WebCrawler
Example user agent
Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://netarkivet.dk/webcrawler/)- Example user agent
TrendWinHttp/ Wotbox
Example user agent
Wotbox/0.7-alpha (bot@wotbox.com; http://www.wotbox.com)MicrosoftPrototypeCrawler
Example user agent
MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com)UOLCrawler
Example user agent
UOLCrawler (soscrawler@uol.com.br)MySpaceScraper
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 6.1; en; rv:1.9.1.3; MySpaceScraper) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)imrbot
Example user agent
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.com/search/spider.htm)gonzo1
Example user agent
gonzo1[D] mailto:crawleradmin.t-info@telekom.demsnbot
Example user agent
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)HttpSpider
Example user agent
HttpSpider/0.91UA crawler
Example user agent
Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler)Googlebot
Example user agent
Googlebot/2.1 (http://www.googlebot.com/bot.html)Web-sniffer
Example user agent
Mozilla/5.0 (Windows; Windows NT 6.1; WOW64; rv:2.0b8pre; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; SLCC1; SLCC2; Media Center PC 6.0; GTB6.4; InfoPath.2; en-US; FunWebProducts; Zango 10.1.181.0; SV1; PRTG Network Monitor (www.paessler.com)) Gecko/20101114 Firefox/4.0b8pre QuickTime/7.6.2 Songbird/1.1.2 Web-sniffer/1.0.36 lftp/3.7.4 libwww-perl/5.820 GSiteCrawler/v1.12 rev. 260 Snoopy v1.2mapion-news-bot
Example user agent
mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)VoilaBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)- Example user agent
SearQuBot/SearQuBot v1.0 PAD-bot
Example user agent
Mozilla/5.0 (compatible; PAD-bot/9.0; +http://www.descargarprogramagratis.com/)MQbot
Example user agent
MQbot http://metaquerier.cs.uiuc.edu/crawlermsnbot
Example user agent
msnbot/2.0 (+http://search.msn.com/msnbot.htm)Symfony Spider
Example user agent
Symfony Spider (http://symfony.com/spider)memorybot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)discoverybot
Example user agent
Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)it2media-domain-crawler
Example user agent
it2media-domain-crawler/1.0 on crawler-prod.it2media.deadspider
Example user agent
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.21 (KHTML, like Gecko) adspider Safari/537.21JobdiggerSpider
Example user agent
Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)WebCrunch
Example user agent
Mozilla/5.0 (compatible; WebCrunch/1.2; +http://webcrunch.net/crawler)CamontSpider
Example user agent
CamontSpider/1.0 http://epweb2.ph.bham.ac.uk/user/slater/camont/info.htmlNWSpider
Example user agent
NWSpider 0.9baiduspider-mobile-gate
Example user agent
baiduspider-mobile-gateUS-Crawler
Example user agent
Links4US-Crawler, ( http://links4us.com/)LoadTimeBot
Example user agent
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)AdsBot-Google
Example user agent
AdsBot-GoogleXenu Link Sleuth
Example user agent
Xenu Link Sleuth 1.1faudioCrawlerBot
Example user agent
audioCrawlerBot/1.0 (http://www.audiocrawler.com/)heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot)envolk
Example user agent
envolk/1.7 ( http://www.envolk.com/envolkspider.html)Amfibibot
Example user agent
Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)YRSpider
Example user agent
Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html)CMS crawler
Example user agent
CMS crawler ( http://buytaert.net/crawler/)Socialradarbot
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com)MarketwireBot
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com)SISTRIX Crawler
Example user agent
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)GooglePlusBot
Example user agent
Google (+https://developers.google.com/+/web/snippet/)XoviBot
Example user agent
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)Axtaris
Example user agent
Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; http://axtaris.org; crawler at axtaris dot org)mixi-crawler
Example user agent
mixi-crawler/2.00 (http://mixi.jp/)NING
Example user agent
NING/1.0rogerbot
Example user agent
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)- Example user agent
ShowyouBot (http://showyou.com/crawler) ProloCrawler
Example user agent
ProloCrawler/1.0 (http://www.prolo.com)HubSpot Crawler
Example user agent
HubSpot Crawler 1.0 http://www.hubspot.com/Esribot
Example user agent
Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)Googlebot
Example user agent
Googlebot/2.1 ( http://www.googlebot.com/bot.html)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)compatible crawler
Example user agent
http://www.Syntryx.com/ ANT Chassis 9.27; Mozilla/4.0 compatible crawlerMETASpider
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) METASpiderPDFBot
Example user agent
PDFBot (crawler@pdfind.com)LinkpadBot
Example user agent
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)KIT webcrawler
Example user agent
KIT webcrawler/0.2.4DotSpotsBot
Example user agent
DotSpotsBot/0.2 (crawler; support at dotspots.com)beta crawler
Example user agent
snap.com beta crawler v0GSiteCrawler
Example user agent
GSiteCrawler/v1.12 rev. 260 (http://gsitecrawler.com/)SurveyBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)BravoBrian SpiderEngine
Example user agent
BravoBrian SpiderEngine MarcoPoloVoilaBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)PsSpider
Example user agent
PsSpider/0.7 (spider@bildkiste.de)BT Crawler
Example user agent
BT Crawler 1.0,gzip(gfe),gzip(gfe),gzip(gfe)Zookabot
Example user agent
Zookabot/2.5;++http://zookabot.comGnam Spider
Example user agent
Gnam Gnam SpiderQuickFinder Crawler
Example user agent
QuickFinder CrawlerFacebookBot
Example user agent
facebookexternalhit/1.1360spider
Example user agent
360spider-imageFinderbots
Example user agent
Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)CFG_SPIDER_USER_AGENT
Example user agent
CFG_SPIDER_USER_AGENTSafari
Example user agent
NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)Googlebot
Example user agent
Googlebot/2.1 (+http://www.googlebot.com/bot.html)Googlebot
Example user agent
Mozilla/5.0 (Windows; Mozilla 4.0(compatible; Googlebot/5.0; http://www.google.com/googlebot); en-US;)Axadine Crawler
Example user agent
axadine/ (Axadine Crawler; http://www.axada.de/; )EveryoneSocialBot
Example user agent
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)Jambot
Example user agent
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)WebThumb
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail GeneratBabalooSpider
Example user agent
BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)Llaut
Example user agent
Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)Grub
Example user agent
Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org)ldspider
Example user agent
ldspider (BTC 2011 crawl, harth@kit.edu, http://code.google.com/p/ldspider/wiki/Robots)RoboCrawl
Example user agent
RoboCrawl (www.canadiancontent.net)Internet Crawler
Example user agent
Willow Internet Crawler by Twotrees V2.1Comodo Spider
Example user agent
Comodo Spider 1.1yp-crawl
Example user agent
yp-crawl@attinteractive.comAESOP_com_SpiderMan
Example user agent
AESOP_com_SpiderManYahooSeeker
Example user agent
YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)archive_crawler
Example user agent
Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 http://www.archive.org/details/archive_crawler)Using Fast Enterprise Crawler
Example user agent
RB2B-bot v0.1 (Using Fast Enterprise Crawler 6 search@reedbusiness.com)com crawler
Example user agent
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)yrspider
Example user agent
yrspider (Mozilla/5.0 (compatible; YRSpider; http://www.yunrang.com/yrspider.html))ArchiveTeam ArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36AjaxSnapBot
Example user agent
Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html)Baiduspider
Example user agent
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/searchSlackbot
Example user agent
Slackbot-LinkExpanding+(+https://api.slack.com/robots)DNSGroup
Example user agent
DNSGroup/0.1 (DNS Group Crawler; http://www.dnsgroup.com/; crawler@dnsgroup.com)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)5Bot
Example user agent
Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net)Engage Crawler
Example user agent
Adknowledge Engage CrawlerAcoonBot
Example user agent
Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)My Spider
Example user agent
My Spider 1.0,gzip(gfe),gzip(gfe)Gigabot
Example user agent
Gigabot/3.0 (http://www.gigablast.com/spider.html)GetMeLinked Spider
Example user agent
Mozilla 4.0 - GetMeLinked Spider www.GetMeLinked.com Web DirectorySteeler
Example user agent
Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)m65bot
Example user agent
DoCoMo/2.0 P904i( m65bot/0.1; c; http://m65.jp/bot.html )DuckDuckGo-Favicons-Bot
Example user agent
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)ichiro/mobile
Example user agent
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)mCrawler
Example user agent
Vodafone mCrawler (bergum@fast.no)SpokeSpider
Example user agent
SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)Automattic Analytics Crawler
Example user agent
Automattic Analytics Crawler/0.2;+http://wordpress.com/crawler/Java
Example user agent
Java/1.8.0_25sproose
Example user agent
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com)HRCrawler
Example user agent
HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko)voyager
Example user agent
voyager/1.0Googlebot
Example user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Denodo IECrawler
Example user agent
Denodo IECrawler/4.5,gzip(gfe),gzip(gfe)YATS crawler
Example user agent
Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)HiddenMarket
Example user agent
HiddenMarket-1.0-beta (www.hiddenmarket.net/crawler.php)NerdyBot
Example user agent
NerdyBotblackspider
Example user agent
blackspiderIndex crawler
Example user agent
IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org)InternetArchive
Example user agent
InternetArchive-1.0Research spider
Example user agent
Research spider - ak1835@albany.eduFAST FreshCrawler
Example user agent
schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; Contact: webcrawl@schibstedsok.no;)gsa-crawler
Example user agent
gsa-crawlerwoobot
Example user agent
woobot/1.1boitho crawler
Example user agent
boitho crawlerGooglebot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)CRAWL-E
Example user agent
CRAWL-E/0.6.4MonkeyCrawl
Example user agent
MonkeyCrawl/0.05 (MonkeyCrawl; http://www.monkeymethods.org; )msnbot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; msnbot/2.1) Jakarta Commons-HttpClient/3.0-rc3 PHPCrawl GStreamer souphttpsrc libsoup/2.27.4 PycURL/7.19.0 XML-RPC for PHP 2.2.1 GoogleFriendConnect/1.0 HTMLParser/1.6 gPodder/0.15.2 ( http://gpodder.org/) anw webtool LoadControl/1.3 WinHttp urlgrabber/3.1.0info Crawler
Example user agent
visaduhoc.info CrawlerISSpider
Example user agent
BlogPulse (ISSpider-3.0)QuerySeekerSpider
Example user agent
QuerySeekerSpider ( http://queryseeker.com/bot.html )OpenWebSpider
Example user agent
OpenWebSpider v0.1.4 (http://www.openwebspider.org/)bingbot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)Crawlzilla
Example user agent
Crawlzilla/1.0 (Crawlzilla; http://www.crawlzilla.com; crawler@crawlzilla.com)AdsBot-Google
Example user agent
AdsBot-Google ( http://www.google.com/adsbot.html)NE Crawler
Example user agent
netEstate NE Crawler (+http://www.sengine.info/)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)Comodo-Webinspector-Crawler
Example user agent
Comodo-Webinspector-Crawler 2.2.2, http://www.comodorobot.comDoubleVerify Crawler
Example user agent
DoubleVerify Crawleracquia-crawler
Example user agent
acquia-crawler (detected bad behaviour? please tell us at it@acquia.com),gzip(gfe),gzip(gfe)Googlebot
Example user agent
GooglebotArchitextSpider
Example user agent
ArchitextSpiderTweetedTimes Bot
Example user agent
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0 (resolver); +http://tweetedtimes.com)DotBot
Example user agent
DotBot/1.0.1 (http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)Nutch
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7Steeler
Example user agent
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)Sosoblogspider
Example user agent
Sosoblogspider+(+http://help.soso.com/soso-blog-spider.htm)archiver/3.1.1 +http://www.archive.org/details/archive.org_bot
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)lbot
Example user agent
Mozilla/5.0 (lbot/2.6.4) lbot@mailme.notredditbot
Example user agent
Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)unchaos_crawler_2
Example user agent
unchaos_crawler_2.0.2 (search.engine@unchaos.com)uipbot
Example user agent
uipbot/1.0 (uipbot@semasio.net)- Example user agent
Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html) AntBot
Example user agent
AntBot/1.0 (http://www.ant.com)de/bot
Example user agent
Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)- Example user agent
R6_CommentReader(www.radian6.com/crawler) - Example user agent
Yahoo Pipes 2.0 WeSEE:Search
Example user agent
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)Sogou Orion spider
Example user agent
Sogou Orion spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)YandexNewslinks
Example user agent
Mozilla/5.0 (compatible; YandexNewslinks; +http://yandex.com/bots)Googlebot-News
Example user agent
Googlebot-NewsSocialSpider
Example user agent
SocialSpider-Finder/0.2ptd-crawler
Example user agent
Mozilla/5.0 (compatible; ptd-crawler; http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)Fujiko Spider
Example user agent
Fujiko Spider (fujiko@inelegant.org)WordPress
Example user agent
WordPress 4.0-alpha Feed Client Mozilla/5.0 CompatibleEx-Crawler
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/v0.1.5ALPHA SVN rev58; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)MixBot
Example user agent
MixBot (+http://t.co/GSRLLKex24)BlogRangerCrawler
Example user agent
BlogRangerCrawler/1.0Ex-Crawler
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/0.1.5a; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)mozDex
Example user agent
mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)boitho.com-dc
Example user agent
boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )OpenindexSpider
Example user agent
Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)boitho.com-dc
Example user agent
boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html )archive.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)GoGuidesBot
Example user agent
Mozilla/5.0 (compatible; GoGuidesBot; http://www.goguides.org/spider.html)Jomjaibot
Example user agent
Jomjaibot/1.0 Crawl (+http://www.jomjaibot.com/)omgilibot
Example user agent
omgilibot/0.3 +http://www.omgili.com/Crawler.htmlcrawl
Example user agent
Mozilla/5.0 (compatible; worldwebheritage.org/1.0; +crawl@worldwebheritage.org)semanticdiscovery
Example user agent
semanticdiscovery/0.2(http://www.semanticdiscovery.com/sd/robot.html)AppCodes crawler
Example user agent
AppCodes crawler - looking for iOS app mentions. More info: support@appcodes.com. Robots.txt id: AppCodesCrawlerLogict IPv6 Crawler
Example user agent
Logict IPv6 Crawler/1.0 (http://ipv6search.logict.net),gzip(gfe),gzip(gfe)Scrubby
Example user agent
Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)fastlwspider
Example user agent
fastlwspider/1.0PhoenixWebBot
Example user agent
PhoenixWebBot Betaca-crawler
Example user agent
Mozilla/5.0 (compatible; ca-crawler/1.0)larbin
Example user agent
08_800w_web1 (larbin2.6.3@unspecified.mail)bingbot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)MJ12bot
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)MojeekBot
Example user agent
Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)449 Overture-WebCrawler
Example user agent
449 Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.aspNutch
Example user agent
ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com)ichiro
Example user agent
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)SCrawlTest
Example user agent
SCrawlTest/CR1 (CWD; http://sproose.com; crawl@sproose.com)NimbleCrawler
Example user agent
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.comBaiduspider-image
Example user agent
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)ObjectsSearch
Example user agent
ObjectsSearch/0.2 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)pagefreezer crawler
Example user agent
PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)BOTW Spider
Example user agent
Mozilla/4.0 (compatible; BOTW Spider; http://botw.org)MedSpider
Example user agent
MedSpider v0.0.1Rakutenbot
Example user agent
Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)AppEngine-Google
Example user agent
AppEngine-Google; ( http://code.google.com/appengine)kizasi-spider
Example user agent
kizasi-spider/1.0 (+http://kizasi.jp/)BeijingCrawler
Example user agent
BeijingCrawlerYahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0 ; http://help.yahoo.com/help/us/ysearch/slurp)SandCrawler
Example user agent
SandCrawler - Compatibility TestingIWE Spider
Example user agent
IWE Spider v.01 - www.pavka.com.auPagesInventory
Example user agent
PagesInventory (robot http://www.pagesinvenotry.com)CrawlerProcess
Example user agent
CrawlerProcess (http://www.PowerMapper.com) /5.5.678.0BejiBot
Example user agent
BejiBot Crawler (BNL Services; http://www.bejjan.net/crawler/)Nutch
Example user agent
Your Nutch Spider/Nutch-2.2.1Yeti
Example user agent
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)woobot
Example user agent
woobot/2.0msnbot
Example user agent
msnbot/2.0b ( http://search.msn.com/msnbot.htm)BuildCMS crawler
Example user agent
BuildCMS crawler (http://www.buildcms.com/crawler)Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2,01X-Crawler
Example user agent
X-CrawlerGooglebot
Example user agent
Locubot (compatible; Googlebot; msnbot)GrowerIdeas Crawler
Example user agent
GrowerIdeas Crawler/GrowerIdeas-nutch-1.6 (Crawls URLs for indexing content for our new search startup which aims to provide simple and smart search across curated content. For more info please contact helloworld@growerideas.com. If you thinkDealGates Bot
Example user agent
DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)Python-urllib
Example user agent
IMVU Client/498.0 Python-urllib/2.7WebAlta Crawler
Example user agent
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2.40ThumbShotsBot
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010091808 ThumbShotsBot (KFSW 3.0.6-3)AstraSpider
Example user agent
Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)ScreenerBot
Example user agent
ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)Yahoo-VerticalCrawler-FormerWebCrawler
Example user agent
Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawlerdtSearchSpider
Example user agent
dtSearchSpideracedo crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0; acedo crawler extension)BaiduMobaider
Example user agent
DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0; http://www.baidu.jp/spider/)webscraper
Example user agent
webscraper/1.0SeznamBot
Example user agent
SeznamBot/3.0 (+http://fulltext.sblog.cz/)Nutch
Example user agent
NutchSpider/Nutch-1.4Patwebbot
Example user agent
Patwebbot (http://www.herz-power.de/technik.html)CrawlBot
Example user agent
Mozilla/5.0 (compatible; CrawlBot/1.0.0)rating crawler
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) Russian CMS rating crawler (itrack.ru/cmsrate, avlasov@itrack.ru)LOOQ
Example user agent
LOOQ/0.1 alfa (LOOQ Crawler for european sites; http://looq.eu; root (at) looq dot eu)- Example user agent
Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk Squrl Java
Example user agent
Squrl Java/1.6.0_22online-webceo-bot
Example user agent
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)Googlebot
Example user agent
Googlebot (gocrawl v0.4)Spider_Monkey
Example user agent
Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)Googlebot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)ezooms.bot
Example user agent
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)ichiro/mobile
Example user agent
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)ByWebSite-Search-Spider
Example user agent
ByWebSite-Search-Spider,gzip(gfe),gzip(gfe)wminer
Example user agent
wminer/1.2 (North America Web Crawler; http://www.wminer.com/bot; info at wminer dot com)bendercrawler
Example user agent
Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler)Daumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2.22haupia-crawler
Example user agent
haupia-crawlerWSDL Crawler
Example user agent
WSDL Crawlersemantics webbot
Example user agent
semantics webbot/1.0WASALive Bot
Example user agent
Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/)AcquiaCrawler
Example user agent
AcquiaCrawler,gzip(gfe),gzip(gfe)Accoona-AI-Agent
Example user agent
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)Exabot
Example user agent
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)YandexBot
Example user agent
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)archive.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)Yahoo-MMCrawler
Example user agent
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)Java
Example user agent
Java/1.7.0_71altavista
Example user agent
AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.netFAST-WebCrawler
Example user agent
FAST-WebCrawler/2.2.11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)WIRE
Example user agent
WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler)EventGuruBot
Example user agent
Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)ichiro
Example user agent
ichiro/1.0 (ichiro@nttr.co.jp)Sogou Pic Spider
Example user agent
Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)bitlybot
Example user agent
bitlybotWebcrawler
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot WebcrawlerFTRF: Friendly robot
Example user agent
FTRF: Friendly robot/1.3Baiduspider
Example user agent
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)None-Crawler
Example user agent
Mozilla/5.0 (compatible; .None-Crawler/0.1 +http://domains.ericbinek.None/)ScSpider
Example user agent
ScSpider/0.2rogerbot
Example user agent
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)YandexImageResizer
Example user agent
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)SeznamBot
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)BotSeer
Example user agent
Mozilla 4.0(compatible; BotSeer/1.0; http://botseer.ist.psu.edu)msnbot-media
Example user agent
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)DomainCrawler
Example user agent
DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/botiaskspider2
Example user agent
iaskspider2 (iask@staff.sina.com.cn)SputnikImageBot
Example user agent
Mozilla/5.0 (compatible; SputnikImageBot/2.2)MonoBot
Example user agent
Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)OMGCrawler
Example user agent
OMGCrawler 1.0FacebookBot
Example user agent
(Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.0URLAppendBot
Example user agent
Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)Python-urllib
Example user agent
Python-urllib/1.15PluckItCrawler
Example user agent
PluckItCrawler/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com;)tCrawler
Example user agent
tCrawler/0.1,gzip(gfe),gzip(gfe),gzip(gfe)Sogou head spider
Example user agent
Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)TAMU_CS_IRL_CRAWLER
Example user agent
TAMU_CS_IRL_CRAWLER/1.0grub-client
Example user agent
grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)Synthesio Crawler
Example user agent
Synthesio Crawler release MonaLisa (contact at synthesio dot fr)Server Crawler
Example user agent
ownCloud Server CrawlerWebVulnCrawl
Example user agent
WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803Runet-Research-Crawler
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru)BlogramCrawler
Example user agent
BlogramCrawler/1.0.1(+http://blogram.jp/)envolk
Example user agent
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)diffbot
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15semanticdiscovery
Example user agent
semanticdiscovery/2.0(http://www.semanticdiscovery.com/robot.html)Slackbot
Example user agent
Slackbot (+https://api.slack.com/robots)FunkyCrawler
Example user agent
Mozilla/5.0 (compatible; FunkyCrawler; )Pete-Spider
Example user agent
Pete-Spider/1.1Finderlein Research Crawler
Example user agent
Finderlein Research Crawler 1.0Googlebot
Example user agent
Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)Baiduspider
Example user agent
'Mozilla/5.0 (compatible; Baiduspider/2.0; +ht'Scrapy
Example user agent
Scrapy/0.16.4 (+http://scrapy.org)IconSurf
Example user agent
IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)TurnitinBot
Example user agent
TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)JNLP
Example user agent
JNLP/6.0 javaws/1.6.0_14 (b08) Java/1.6.0_14noxtrumbot
Example user agent
noxtrumbot/1.0 (crawler@noxtrum.com)uk_lddc_bot
Example user agent
bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)Crawler
Example user agent
Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)NetWhatCrawler
Example user agent
NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com)archiver
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)Host-Spy Crawler
Example user agent
Mozilla/5.0 (compatible; Host-Spy Crawler/1.2; +http://www.host-spy.com/)InternetArchive
Example user agent
InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)SoftListBot
Example user agent
Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)bingbot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0 http://www.bing.com/bingbot.htm)ScollSpider
Example user agent
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)YioopBot
Example user agent
Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)com/bot
Example user agent
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)WebThumb
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)psbot
Example user agent
psbot-page (+http://www.picsearch.com/bot.html)ichiro/mobile
Example user agent
Mozilla/5.0 (Linux; U; Android 1.6; ja-jp; SonyEricssonSO-01B Build/R1EA018) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 (compatible; ichiro/mobile goo; http://help.goo.ne.jp/help/article/1142)Nutch
Example user agent
testnutch/Nutch-1.8redbot
Example user agent
RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)WordSurfer Spider
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; WordSurfer Spider 2.2;))TayaCrawler
Example user agent
TayaCrawler (Beta; v0.1; tayadev@nexdegree.com)Webspider
Example user agent
Webspider/1.0 (web spider; )SMXCrawler
Example user agent
SMXCrawler/1.0 (www.socialmetrix.com)YATS crawler
Example user agent
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)DefaultCrawlTest
Example user agent
DefaultCrawlTest/0.6 (Ram Crawl Test; devarajaswami at yahoo dot com)iaskspider
Example user agent
Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/na-Webcrawler
Example user agent
na-Webcrawler (helpdesk@newsaktuell.de)uk_lddc_bot
Example user agent
Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)AppEngine-Google
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxyfile1-hrd)Nutch
Example user agent
Spider/Nutch-2.3-SNAPSHOT (Webcrawler)NalezenCzBot
Example user agent
NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)Nutch
Example user agent
spider/Nutch-1.5.1 (spider; http://www.xxx.com)flatlandbot
Example user agent
flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)Wotbox
Example user agent
Wotbox/2.01 (+http://www.wotbox.com/bot/)FAST-WebCrawler
Example user agent
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)YebolBot
Example user agent
YebolBot (Email: yebolbot@gmail.com; If the web crawling affects your web service, or you don't like to be crawled by us, please email us. We'll stop crawling immediately.)DotBot
Example user agent
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)CowCrawler
Example user agent
CowCrawler/CowCrawler-dev (+http://beta.cow.com),gzip(gfe),gzip(gfe)Sitespider
Example user agent
Mozilla/5.5 (compatible; MSIE 5.5; Windows NT 5.1) Sitespider+ b432.1NutchOSUOSL
Example user agent
NutchOSUOSL/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)Trends Crawler
Example user agent
Trends Crawler, Real time trends bot (info@trendscrawler.com)Speedy Spider
Example user agent
Speedy Spider (http://www.entireweb.com)RSScrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; RSScrawler/4.0 (compatible; MSIE 6.0; Windows NT 5.0); .NET CLR 2.0.50727SEOENGWorldBot
Example user agent
SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)WebIndexer
Example user agent
WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com)socialbm_bot
Example user agent
Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)bandscraper
Example user agent
bandscraperbingbot
Example user agent
Mozilla/5.0 (compatible;bingbot/2.0;+http://www.bing.com/bingbot.htm)RSSIncludeBot
Example user agent
RSSIncludeBot/1.0 (http://www.rssinclude.com/spider)hivaBot
Example user agent
hivaBot(http://gorgor.ir and info@gorgor.ir)/hivaBot-0.2BuiBui-Bot
Example user agent
BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com)Python-urllib
Example user agent
Python-urllib/3.4hivaBot
Example user agent
hivaBot/hivaBot-0.2 (info@gorgor.ir)seekbot
Example user agent
Mozilla/4.0 (compatible; focuseekbot)EnaBot
Example user agent
EnaBot/1.2 (http://www.enaball.com/crawler.html)- Example user agent
Curious George - www.analyticsseo.com/crawler FreeCrawl
Example user agent
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (Atsameip FreeCrawl!)msnbot
Example user agent
msnbot/2.0b (+http://search.msn.com/msnbot.htm)Sangfor Spider
Example user agent
Sangfor Spider/0.7- Example user agent
Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/) fastbot
Example user agent
fastbot crawler beta 4.0 (+http://www.fastbot.de)TTop-Crawler
Example user agent
TTop-Crawler,gzip(gfe),gzip(gfe)SetLinks bot
Example user agent
Mozilla/5.0 (compatible; SetLinks bot 1.2)Reaper
Example user agent
Reaper/2.07 ( http://www.sitesearch.ca/reaper)- Example user agent
Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html) Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)JobRoboter
Example user agent
JobRoboter/1.1MsnBot
Example user agent
\x22Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b\x22360Spider
Example user agent
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360SpiderLNSpiderguy
Example user agent
LNSpiderguyCrawlFire
Example user agent
CrawlFire - you can disable this Robot: http://pastebin.de/25277MsnBot
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0bUnisterBot
Example user agent
Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)EtaoSpider
Example user agent
Mozilla/5.0 (compatible; EtaoSpider/1.0; http://open.etao.com/dev/EtaoSpider)DBot
Example user agent
Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com)MSIndianWebcrawl
Example user agent
MSIndianWebcrawlSeekOn Spider
Example user agent
SeekOn Spider 1.9(+http://www.seekon.com/spider.html)io bot
Example user agent
Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/)OrangeBot
Example user agent
Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com)OpenX Spider
Example user agent
Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org)WEBB Crawler
Example user agent
WEBB Crawler - see: http://badcheese.com/robots.html- Example user agent
A6-Indexer NewsGator
Example user agent
NewsGator/3.0,gzip(gfe),gzip(gfe)WNMbot
Example user agent
WNMbot/0.4.0Fluffy the spider
Example user agent
Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)Cityreview Robot
Example user agent
Cityreview Robot (+http://www.cityreview.org/crawler/)lycos
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; LYCOSA;+http://lycosa.se)pythonic-crawler
Example user agent
pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp)discobot
Example user agent
Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)FAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/5.3.4 (crawler@fast.no)Genieo
Example user agent
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)YRSpider
Example user agent
Mozilla/5.0 (Android 3.0; YRSpider; +http://www.yunrang.com/yrspider.html)Superfeedr bot
Example user agent
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touchMicrosoft MSN SocialStreams Bot
Example user agent
Microsoft MSN SocialStreams Bot,gzip(gfe),gzip(gfe)GooglePlusBot
Example user agent
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot/0.97sa; +http://www.semrush.com/bot.html)BaiduImagespider
Example user agent
BaiduImagespider ( http://help.baidu.jp/system/05.html)Gigabot
Example user agent
Gigabot/2.0/gigablast.com/spider.htmlcrawl
Example user agent
Mozilla/5.0 (compatible; publiclibraryarchive.org/1.0; +crawl@publiclibraryarchive.org)SiteCat Webbot
Example user agent
CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)Googlebot-Mobile
Example user agent
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Y!J-BRW
Example user agent
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)jikespider
Example user agent
jikespider "Mozilla/5.0Sosospider
Example user agent
Sosospider+(+http://help.soso.com/webspider.htm)crawler4j
Example user agent
crawler4j (http://code.google.com/p/crawler4j/)Sangfor Spider
Example user agent
Sangfor SpiderNutch
Example user agent
CB/Nutch-1.7Microsoft Bing Mobile SocialStreams Bot
Example user agent
Microsoft Bing Mobile SocialStreams BotAdsBot-Google
Example user agent
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile SafariURLfilterDB-crawler
Example user agent
Mozilla/5.0 (compatible; Linux i686; en-US; URLfilterDB-crawler/1.0) ufdb/1.0New-Sogou-Spider
Example user agent
New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98)robot
Example user agent
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html)Lite Bot
Example user agent
Lite Bot0316Bworiobot
Example user agent
Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)Kyoto-Crawler
Example user agent
Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)Nutch
Example user agent
MSNBot/Nutch-1.5.1Googlebot
Example user agent
Googlebot v2.1 (+http://www.google.com/bot.html) (http://www.openwebspider.org/)nyobot
Example user agent
nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com),gzip(gfe),gzip(gfe)tivraSpider
Example user agent
tivraSpider/1.0 (crawler@tivra.com)PicSpider
Example user agent
PicSpider/1.1 (spider@217-20-118-26.internetserviceteam.com; http://www.bildkiste.de)AhrefsBot
Example user agent
AhrefsBot.Feeds v0.1; http://ahrefs.com/- Example user agent
Mozilla/5.0 (linux; u; android 4.1.1; ja-jp; sgpt12 build/tjs0166) applewebkit/534.30 (khtml, like gecko) version/4.0 safari/534.30 yjapp-android jp.co.yahoo.android.yjtop/2.1.8 jrCrawler
Example user agent
jrCrawler/1.0bAppEngine-Google
Example user agent
AppEngine-Google; (+http://code.google.com/appengine; appid: xbmacx)JoobleBot
Example user agent
JoobleBot (compatible; http://jooble.org/jooble-bot)MSR-ISRCCrawler
Example user agent
MSR-ISRCCrawlerSiteCrawler
Example user agent
SiteCrawler,gzip(gfe),gzip(gfe)Nutch; http:
Example user agent
SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)Googlebot-News
Example user agent
Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))ZyBorg
Example user agent
Mozilla/4.0 (compatible; MSIE; ZyBorg; Win32)Shim-Crawler
Example user agent
Shim-CrawlerYandexMetrika
Example user agent
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)HiveCrawler
Example user agent
Mozilla/5.0 (compatible; HiveCrawler/1.2 http://www.businessinsider.com)MetaJobBot
Example user agent
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)BrightCrawler
Example user agent
BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)Nutch
Example user agent
Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)nuSearch Spider
Example user agent
nuSearch Spider www.nusearch.com (compatible; MSIE 4.01; Windows NT)cis455crawler
Example user agent
cis455crawlerAsterias Crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.0ichiro
Example user agent
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)PipeLine Spider
Example user agent
pipeLiner/0.10 (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)WebAlta Crawler
Example user agent
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)vertical-crawl
Example user agent
Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com)IstellaBot
Example user agent
Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)- Example user agent
wsr-agent/1.0 memorybot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)Accoona-Biz-Agent
Example user agent
Accoona-Biz-Agent/1.1.1 crawler@accoona.comHooWWWer
Example user agent
HooWWWer/2.1.0 ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi)heritrix
Example user agent
flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)media-percbotspider
Example user agent
media-percbotspider Liquida Spider
Example user agent
Liquida Spider 1.0 +http://liquida.com/Jaxified Crawler
Example user agent
Jaxified Crawler 1.0a (+http://www.jaxified.com/),gzip(gfe),gzip(gfe)ICC-Crawler
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html; zurukko1552995316;)spider
Example user agent
Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org)ZooShot
Example user agent
ZooShot 0.42robots
Example user agent
robotsLinguee Bot
Example user agent
Linguee Bot (bot@linguee.com)Mediapartners-Google
Example user agent
DoCoMo/2.0 SH905i(c100;TB;W24H16) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)LargeSmall Crawler
Example user agent
LargeSmall CrawlerGooglebot-Mobile
Example user agent
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)suggybot
Example user agent
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)Argus
Example user agent
Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)Nutch
Example user agent
5acd734a-1ed3-4a17-ad9d-0b09ae7bc2d2/Nutch-1.7200PleaseBot
Example user agent
Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)agbot
Example user agent
agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com)Feed Crawler
Example user agent
www2.apserver.net ASP-Ranker Feed Crawlerlbot
Example user agent
Mozilla/5.0 (lbot/2.6.4) (lbot@mailme.not)Gazeta Bot
Example user agent
Mozilla/5.0 (compatible; Linux i686; Yandex.Gazeta Bot/1.0; +http://gazeta.yandex.ru)Webbot
Example user agent
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)- Example user agent
R6_FeedFetcher_(www.radian6.com/crawler) OrangeBot
Example user agent
Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)MetaWeb Crawler
Example user agent
FAST MetaWeb Crawler (helpdesk at fastsearch dot com)LinkStats Bot
Example user agent
LinkStats BotNLCrawler
Example user agent
Mozilla/5.0 (compatible; NLCrawler/2.0.15; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko)Web-Robot
Example user agent
Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3TinEye-bot
Example user agent
TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)spiderpig
Example user agent
spiderpig/0.1,gzip(gfe),gzip(gfe)rogerbot
Example user agent
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)ayna-crawler
Example user agent
Mozilla/5.0 (compatible; ayna-crawler http://www.ayna.com)Top10Ranking Spider
Example user agent
Top10Ranking Spider/3.1 ( http://www.top10Ranking.nl/, Top10ranking.nl heeft op een aantal woorden uw posities in Google gecheckt)rogerBot
Example user agent
Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)CareerBot
Example user agent
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)SlugBug Spider
Example user agent
SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com)TweetmemeBot
Example user agent
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0CazoodleBot
Example user agent
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com)- Example user agent
Bot.AraTurka.com/0.0.1 Inspyder-Crawler
Example user agent
Mozilla/4.0 (compatible; Inspyder-Crawler; http://www.inspyder.com)Slurp
Example user agent
Mozilla/3.0 (Slurp.so/1.0; slurp@inktomi.com; http://www.inktomi.com/slurp.html)YandexImages
Example user agent
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)lb-spider
Example user agent
lb-spiderHPI FeedCrawler
Example user agent
HPI FeedCrawler/0.1 (+http://www.hpi.uni-potsdam.de/meinel/bross/feedcrawler)Riddler
Example user agent
Riddler (http://riddler.io/about.html)Help:Bot
Example user agent
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)findlinks
Example user agent
findlinks/2.6 ( http://wortschatz.uni-leipzig.de/findlinks/)Daumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0Brekiri crawler
Example user agent
Brekiri crawler/1.0RSS Crawler
Example user agent
Jaxified Public RSS Crawler ( http://www.jaxified.com/)INGRID
Example user agent
INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)EcoGrader Crawler
Example user agent
EcoGrader Crawler: BetaSeznamBot
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)YFSJ crawler
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248)- Example user agent
Java/1.17 PArchiveCrawler
Example user agent
PArchiveCrawler/1.0- Example user agent
Java/1.7 Yeti
Example user agent
Yeti-FeedItemCrawler/1.0 (NHN Corp.;+http://help.naver.com/robots/)Google-HTTP-Java-Client
Example user agent
Google-HTTP-Java-Client/1.17.0-rc (gzip)WordPress
Example user agent
Pachacutec/0.5 (qpImageBuilders.com); WordPress/3.1; http://contacto-latino.com/newsdCrawlBot
Example user agent
dCrawlBot/1.0.1120FAST-WebCrawler
Example user agent
Opera/30.0.1835.59 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/29.16 AppleWebKit/599.0+ (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 FAST-WebCrawler/3.8Searchspider
Example user agent
Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com)masidani_bot
Example user agent
masidani_bot_v0.3 (info@masidani.com)ia_archiver
Example user agent
ia_archiver/8.7 (Windows NT 5.0; )IC OpenGraph Crawler
Example user agent
IC OpenGraph Crawler 4.5 (proprietary)msrbot
Example user agent
MSRBOT (http://research.microsoft.com/research/sv/msrbot)Daumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0Thesis_php_crawler
Example user agent
Thesis_php_crawlerVagabondo
Example user agent
Mozilla/5.0 (compatible; Vagabondo/Wapspider; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)Medialab Spider
Example user agent
Mozilla/4.0 (compatible); MSIE 5.0; Medialab Spiderrogerbot
Example user agent
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)Slackbot
Example user agent
Slackbot-LinkExpanding (+https://api.slack.com/robots)Mail.RU_Bot
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)BacklinkCrawler
Example user agent
BacklinkCrawler (http://www.backlinktest.com/crawler.html)AOLbot
Example user agent
Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/)BarraHomeCrawler
Example user agent
BarraHomeCrawler (albertof@barrahome.org)favicon_archive
Example user agent
FaviconArchiver/1.0 (+http://moblur.org/workshop/favicon_archive/)YodaoBot
Example user agent
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )Apexoo Spider
Example user agent
Apexoo Spider (http://www.apexoo.com/spider/)es-spider
Example user agent
Enterprise_Search/1.0 110 (http://www.innerprise.net/es-spider.asp)FacebookBot
Example user agent
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)Sosoimagespider
Example user agent
Mozilla/5.0 (compatible; Sosoimagespider/2.0; +http://help.soso.com/soso-image-spider.htm)CSimpleSpider
Example user agent
CSimpleSpider RobotyoozBot
Example user agent
Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)nbot
Example user agent
Mozilla/5.0 (compatible; nbot/2.0)BabalooSpider
Example user agent
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)Crawler
Example user agent
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)YRSpider
Example user agent
Mozilla/5.0 (compatible; YRSpider; +http://www.yunyun.com/SiteInfo.php?r=about)Sogou web spider
Example user agent
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)SNK Screenshot Bot
Example user agent
SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com )ZumBot
Example user agent
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)mapion-news-bot
Example user agent
"mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)"Feed Seeker Bot
Example user agent
Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php)PagePeeker
Example user agent
PagePeeker.comoBot
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)heritrix
Example user agent
Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters)NutchOrg
Example user agent
NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)Comodo Spider
Example user agent
Comodo Spider 1.2OpenWebSpider
Example user agent
OpenWebSpider/0.6 (http://www.openwebspider.org)Kurzor
Example user agent
Kurzor/1.0 (Kurzor; http://adcenter.hu/docs/en/bot.html; cursor@easymail.hu)Lipperhey Spider
Example user agent
Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)NewsGator FetchLinks extension
Example user agent
NewsGator FetchLinks extension/0.2.0 (http://graemef.com)SearchSpider
Example user agent
SearchSpider/1.2.10Noobot
Example user agent
noobot Spider/Noobot-1.2 (noobot-bot; http://www.noobot.fr)Kraken
Example user agent
Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)Detector Crawler
Example user agent
StatusCake Error Detector Crawler V0.01seebot
Example user agent
seebot/2.0 (+http://www.seegnify.com/bot)GrapeshotCrawler
Example user agent
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)bingbot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)BlinkaCrawler
Example user agent
Mozilla/5.0 (compatible; BlinkaCrawler/1.0; http://www.blinka.jp/crawler/)Hailoobot
Example user agent
Mozilla/5.0 (compatible; Hailoobot/1.2; http://www.hailoo.com/spider.html)CloudServerMarketSpider
Example user agent
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)360Spider
Example user agent
360SpiderJob Roboter
Example user agent
Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)echocrawl
Example user agent
echocrawl 2.0URLfilterDB-crawler
Example user agent
Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0MPICrawler
Example user agent
MPICrawler/0.1 +http://kermit.news.cs.nyu.edu/crawler.html,gzip(gfe),gzip(gfe)008
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620EverbeeCrawler
Example user agent
EverbeeCrawler- Example user agent
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/) heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.run4dom.com)Yowedo
Example user agent
YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com)EuripBot
Example user agent
EuripBot/0.5 (+http://www.eurip.com) PreCheckMail.RU_Bot
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)fetchurl
Example user agent
SAE/fetchurl-22wy0njxnm WordPress/3.2.1; http://1.webfront.sinaapp.comExactSeekCrawler
Example user agent
ExactSeekCrawler/1.08bo Crawler Bot
Example user agent
8bo Crawler BotoBot
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)SuperSpider
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; SuperSpider/139; Windows 98; Win 9x 4.90)com robot
Example user agent
AddThis.com robot tech.support@clearspring.comYoudaoBot
Example user agent
Mozilla/5.0 (compatible; YoudaoBot-rts/1.0; http://www.youdao.com/help/webmaster/spider/; )BIXOCRAWLER
Example user agent
Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)Mediapartners-Google
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)LinkAider
Example user agent
LinkAider (http://linkaider.com/crawler/)Scrapy
Example user agent
Scrapy/0.24.6 (+http://scrapy.org)EDI
Example user agent
EDI/1.6.0 (Edacious & Intelligent Web Crawler)htdig
Example user agent
htdig/3.1.6 (romieu@bastide-medical.fr)FeedlyBot
Example user agent
FeedlyBot/1.0 (http://feedly.com)Superfeedr bot
Example user agent
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!OmniExplorer_Bot
Example user agent
OmniExplorer_Bot/6.70 ( http://www.omni-explorer.com) WorldIndexerExabot
Example user agent
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)CazoodleBot
Example user agent
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)ConveraMultiMediaCrawler
Example user agent
ConveraMultiMediaCrawler/0.1 ( http://www.authoritativeweb.com/crawl)Aboundex
Example user agent
Aboundex/0.2 (http://www.aboundex.com/crawler/)CRAZYWEBCRAWLER
Example user agent
CRAZYWEBCRAWLER 0.9.0, http://www.crazywebcrawler.comCliqzbot
Example user agent
Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)CCBot
Example user agent
CCBot/1.0 ( http://www.commoncrawl.org/bot.html)Googlebot-Mobile
Example user agent
Googlebot-Mobile (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)BurstFindCrawler
Example user agent
BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)Xerka WebBot
Example user agent
Xerka WebBot v1.0.0 [AIDO_CREA_ab]Python-urllib
Example user agent
Python-urllib/1.16Zspider
Example user agent
Zspider (+http://www.zhanzhangsou.com/index.htm)ysm-keystone-crawl
Example user agent
Mozilla/5.0 (Yahoo-Test/4.0 ysm-keystone-crawl-support@yahoo-inc.com)YandexDirect
Example user agent
Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)ichiro/mobile
Example user agent
DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo; http://help.goo.ne.jp/door/crawler.html)SEOkicks-Robot
Example user agent
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)LarbinWebCrawler
Example user agent
LarbinWebCrawler (internet@bredband.net)Xenu Link Sleuth
Example user agent
Xenu Link Sleuth/1.3.8archive.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)deepcrawler
Example user agent
deepcrawler/3.1 (http://www.queusearch.com/whatis_deepcrawler.php),gzip(gfe),gzip(gfe)CMS Crawler
Example user agent
Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)voyager
Example user agent
voyager/2.0 (http://www.kosmix.com/crawler.html)Scrapy
Example user agent
Scrapy/0.22.2 (+http://scrapy.org)heritrix
Example user agent
mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)Nutch
Example user agent
Mozilla/5.0 (compatible; summers;)/Nutch-1.7ia_archiver
Example user agent
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)GeekTools Crawler
Example user agent
GeekTools Crawler - http://domains.geek-tools.orgMozilla crawl
Example user agent
Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc)A6-Indexer
Example user agent
A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)Media Crawler
Example user agent
Social Media Crawler using your Home URL on Twitter,Facebook,Myspace,Linkedin by ProfileCapture - contact profilecapture@gmail.com to report any problems with my crawling. http://profilecapture.comYeti
Example user agent
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)msnbot-media
Example user agent
msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)MerchantCentricBot
Example user agent
Mozilla/5.0 (compatible; MerchantCentricBot/1.0)persomm-spider
Example user agent
persomm-spider/v1.0MJ12bot
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)Wocodi Web Crawler
Example user agent
WocBot/Mozilla/5.0 (Wocodi Web Crawler 1.0; http://www.wocodi.com/crawler; crawler@wocodi.com)Netvibes
Example user agent
Netvibes (http://www.netvibes.com)NE Crawler
Example user agent
netEstate NE Crawler (+http://www.website-datenbank.de/)WhatWeb
Example user agent
WhatWeb/0.4.8-devmSEARCH Crawler
Example user agent
FAST-mSEARCH Crawler 0.1 (bergum@fast.no)SearchSpider
Example user agent
SearchSpider.com/1.1360Spider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360SpiderSpiderlytics
Example user agent
Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com)iisbot
Example user agent
iisbot/1.0 (+http://www.iis.net/iisbot.html)BoardReader Blog Indexer
Example user agent
BoardReader Blog Indexer(http://boardreader.com)Riddler
Example user agent
Riddler (http://riddler.io/about)Comodo HTTP(S) Crawler
Example user agent
Comodo HTTP(S) Crawler - http://www.instantssl.com/crawlerNutch
Example user agent
asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)Orbiter
Example user agent
Orbiter ( http://www.dailyorbit.com/bot.htm)LinksCrawler
Example user agent
LinksCrawler 0.1betalemurwebcrawler
Example user agent
Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/)Nutch
Example user agent
emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)TweetedTimes Bot
Example user agent
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)com WebBot
Example user agent
EmeraldShield.com WebBotJayde Crawler
Example user agent
Jayde Crawler. http://www.jayde.comStudent Spider
Example user agent
PKU Student Spideryacybot
Example user agent
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.htmlINGRID
Example user agent
INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)Sogou Pic Spider
Example user agent
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htmkototoi-crawl
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) compatible; kototoi-crawl@yl.is.s.u-tokyo.ac.jpKyluka crawl
Example user agent
Mozilla/5.0 (compatible; Kyluka crawl; crawl@kyluka.com; http://www.kyluka.com/static/crawl.html)WeSEE:Search
Example user agent
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)Socialradarbot
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com)Peew
Example user agent
Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)Mozilla-Firefox-Spider
Example user agent
Mozilla-Firefox-Spider(Wenanry)GIDBot
Example user agent
GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php)Theme Spider
Example user agent
Theme Spider ( http://www.themespider.com/spider.html)Slurp
Example user agent
Slurp/2.0PathDefender
Example user agent
Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler)Zao
Example user agent
Zao-CrawlerTwitmunin Crawler
Example user agent
Twitmunin Crawler http://www.twitmunin.comTasapspider
Example user agent
Tasapspider/0.9CrawlerBoy
Example user agent
CrawlerBoy Pinpoint.comWordPress
Example user agent
Jetpack by WordPress.comheritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dla-marbach.de)FacebookBot
Example user agent
(Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.4.5.3000 Firefox/38SimpleCrawler
Example user agent
SimpleCrawler/0.1AppEngine-Google
Example user agent
AppEngine-GooglememoryBot
Example user agent
Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)ContextAd Bot
Example user agent
ContextAd Bot 1.0redbot
Example user agent
RED/1 (http://redbot.org/)WebCrawler
Example user agent
WebCrawler v1.3DomainSigmaCrawler
Example user agent
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)Sogou-Test-Spider
Example user agent
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)Googlebot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ASProxy/5.5b5Norbert the Spider
Example user agent
Norbert the Spider(Burf.com)GarlikCrawler
Example user agent
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)SAS Crawler
Example user agent
Teragram/SAS CrawlerUCMore Crawler
Example user agent
UCMore Crawler AppScollSpider
Example user agent
ScollSpider/2.0 ( http://www.webwobot.com/ScollSpider.php)robot
Example user agent
Mozilla/5.0 (compatible; DomainSCAN/2.0; +http://domainscan.info/robot/)Dazoobot
Example user agent
Mozilla/5.0 (compatible; Dazoobot/0.1; +http://dazoo.fr)TECOMAC-Crawler
Example user agent
TECOMAC-Crawler/0.3Nutch
Example user agent
DiscoverEd/Nutch-1.7 (OER search crawler; http://wiki.creativecommons.org/DiscoverEd; webmaster@creativecommons.org)1470.net crawler
Example user agent
1470.net crawlerAISearchBot
Example user agent
AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.)FANGCrawl
Example user agent
FANGCrawl/0.01PiplBot
Example user agent
Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)- Example user agent
Superarama.com - BOT/v.0.1 scooter
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; scooter; .NET CLR 1.0.3705)PHPCrawl
Example user agent
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 3.5.21022; .NET CLR 3.5.30729; MS-RTC LM 8; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; .NET CLR 3.0.30729) Jakarta Commons-HttpClient/3.0-rc3 PHPCrawl GStreamer souphttpsrc libsoup/2.27.4 PycURL/7.19.0 XML-RPC for PHP 2.2.1 GoogleFriendConnect/1.0 HTMLParser/1.6 gPodder/0.15.2 ( http://gpodder.org/) anw webtool LoadControl/1.3 WinHttp urlgrabber/3.1.0mixi-news-crawler
Example user agent
mixi-news-crawler/1.00 (http://mixi.jp/)Punk Spider
Example user agent
Punk Spider/PunkSPIDER-v0.1pic scraper
Example user agent
Mozilla / reddit pic scraper v0.8 (bklug@tyx.net)LinkedInBot
Example user agent
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)ru Screenshot Bot
Example user agent
pr-cy.ru Screenshot Botmsnbot
Example user agent
"msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)DBot
Example user agent
Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws)MetamojiCrawler
Example user agent
Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.htmlPeeplo Screenshot Bot
Example user agent
Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )VSE
Example user agent
VSE/1.0 (vsecrawler@hotmail.com)pub-crawler
Example user agent
Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)iaskspider
Example user agent
iaskspider/2.0( http://iask.com/help/help_index.html)uMBot
Example user agent
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)NutchCVS
Example user agent
NutchCVS/0.06-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)Y!J-BRI
Example user agent
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )YisouSpider
Example user agent
YisouSpiderWinHTTP
Example user agent
Asynchronous WinHTTP/1.0AcquiloSpider
Example user agent
AcquiloSpider/1.0,gzip(gfe),gzip(gfe)blog_crawler
Example user agent
blog_crawler/1.0IlTrovatore
Example user agent
IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)Scopia Crawler
Example user agent
Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)EasouSpider
Example user agent
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)SWEBot
Example user agent
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)ZeBot_www.ze.bz
Example user agent
ZeBot_www.ze.bz (ze.bz@hotmail.com)Googlebot
Example user agent
Googlebot/2.2aggregator VocusBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; aggregator VocusBot 0.4; +http://www.vocus.com/vnhs.html)WebZIP
Example user agent
WebZIP/7.0 (http://www.spidersoft.com)EasouSpider
Example user agent
EasouSpiderlworldspider
Example user agent
lworldspiderheritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.fi/eng/project.html)Altresium
Example user agent
Altresium/4.6 (+http://www.altresium.com/bot.html)Computer_and_Automation_Research_Institute_Crawler
Example user agent
Computer_and_Automation_Research_Institute_Crawler (crawler@info.ilab.sztaki.hu)T3census-Crawler
Example user agent
T3census-Crawler/1.0Flamingo_SearchEngine
Example user agent
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)BlogBridge
Example user agent
BlogBridge 6.7 (http://www.blogbridge.com/) 1.6.0_18FlaxCrawler
Example user agent
FlaxCrawler/1.0,gzip(gfe),gzip(gfe)Orgbybot
Example user agent
Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)Apexoo Spider
Example user agent
Apexoo Spider 1.1Googlebot-Mobile
Example user agent
DoCoMo//2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Lightningspider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2FastCrawler3
Example user agent
Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no)Crawly
Example user agent
Mozilla/5.0 (compatible; Crawly/1.9; http://92.51.162.40/crawler.html)ICC-Crawler
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)BaiDuSpider
Example user agent
BaiDuSpiderAutomattic Analytics Crawler
Example user agent
Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/SensikaBot
Example user agent
SensikaBot/x.33 (+http://sensika.com)VSE
Example user agent
VSE/1.0 (testcrawler@hotmail.com)Python-urllib
Example user agent
Mozilla/5.0 (Python-urllib2)MJ12bot
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)PaxleFramework
Example user agent
Mozilla/5.0 (compatible; PaxleFramework/0.1.1; http://www.paxle.net/en/bot)WebbCrawler
Example user agent
WebbCrawler 1.0 ( http://badcheese.com/crawler.html )Inar_spider2
Example user agent
Inar_spider2 (tspyyp@tom.com)Sogou Pic Spider
Example user agent
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Feedjit Favicon Crawler
Example user agent
Feedjit Favicon Crawler 1.0BLEXBot
Example user agent
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)Nutch
Example user agent
agent6/Nutch-1.1Daumoa-feedfetcher
Example user agent
Mozilla/5.0 (compatible; Firefox compatible; MS IE compatible; http://search.daum.net/) Daumoa-feedfetcher/2.0spider
Example user agent
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)TLSProber
Example user agent
TLSProber/0.8Nutch
Example user agent
your sipder name/Nutch-1.7Genieo
Example user agent
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.htmlZoomSpider
Example user agent
ZoomSpider - wrensoft.comYet-Another-Spider
Example user agent
Mozilla/4.0 (compatible; Yet-Another-Spider; )Adaxas Spider
Example user agent
Adaxas Spider (http://www.adaxas.net/)VisBot
Example user agent
VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)Webspider
Example user agent
RankFlex.com WebspiderCrawlerProcess
Example user agent
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.10.736.0Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/1.10Lijit Crawler
Example user agent
Lijit Crawler (+http://www.lijit.com/robot/crawler)WSDLSpider
Example user agent
WSDLSpider/1.0 (http://www.wsdlworld.com)BoardReader
Example user agent
BoardReader/1.0 (http://boardreader.com/info/robots.htm)-Machine1CrawlWave
Example user agent
CrawlWave/1.2 (crawlwave[at]circular.gr http://www.spiderwave.aueb.gr/parallelContextFocusCrawler1
Example user agent
parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.jp/spider/)YandexAntivirus
Example user agent
Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)VinjaVideoSpider
Example user agent
VinjaVideoSpider/1.1WebCompanyCrawler
Example user agent
WebCompanyCrawlerDigincore bot
Example user agent
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.stq_bot
Example user agent
stq_bot (+http://www.searchteq.de)ExB Language Crawler
Example user agent
ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)WordPress
Example user agent
WordPress/1.2.1 PHP/4.3.9-1magpie-crawler
Example user agent
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)ZumBot
Example user agent
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)YandexNews
Example user agent
Mozilla/5.0 (compatible; YandexNews/3.0; +http://yandex.com/bots)YFSJ crawler
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)Baiduspider-image
Example user agent
Baiduspider-image ( http://www.baidu.com/search/spider.htm)Baiduspider
Example user agent
Baiduspider ( http://www.baidu.com/search/spider.htm)StreamScraper
Example user agent
Mozilla/5.0 (compatible; StreamScraper/1.0; +http://code.google.com/p/streamscraper/)robot
Example user agent
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)Votay bot
Example user agent
Mozilla/5.0 (Votay bot/4.0; http://www.votay.com/arts/comics/)NodejsSpider
Example user agent
NodejsSpider/1.0zcspider
Example user agent
IS Alpha/zcspider-0.1OpenWebSpider
Example user agent
OpenWebSpider (link collector; http://links.port30.se/cia.html) v (http://www.openwebspider.org/)Sogou develop spider
Example user agent
Sogou develop spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)- Example user agent
R6_FeedFetcher(www.radian6.com/crawler) RSpider
Example user agent
RSpider/1.0Simpy
Example user agent
Simpy/1.1 (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)JobSpider_BA
Example user agent
JobSpider_BA/1.1008
Example user agent
Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620memorybot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html)spider
Example user agent
Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html)SeznamBot
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)Wireless Crawler
Example user agent
EricssonR320/R1A (Fast Wireless Crawler)MetaCrawler
Example user agent
DomainsDB.net MetaCrawler v.0.9.7b (http://domainsdb.net/)BLEXBot
Example user agent
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)PagesInventory
Example user agent
PagesInventory (robot http://www.pagesinventory.com)Googlebot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)amphetameme crawler
Example user agent
amphetameme crawler (crawler@amphetameme.com)Twitterbot
Example user agent
Twitterbot/1.0Crawler
Example user agent
Crawler 0.1,gzip(gfe),gzip(gfe)KaloogaBot
Example user agent
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)AboutUsBot
Example user agent
AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)furlbot
Example user agent
Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net)Internet Spider
Example user agent
Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)Iplexx Spider
Example user agent
Mozilla/4.0 (compatible; Iplexx Spider/1.1)DRKSpider
Example user agent
DRKSpider - Website link validator - http://www.drk.com.ar/spider/,gzip(gfe),gzip(gfe),gzip(gfe)js bot
Example user agent
Phantom.js botTurnitinBot
Example user agent
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)Blogshares Spiders
Example user agent
Blogshares Spiders (Renewed V1.65)ZillaCrawler
Example user agent
ZillaCrawlerSputnikBot
Example user agent
Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)Alexabot
Example user agent
Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)MultiCrawler
Example user agent
MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.htmlPaperLiBot
Example user agent
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)msnbot
Example user agent
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._LivelapBot
Example user agent
LivelapBot/0.2 (http://site.livelap.com/crawler)NationalDirectory-WebSpider
Example user agent
NationalDirectory-WebSpider/1.3BebopBot
Example user agent
BebopBot/2.5.1 (compatible; media crawler V1; http://www.apassion4jazz.net/bebopbot.html;)Autonomy Spider
Example user agent
Autonomy SpiderNuSearch Spider
Example user agent
NuSearch Spider www.nusearch.comdoc/botinfo
Example user agent
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)GingerCrawler
Example user agent
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)Ant-Nutch
Example user agent
Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)Analyzer Crawler
Example user agent
www.website-analyzer.net Website Analyzer CrawlerTweetedTimes Bot
Example user agent
TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, +http://tweetedtimes.com)TheUsefulbot
Example user agent
TheUsefulbot_2.3.62 (bot@theusefulnet.com)lmspider
Example user agent
lmspider (lmspider@scansoft.com)Web Crawler
Example user agent
Arachnida Web Crawlersgbot
Example user agent
Mozilla/5.0 (compatible; sgbot v0.01a, sgcrawlerbot@gmail.com)Spider
Example user agent
Spider/5.0Isara
Example user agent
Isara-Search/Isara-1.0 (A non-profit web crawler operated by a charity organization.; www.isara.org; webmaster@isara.org)Ask Jeeves
Example user agent
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)Java
Example user agent
Java/1.5.0_17Cliqzbot
Example user agent
Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)Steeler
Example user agent
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)NewzCrawler
Example user agent
NewzCrawler/1.9 (compatible; MSIE 6.00; Newz Crawler 1.9; http://www.newzcrawler.com/ )ProjectWF-java-test-crawler
Example user agent
ProjectWF-java-test-crawlerTestCrawler
Example user agent
Mozilla/5.0 (compatible; TestCrawler)bingbot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0; )"Scrapy
Example user agent
Scrapy/0.20.1 (+http://scrapy.org)SpiderView
Example user agent
Mozilla/4.0 (compatible; SpiderView 1.0;unix)Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo!; U; Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 ()Googlebot
Example user agent
Mozilla/5.0 AppleWebKit/537.4 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.4Nutch
Example user agent
Googlebot/Nutch-1.7YoonoCrawler
Example user agent
YoonoCrawler/1.0 (crawler@yoono.com)Zeusbot
Example user agent
Zeusbot/0.8.1 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)loc-crawler
Example user agent
Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://loc.gov),gzip(gfe),gzip(gfe),gzip(gfe)holmes
Example user agent
holmes/2.3- Example user agent
YahooCacheSystem StoneSunSpider
Example user agent
Mozilla/5.0 (compatible; StoneSunSpider/1.1)PagesInventory
Example user agent
PagesInventory (robot +http://www.pagesinventory.com)VoilaBot
Example user agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)Netseer crawler
Example user agent
compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.comXSpider
Example user agent
XSpider 0.01;http://uncool.oicp.net/spider.htmlRavenCrawler
Example user agent
RavenCrawlerGoogle-HTTP-Java-Client
Example user agent
Google-HTTP-Java-Client/1.13.1-beta (gzip)exooba crawler
Example user agent
exooba/exooba crawler (exooba; exooba)Hatena::Crawler
Example user agent
Hatena::Crawler/0.01Arikus_Spider
Example user agent
Arikus_SpiderOrangebot
Example user agent
Mozilla/5.0 (compatible; Orangebot/2.0; support.orangebot@orange.com)OpenHoseBot
Example user agent
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)MixrankBot
Example user agent
Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)meanpathbot
Example user agent
Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)uMBot
Example user agent
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)- Example user agent
mbot v.1.16 SMTBot
Example user agent
Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)Lycos_Spider_
Example user agent
Lycos_Spider_(modspider)- Example user agent
CiBra Data Collector (http://cibra.de/) KomodiaBot
Example user agent
Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0SMTBot
Example user agent
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)ExactSeek_Spider
Example user agent
ExactSeek_SpiderFAST EnterpriseCrawler
Example user agent
FAST EnterpriseCrawler 6LEIA
Example user agent
LEIA/3.01pr (LEIAcrawler; leia@gseek.com; http://www.gseek.com)360Spider
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)crawl
Example user agent
crawl/0.4 langcrawl/0.1adbeat_bot
Example user agent
adbeat_botTweetmemeBot
Example user agent
Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)TouTrix crawler
Example user agent
TouTrix crawler/1.03 Robot
Example user agent
Abrave Spider v5.3 Robot 2 (http://robot.abrave.com)bingbot
Example user agent
bingbotMetadata_Scraper
Example user agent
CC Metadata Scaper http://wiki.creativecommons.org/Metadata_ScraperNetvibes
Example user agent
Netvibes (http://www.netvibes.com/; 8 subscribers; feedID: 2244192)Nutch
Example user agent
My Nutch Spider/Nutch-1.6nlcrawler
Example user agent
nlcrawler/1.0 (+http://northernlight.com/)microdata crawler
Example user agent
SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)Comodo-Webinspector-Crawler
Example user agent
Comodo-Webinspector-Crawler 2.1Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)niki-bot
Example user agent
niki-botGoogle-HTTP-Java-Client
Example user agent
Google-YouTubeSample/1.0 Google-HTTP-Java-Client/1.7.0-beta (gzip) Google-HTTP-Java-Client/1.7.0-beta (gzip)ShopSalad
Example user agent
SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT shopsalad.com)- Example user agent
Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com) - Example user agent
Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/) psbot
Example user agent
psbot/0.1 (+http://www.picsearch.com/bot.html)bingbot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.1; +http://www.bing.com/bingbot.htm)Java
Example user agent
Java/1.6.0_43RyzeCrawler
Example user agent
RyzeCrawler/1.1.0 (+http://www.ryze.nl/crawler/)linkdexbot
Example user agent
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)Xaldon WebSpider
Example user agent
Xaldon WebSpider 2.7.b6Googlebot-Mobile
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Mail.RU_Bot
Example user agent
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)IRLbot
Example user agent
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)psbot
Example user agent
psbot-image (+http://www.picsearch.com/bot.html)IRLbot
Example user agent
IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler)i1searchbot
Example user agent
i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)MojeekBot
Example user agent
Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)SpiderKU
Example user agent
SpiderKU/0.9ichiro/mobile
Example user agent
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)clumboot
Example user agent
Mozilla/5.0 (compatible; clumboot; +http://localhost/clumpit/bot.php)YRL_ODP_CRAWLER
Example user agent
YRL_ODP_CRAWLERFAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/6 (www.fastsearch.com)Research Crawler
Example user agent
LinkStar Research Crawler (http://linkstar.com/),gzip(gfe),gzip(gfe)Speedy Spider
Example user agent
Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)search crawler
Example user agent
camoca/camoca-n.1.2 (super-agent; search crawler; info at does not exist dot com)PostPost
Example user agent
PostPost/1.0 ( http://postpo.st/crawlers)OpenCrawler
Example user agent
Mozilla/5.0 (compatible; OpenCrawler/0.1.6.6; http://code.google.com/p/opencrawler/),gzip(gfe),gzip(gfe)Impressumscrawler
Example user agent
netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)bmdspider
Example user agent
Mozilla/4.0 (compatible; bmdspider; windows 5.1)Nutch
Example user agent
nutch crawler/Nutch-2.2magpie-crawler
Example user agent
magpie-crawler/1.1 (U; Linux amd64; en-GB; http://www.brandwatch.net)AppEngine-Google
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1626.1 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~hex-plus)Tailsweep
Example user agent
TailsweepBlogCrawler/Tailsweep-2.9-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)GoScraper
Example user agent
GoScraperNutch
Example user agent
Nutch12/Nutch-1.2Toms Spider
Example user agent
Toms Spider 0.3Yahoo-Newscrawler
Example user agent
Yahoo-Newscrawler/3.9 (news-search-crawler at yahoo-inc dot com)Googlebot
Example user agent
Mozilla (Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html))Engine Spider
Example user agent
ByWebSite Search Engine SpiderAdsBot-Google
Example user agent
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36ProoXiBot
Example user agent
ProoXiBotSWAT Crawler
Example user agent
SWAT Crawler. AGH University project. In case of problem contact: opal@tempus.metal.agh.edu.pl. Thanks.Kotoss Crawler
Example user agent
Kotoss Crawler http://hitokoto.kotoss.com,gzip(gfe),gzip(gfe)AmorankSpider
Example user agent
AmorankSpider/0.1; +http://amorank.com/webcrawler.htmlSpiderMan
Example user agent
SpiderMan 3.0.1-2-11-111 (CP/M;8-bit)ArchiveTeam ArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36FAST Crawler
Example user agent
Mozilla/5.0 (compatible; FAST Crawler 6.3)Comodo-Certificates-Spider
Example user agent
Comodo-Certificates-SpiderFollowSite Bot
Example user agent
FollowSite Bot ( http://www.followsite.com/bot.html )holmes
Example user agent
holmes/3.12.4 (http://morfeo.centrum.cz/bot)Googlebot-Image
Example user agent
Googlebot-Image/1.0pregnancycrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; http://www.pregnancycrawler.com)WASALive-Bot
Example user agent
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)YandexFavicons
Example user agent
Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)TelemetrySpider2
Example user agent
TelemetrySpider2/0.1 linuxLB-Crawler
Example user agent
LB-Crawler/1.0msnbot
Example user agent
adidxbot/2.0 (+http://search.msn.com/msnbot.htm)msnbot-media
Example user agent
msnbot-media/1.0 ( http://search.msn.com/msnbot.htm)PingdomBot
Example user agent
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)TweetmemeBot
Example user agent
Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)Baiduspider
Example user agent
Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)ERACrawler
Example user agent
ERACrawler/1.0KiwiStatus Spider
Example user agent
KiwiStatus (NZS.com)/0.2 (NZS.com KiwiStatus Spider, Local Search New Zealand; http://www.nzs.com; bot-at-nzs dot com)PortalBSpider
Example user agent
PortalBSpider/2.0 (spider@portalb.com)yacybot
Example user agent
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.htmlarchive.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)googlebot
Example user agent
mozilla/5.0 (iphone; cpu iphone os 8_3 like mac os x) applewebkit/600.1.4 (khtml, like gecko) version/8.0 mobile/12f70 safari/600.1.4 (compatible; googlebot/2.1; +http://www.google.com/bot.html)- Example user agent
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/) Link Crawler
Example user agent
DocWeb Link Crawler (http://doc.php.net)AboutUsBot
Example user agent
Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)zitebot
Example user agent
Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)Experibot_v1
Example user agent
Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)SemanticBot
Example user agent
Mozilla/5.0 (compatible; SemanticBot/1.1)AdvBot
Example user agent
Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)360Spider
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360SpiderNetchart Adv Crawler
Example user agent
Netchart Adv Crawler 1.0CompSpyBot
Example user agent
Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)zyklop-crawler
Example user agent
Mozilla/5.0 (compatible; zyklop; +http://www.seoratio.de/zyklop-crawler/)CrawlRobot
Example user agent
Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php)archive.bibalex.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.bibalex.org_bot; +http://www.test.com)our-crawler
Example user agent
Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)MetaGeneratorCrawler
Example user agent
MetaGeneratorCrawler/1.3.8 (www.metagenerator.info)robot
Example user agent
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)Yahoo-MMCrawler
Example user agent
Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)Vagabondo
Example user agent
Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)ImplisenseBot
Example user agent
ImplisenseBot 1.0VSMCrawler
Example user agent
Mozilla/5.0 (compatible; VSMCrawler; http://www.visualsitemapper.com/crawler/)NimbleCrawler
Example user agent
Mozilla/4.0 (NimbleCrawler 0.1) obeys NimbleCrawler directive contact jpump@looksmart.netlinkdexbot
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/)com Bot
Example user agent
Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)Pompos
Example user agent
Pompos/1.2 http://pompos.iliad.frFAST-WebCrawler
Example user agent
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)GoogleBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1Exabot
Example user agent
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)MSIECrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/4.0; FDM; MSIECrawler; Media Center PC 5.0)Aleksika Spider
Example user agent
Aleksika Spider/1.0 ( http://www.aleksika.com/)AppEngine-Google
Example user agent
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~chengzhoukun)ConveraCrawler
Example user agent
ConveraCrawler/0.9c ( http://www.authoritativeweb.com/crawl)SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)Xenu's Link Sleuth
Example user agent
Xenu's Link Sleuth 1.0pScreaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2,55WBSearchBot
Example user agent
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)googlebot
Example user agent
googlebotICC-Crawler
Example user agent
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp)NMG Spider
Example user agent
NMG Spider/0.3 (szukanko.com)YandexBot
Example user agent
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: twitter-mirror),gzip(gfe),gzip(gfe),gzip(gfe)Googlebot-Mobile
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)MetaHeadersBot
Example user agent
MetaHeadersBot (+http://www.metaheaders.com/bot.html)domainsbot
Example user agent
domainsbot (+http://www.domainsbot.com)MSIECrawler
Example user agent
Mozilla/4.0 (compatible; MSIE 4.0; MSIECrawler; Windows 95)ly spider
Example user agent
trunk.ly spider contact@trunk.lyAhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)LesnikBot
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)BountiiBot
Example user agent
Mozilla/5.0 (compatible; BountiiBot/1.1; +http://bountii.com/contact.php)8webcrawler
Example user agent
Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8webcrawler) http://skateboarddirectory.comGooglebot
Example user agent
Googlebot/2.1(+http://www.googlebot.com/bot.html)PWBot
Example user agent
Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw)OrangeBot
Example user agent
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)baiduspider
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50215; baiduspider)Thinglink ImageBot
Example user agent
Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)Tutorial Crawler
Example user agent
Tutorial Crawler 1.4CrawlDaddy
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.1.1.0 http://code.google.com/p/abot)fr_bot
Example user agent
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)- Example user agent
Yahoo:LinkExpander:Slingstone JikeSpider
Example user agent
JikeSpider; +http://shoulu.jike.com/spider.htmlThumbshots.ru
Example user agent
Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3PornSpider
Example user agent
Mozilla/5.0 (compatible; PornSpider/1.0; +http://www.pornspider.net)AnomaliesBot
Example user agent
AnomaliesBot/0.06-dev (The Anomalies Network Search Spider; http://www.anomalies.net; info@anomalies.net)updated crawler
Example user agent
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)- Example user agent
bot GomezAgent
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1Sogou web spider
Example user agent
Sogou web spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)fr-crawler
Example user agent
Mozilla/5.0 (compatible; fr-crawler/1.1)008
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620Charlotte
Example user agent
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@beta.spider.com)WordPress
Example user agent
WordPress/3.2.1; http://www.painlog.jpheritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com)SheenBot
Example user agent
SheenBot/SheenBot-1.0.4 (Sheen web crawler)JUST-CRAWLER
Example user agent
JUST-CRAWLER( http://www.justsystems.com/jp/tech/crawler/)Sogou Push Spider
Example user agent
Sogou Push Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)Checklinks
Example user agent
Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks)GG PeekBot
Example user agent
GG PeekBot 2.0 ( http://gg.pl/ http://info.gadu-gadu.pl/praca )Blekkobot
Example user agent
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)CCBot
Example user agent
CCBot/2.0 (http://commoncrawl.org/faq/)msnbot
Example user agent
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)Mail.RU_Bot
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)LinkedInBot
Example user agent
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 http://www.linkedin.com)WinkBot
Example user agent
WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)cloudforecastbot
Example user agent
cloudforecastbotYahooSeeker
Example user agent
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)eseek-crawler
Example user agent
eseek-crawler.0.5 (crawler@exactseek.com)IlTrovatore-Setaccio
Example user agent
IlTrovatore-Setaccio ( http://www.iltrovatore.it)FacebookBot
Example user agent
facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php)facebookscraper
Example user agent
facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php)Crayon Crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Crayon Crawler)CrawlConvera
Example user agent
CrawlConvera0.1 (CrawlConvera@yahoo.com)Crawl
Example user agent
Crawl/0.1 langcrawl/0.1TwitterCrawler
Example user agent
Mozilla/5.0 (compatible; TwitterCrawler)ANTIPLAG SPIDER
Example user agent
Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)bingbot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)YandexBlogs
Example user agent
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)Wotbox
Example user agent
Wotbox/2.01 ( http://www.wotbox.com/bot/)msnbot
Example user agent
msnbot/1.1 (+http://search.msn.com/msnbot.htm)Facebot
Example user agent
Facebot/1.0ip-web-crawler
Example user agent
ip-web-crawler.comAboundex
Example user agent
Aboundex/0.3 (http://www.aboundex.com/crawler/)Goodzer
Example user agent
Mozilla/5.0 (compatible; Goodzer/2.0; crawler@goodzer.com)Yeti
Example user agent
Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)archive.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.1-x http://pandora.nla.gov.au/crawl.html)ProCogBot
Example user agent
Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)VodpodCrawler
Example user agent
Mozilla/5.0 (compatible; VodpodCrawler/1.0; +http://vodpod.com/site/help)ssearch_bot
Example user agent
ssearch_bot (sSearch Crawler; http://www.semantissimo.de)TwitSpider
Example user agent
TwitSpiderBaiduspider-testbranch
Example user agent
Baiduspider-testbranch360Spider
Example user agent
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360SpiderWinHTTP
Example user agent
WinHTTPmetager2-verification-bot
Example user agent
Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)envolk[ITS]spider
Example user agent
envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)NetzCheckBot
Example user agent
NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com)GomezAgent
Example user agent
Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)XYZ Spider
Example user agent
XYZ SpiderPercolateCrawler
Example user agent
PercolateCrawler/4 (ops@percolate.com)heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://webarhive.nlc.gov.cn)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurpSogou inst spider
Example user agent
Sogou inst spider/4.0(+http://www.sogou.com/docs/help/webmasters.htmZemlyaCrawl
Example user agent
Mozilla/5.0 (compatible; ZemlyaCrawl/1.0; +http://zemlyaozer.com/bot)JNLP
Example user agent
JNLP/1.7.0 javaws/10.17.2.02 () Java/1.7.0_17YYSpider
Example user agent
Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)WeSEE:Ads
Example user agent
WeSEE:Ads/PageBot (http://www.wesee.com/bot/)GLBot
Example user agent
Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co)IXE Crawler
Example user agent
IXE Crawler
Comparison created
2016-02-13 13:13:14 | by
ThaDafinser