Bing
Example user agent
(Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.4.5.3000 Firefox/38
Example user agent
Screaming Frog SEO Spider/2.22Aboundexbot/0.2
Example user agent
Aboundex/0.2 (http://www.aboundex.com/crawler/)
Example user agent
Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
Example user agent
GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php)
Example user agent
RyzeCrawler/1.1.0 (+http://www.ryze.nl/crawler/)
Example user agent
Crawllybot/0.1/0.1 (Crawllybot/0.1; http://www.crawlly.com; crawler@crawlly.com)
Example user agent
Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)
Example user agent
Logict IPv6 Crawler/1.0 (http://ipv6search.logict.net),gzip(gfe),gzip(gfe)UnisterBot
Example user agent
Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)Bing
Example user agent
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)CiteSeerX
Example user agent
mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)
Example user agent
Tasapspider/0.9Exabot/3.0/BiggerBetter
Example user agent
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)PagePeeker
Example user agent
PagePeeker.comcoccoc/1.0
Example user agent
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)
Example user agent
BaiDuSpider
Example user agent
Mozilla/5.0 (compatible; LikaholixCrawler/1.0; +http://mylikes.com/about/crawler)
Example user agent
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)
Example user agent
camoca/camoca-n.1.2 (super-agent; search crawler; info at does not exist dot com)
Example user agent
omgilibot/0.3 +http://www.omgili.com/Crawler.html
Example user agent
Shim-CrawlerYodaoBot-Image/1.0
Example user agent
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)W3C Link Checker
Example user agent
W3C-checklink/4.81 libwww-perl/5.836
Example user agent
449 Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp
Example user agent
PluckItCrawler/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com;)Baiduspider
Example user agent
'Mozilla/5.0 (compatible; Baiduspider/2.0; +ht'BLEXBot/1.0
Example user agent
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)
Example user agent
WebTarantula.com Crawler
Example user agent
Mozilla/5.0 (compatible; InfuzApp/1.0; +http://www.infuz.com/bot.html)
Example user agent
lworldspider
Example user agent
Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)Googlebot
Example user agent
Googlebot/Nutch-1.7Googlebot Mobile
Example user agent
Googlebot-Mobile (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Y!J-BRJ/YATS
Example user agent
Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Example user agent
Sogou develop spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Example user agent
Content Crawler
Example user agent
Screaming Frog SEO Spider/2,01Pinterest
Example user agent
Pinterest/0.1Archive.org
Example user agent
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.1-x http://pandora.nla.gov.au/crawl.html)Bloglovin
Example user agent
Bloglovin/1.0 (http://www.bloglovin.com; 0 subscribers)MSNBot/1.1
Example user agent
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
Example user agent
ScollSpider/2.0 ( http://www.webwobot.com/ScollSpider.php)iaskspider/2.0
Example user agent
iaskspider/2.0(+http://iask.com/help/help_index.html)Googlebot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ASProxy/5.5b5FeedZirra
Example user agent
feedzirra http://github.com/pauldix/feedzirra/tree/master
Example user agent
Microsoft Bing Mobile SocialStreams BotSEOkicks-Robot
Example user agent
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)Apache Nutch
Example user agent
Mozilla/5.0 (compatible; summers;)/Nutch-1.7
Example user agent
magpie-crawler/1.1 (U; Linux amd64; en-GB; http://www.brandwatch.net)Apache Nutch
Example user agent
nutch crawler/Nutch-2.2
Example user agent
CatchBot/1.0; http://www.catchbot.comAdsBot-Google
Example user agent
AdsBot-Google (+http://www.google.com/adsbot.html)
Example user agent
mixi-news-crawler/1.00 (http://mixi.jp/)
Example user agent
Mozilla/5.0 (compatible; WebCrunch/1.2; +http://webcrunch.net/crawler)Speedy Spider Beta/1.2
Example user agent
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)Bing
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0 http://www.bing.com/bingbot.htm)
Example user agent
Gigabot/2.0/gigablast.com/spider.html
Example user agent
YOURLS v1.7.1 +http://yourls.org/ (running on http://bp7.org)msnbot-NewsBlogs/2.0b
Example user agent
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)Googlebot
Example user agent
Googlebot/2.1 (http://www.googlebot.com/bot.html)CommonCrawl
Example user agent
CCBot/1.0 ( http://www.commoncrawl.org/bot.html)
Example user agent
CMS crawler ( http://buytaert.net/crawler/)Baiduspider
Example user agent
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search
Example user agent
Mozilla/5.0 (compatible; clumboot; +http://localhost/clumpit/bot.php)
Example user agent
VinjaVideoSpider/1.1YandexAntivirus/2.0
Example user agent
Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)lmspider
Example user agent
lmspider (lmspider@scansoft.com)Wget
Example user agent
Wget/1.11.4 Red Hat modified
Example user agent
Screaming Frog SEO Spider/2,22ZyBorg/1.0
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
Example user agent
Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler)Archive.org
Example user agent
Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)
Example user agent
cloudforecastbot
Example user agent
NewzCrawler/1.9 (compatible; MSIE 6.00; Newz Crawler 1.9; http://www.newzcrawler.com/ )
Example user agent
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
Example user agent
OmniExplorer_Bot/6.70 ( http://www.omni-explorer.com) WorldIndexer
Example user agent
PercolateCrawler/4 (ops@percolate.com)
Example user agent
wminer/1.2 (North America Web Crawler; http://www.wminer.com/bot; info at wminer dot com)
Example user agent
Mozilla/5.0 (compatible; WangIDSpider/1.0; +http://www.wangid.com/spider.html)Ezooms/1.0
Example user agent
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Example user agent
Mozilla/5.0 (compatible; GoGuidesBot; http://www.goguides.org/spider.html)
Example user agent
Diffbot/0.1Browsershots
Example user agent
BrowsershotsArchive.org
Example user agent
Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)Cliqzbot
Example user agent
Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)
Example user agent
Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler)
Example user agent
dotbotDealGates Bot/1.1
Example user agent
DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
Example user agent
NodejsSpider/1.0Findexa Crawler
Example user agent
Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece)
Example user agent
GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)Soso Web Spider
Example user agent
Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Example user agent
ArchitextSpider
Example user agent
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
Example user agent
AdnormCrawler www.adnorm.com/crawlerW3C Markup Validator
Example user agent
W3C_Validator/1.767boitho.com-dc/0.86
Example user agent
boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html )
Example user agent
Sogou Orion spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)Googlebot
Example user agent
Mozilla/5.0 (Windows; Mozilla 4.0(compatible; Googlebot/5.0; http://www.google.com/googlebot); en-US;)
Example user agent
EuripBot/0.5 (+http://www.eurip.com) PreCheck
Example user agent
ZoomInfo::Beehive Crawler
Example user agent
SMXCrawler/1.0 (www.socialmetrix.com)
Example user agent
PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)
Example user agent
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm
Example user agent
Punk Spider/PunkSPIDER-v0.1
Example user agent
Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; http://axtaris.org; crawler at axtaris dot org)MJ12bot/v1.4.3
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Example user agent
ProjectWF-java-test-crawler
Example user agent
GoGuidesBot/0.0.1 (GoGuides Indexing Spider; http://www.goguides.org/spider.html)SemrushBot/0.97
Example user agent
Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
Example user agent
wonderbot/JS 1.0LinkAider
Example user agent
LinkAider (http://linkaider.com/crawler/)
Example user agent
www.website-analyzer.net Website Analyzer CrawlerApache Nutch
Example user agent
Spider/Nutch-2.3-SNAPSHOT (Webcrawler)
Example user agent
radian6_default_(www.radian6.com/crawler)
Example user agent
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
Example user agent
ldspider (BTC 2011 crawl, harth@kit.edu, http://code.google.com/p/ldspider/wiki/Robots)
Example user agent
Mozilla/5.0 (compatible; Linux i686; en-US; URLfilterDB-crawler/1.0) ufdb/1.0
Example user agent
OpenWebSpider/0.6 (http://www.openwebspider.org)
Example user agent
Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)
Example user agent
Mozilla/5.0 (compatible; Vagabondo/Wapspider; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)Pinterest
Example user agent
Pinterest/0.1 +http://pinterest.com/iAsk Spider
Example user agent
iaskspider/2.0( http://iask.com/help/help_index.html)
Example user agent
GSiteCrawler/v1.12 rev. 260 (http://gsitecrawler.com/)
Example user agent
Symfony Spider (http://symfony.com/spider)
Example user agent
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Example user agent
MPICrawler/0.1 +http://kermit.news.cs.nyu.edu/crawler.html,gzip(gfe),gzip(gfe)Summify
Example user agent
Summify (Summify/1.0.1; +http://summify.com)
Example user agent
OmniExplorer_Bot/4.32 (+http://www.omni-explorer.com) WorldIndexer
Example user agent
hivaBot/hivaBot-0.2 (info@gorgor.ir)
Example user agent
kizasi-spider/1.0 (+http://kizasi.jp/)
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)
Example user agent
MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.htmlInternetSeer (Web Monitor)
Example user agent
InternetSeer.comScollSpider
Example user agent
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)Mail.RU_Bot/2.0
Example user agent
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Example user agent
FlaxCrawler/1.0,gzip(gfe),gzip(gfe)
Example user agent
Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)BazQux
Example user agent
Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 2 subscribers)
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://vaiqui.com)Picsearch bot
Example user agent
psbot-page (+http://www.picsearch.com/bot.html)
Example user agent
RavenCrawler
Example user agent
Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php)
Example user agent
Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)Larbin
Example user agent
larbin_2.6.3 larbin2.6.3@unspecified.mailEnaBot/1.2
Example user agent
EnaBot/1.2 (http://www.enaball.com/crawler.html)woriobot
Example user agent
Mozilla/5.0 (compatible; woriobot +http://worio.com)
Example user agent
Mozilla/4.0 (compatible; Yet-Another-Spider; )
Example user agent
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Example user agent
FaviconArchiver/1.0 (+http://moblur.org/workshop/favicon_archive/)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130207.001528 +http://webarchiv.cz/kontakty/)YowedoBot/1.0
Example user agent
YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com)Job Roboter Spider 3
Example user agent
Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
Example user agent
OmniExplorer_Bot/6.63b (+http://www.omni-explorer.com) WorldIndexer
Example user agent
Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.run4dom.com)
Example user agent
nlcrawler/1.0 (+http://northernlight.com/)
Example user agent
Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )
Example user agent
Moreoverbot/5.1 ( http://w.moreover.com; webmaster@moreover.com) Mozilla/5.0Googlebot-Mobile/2.1
Example user agent
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Example user agent
Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)
Example user agent
iisbot/1.0 (+http://www.iis.net/iisbot.html)
Example user agent
ExactSeek_SpiderLivedoor
Example user agent
Mozilla/5.0 (compatible; livedoor-Mobile-Gateway/0.02; +http://p.m.livedoor.com/help.html)
Example user agent
Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org)
Example user agent
noobot Spider/Noobot-1.2 (noobot-bot; http://www.noobot.fr)
Example user agent
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
Example user agent
Facebot/1.0
Example user agent
Mozilla/5.0 (compatible; SputnikImageBot/2.2)Steeler/3.3
Example user agent
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Example user agent
Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)Livedoor
Example user agent
livedoor HttpClientBing Preview
Example user agent
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/538.14 (KHTML, like Gecko) BingPreview/1.0b
Example user agent
Mozilla/5.0 (compatible; OpenCrawler/0.1.6.6; http://code.google.com/p/opencrawler/),gzip(gfe),gzip(gfe)Jambot/0.1.1
Example user agent
Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)
Example user agent
Mozilla/5.0 (compatible; AbiLogicBot/1.0; +http://www.abilogic.com/bot.html)
Example user agent
ReadPath_SpiderGoo RSS Reader
Example user agent
gooRSSreader3.7 - build20090410
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://shelby.im)
Example user agent
Mozilla/5.0 (compatible; archive.bibalex.org_bot; +http://www.test.com)
Example user agent
WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)
Example user agent
semantics webbot/1.0
Example user agent
Mozilla/5.0 (compatible; NLCrawler/2.0.15; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko)Baiduspider
Example user agent
Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)Gocrawl
Example user agent
Googlebot (gocrawl v0.4)R6_CommentReader
Example user agent
R6_CommentReader(www.radian6.com/crawler)
Example user agent
bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
Example user agent
Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)
Example user agent
Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/)
Example user agent
Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://oerk.at)Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)sogou spider
Example user agent
sogou spiderAboutUsBot Johnny5/2.0
Example user agent
Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)msnbot-media/1.1
Example user agent
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://u.i4x3.tk)
Example user agent
fastbot crawler beta 4.0 (+http://www.fastbot.de)Jyxobot
Example user agent
Jyxobot/1Google App Engine
Example user agent
AppEngine-Google; ( http://code.google.com/appengine)
Example user agent
Denodo IECrawler/4.5,gzip(gfe),gzip(gfe)Googlebot
Example user agent
Googlebot/1.3
Example user agent
Mozilla/5.0 ( compatible ; Veooz/1.0 ; +http://www.veooz.com/veoozbot.html )
Example user agent
Simpy/1.1 (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)Yahoo! Slurp/3.0
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)IRLbot/3.0
Example user agent
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)Bing
Example user agent
Mozilla/5.0 (compatible;bingbot/2.0;+http://www.bing.com/bingbot.htm)IRLbot/3.0 b
Example user agent
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)Site24x7
Example user agent
Site24x7Mediapartners-Google
Example user agent
Mediapartners-GoogleGoogle Feedfetcher
Example user agent
Mozilla/5.0 (compatible) Feedfetcher-Google;(+http://www.google.com/feedfetcher.html)
Example user agent
Mozilla/5.0 (compatible; PAD-bot/9.0; +http://www.descargarprogramagratis.com/)
Example user agent
Moreoverbot/5.00 (+http://www.moreover.com)
Example user agent
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Example user agent
OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)
Example user agent
IssueCrawler
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://2u4.us)
Example user agent
Automattic Analytics Crawler/0.2;+http://wordpress.com/crawler/
Example user agent
SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com)BabalooSpider/1.3
Example user agent
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)SBIder/0.8dev
Example user agent
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)Yahoo! Pipes
Example user agent
Yahoo Pipes 2.0OOZBOT/0.20
Example user agent
OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
Example user agent
PeerFactor CrawlerServer Density
Example user agent
Server Density Service Monitoring v2
Example user agent
WSDLSpider/1.0 (http://www.wsdlworld.com)
Example user agent
AcquiloSpider/1.0,gzip(gfe),gzip(gfe)
Example user agent
Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)Apache Nutch
Example user agent
Your Nutch Spider/Nutch-2.2.1snap.com beta crawler v0
Example user agent
snap.com beta crawler v0
Example user agent
SearQuBot/SearQuBot v1.0genieBot b
Example user agent
genieBot ((http://64.5.245.11/faq/faq.html))
Example user agent
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
Example user agent
axadine/ (Axadine Crawler; http://www.axada.de/; )CloudFare Always Online
Example user agent
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34uMBot-FC/1.0
Example user agent
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)
Example user agent
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html)
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://clickand.co)
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://pagn.info)FAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/6 (www.fastsearch.com)Scrapy
Example user agent
Scrapy/1.0.3.post6+g2d688cd (+http://scrapy.org)archive.org_bot
Example user agent
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Example user agent
Mozilla/5.0 (compatible; ca-crawler/1.0)EventGuruBot/1.0
Example user agent
Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
Example user agent
Mozilla/5.0 (lbot/2.6.4) lbot@mailme.not
Example user agent
EDI/1.6.0 (Edacious & Intelligent Web Crawler)
Example user agent
Xaldon_WebSpider/2.0.b1Baiduspider
Example user agent
Baiduspider
Example user agent
Mozilla/5.0 (compatible; phpservermon/3.1.1; +http://www.phpservermonitor.org)
Example user agent
BeijingCrawler
Example user agent
Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/)Soso Web Spider
Example user agent
Sosospider+(+help.soso.com/webspider.htm)
Example user agent
uipbot/1.0 (uipbot@semasio.net)
Example user agent
TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
Example user agent
ERACrawler/1.0
Example user agent
Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)
Example user agent
Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)PageBitesHyperBot/600
Example user agent
PageBitesHyperBot/600 (http://www.pagebites.com/)Ask Jeeves/Teoma
Example user agent
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)Googlebot snippet
Example user agent
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
Example user agent
Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)SpokeSpider/1.0
Example user agent
SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)80legs
Example user agent
Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620
Example user agent
Mozilla/4.0 (compatible; BOTW Spider; http://botw.org)
Example user agent
Lycos_Spider_(modspider)
Example user agent
PagesInventory (robot +http://www.pagesinventory.com)CazoodleBot-0.1
Example user agent
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)silk/1.0
Example user agent
silk/1.0 (+http://www.slider.com/silk.htm)/3.7Baiduspider japan
Example user agent
Baiduspider+(+http://www.baidu.jp/spider/)MJ12bot/v1.4.0
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
Example user agent
Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)Apache Nutch
Example user agent
spider/Nutch-1.5.1 (spider; http://www.xxx.com)
Example user agent
Mozilla/5.0 (compatible; Dazoobot/0.1; +http://dazoo.fr)FacebookExternalHit/1.0
Example user agent
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
Example user agent
Mozilla/5.0 (lbot/2.6.4) (lbot@mailme.not)
Example user agent
SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net),gzip(gfe),gzip(gfe)Yahoo! Ad Monitoring
Example user agent
Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)
Example user agent
Screaming Frog SEO Spider/2.40
Example user agent
Slinky v1.0 +http://dentedreality.com.au/projects/slinky/Google News
Example user agent
Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))Google App Engine
Example user agent
AppEngine-Google; (+http://code.google.com/appengine; appid: xbmacx)
Example user agent
XYZ Spider
Example user agent
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0 (resolver); +http://tweetedtimes.com)Exabot/3.0
Example user agent
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Example user agent
Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)facebookplatform/1.0
Example user agent
facebookplatform/1.0 (+http://developers.facebook.com)CloudServerMarketSpider/1.0
Example user agent
Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)
Example user agent
OmniExplorer_Bot/6.70 (+http://www.omni-explorer.com) WorldIndexerAlmaden
Example user agent
http://www.almaden.ibm.com/cs/crawler
Example user agent
Mozilla/5.0 (compatible; TestCrawler)
Example user agent
OmniExplorer_Bot/5.25 (+http://www.omni-explorer.com) WorldIndexerY!J-BSC/1.0
Example user agent
Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)Google Play Newsstand
Example user agent
GoogleProducer; (+http://goo.gl/7y4SX)
Example user agent
OmniExplorer_Bot/5.28 (+http://www.omni-explorer.com) WorldIndexer80legs/0.83 b
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620Sogou Web Spider
Example user agent
Sogou web spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)Googlebot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Example user agent
Sangfor SpiderFlamingo Search
Example user agent
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)updated/0.1-alpha
Example user agent
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
Example user agent
Zeusbot/0.8.1 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)
Example user agent
SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)Bing
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0; )"FAST MetaWeb Crawler
Example user agent
FAST MetaWeb Crawler (helpdesk at fastsearch dot com)
Example user agent
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)ichiro/1.0
Example user agent
ichiro/1.0 (ichiro@nttr.co.jp)
Example user agent
Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
Example user agent
T3census-Crawler/1.0
Example user agent
netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)
Example user agent
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/)
Example user agent
Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)IRLbot/2.0
Example user agent
IRLbot/2.0 (+http://irl.cs.tamu.edu/crawler)VoilaBot BETA 1.2
Example user agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)
Example user agent
tivraSpider/1.0 (crawler@tivra.com)PHP
Example user agent
PECL::HTTP/1.7.4 (PHP/5.4.4)
Example user agent
BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
Example user agent
Arikus_Spider
Example user agent
Comodo HTTP(S) Crawler - http://www.instantssl.com/crawlerYahoo! Japan
Example user agent
Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80)HiddenMarket-1.0-beta
Example user agent
HiddenMarket-1.0-beta (www.hiddenmarket.net/crawler.php)Yahoo! Image
Example user agent
Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)ScoutJet old
Example user agent
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Example user agent
EverbeeCrawler
Example user agent
BejiBot Crawler (BNL Services; http://www.bejjan.net/crawler/)Alexa
Example user agent
ia_archiver/8.7 (Windows NT 5.0; )
Example user agent
zspider/0.9-dev http://feedback.redkolibri.com/
Example user agent
Mozilla/5.0 (compatible; HiveCrawler/1.2 http://www.businessinsider.com)Google Wireless Transcoder
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14
Example user agent
adbeat_bot
Example user agent
TouTrix crawler/1.0
Example user agent
Mozilla/5.0 (compatible; BountiiBot/1.1; +http://bountii.com/contact.php)
Example user agent
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)AcoonBot/4.11.1
Example user agent
Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)Livedoor
Example user agent
Girls/2.0 (livedoor Co.,Ltd.; Peachy 2.1; iPhone; RSS Version 2.0; +http://girls.livedoor.com/)
Example user agent
Mozilla/5.0 (compatible; PornSpider/1.0; +http://www.pornspider.net)YandexFavicons/1.0
Example user agent
Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)bnf.fr_bot
Example user agent
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)Amfibibot/0.07
Example user agent
Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)Hatena Antenna
Example user agent
Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)FAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/5.3.4 (crawler@fast.no)Yahoo! Slurp China
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)EasyBib AutoCite
Example user agent
EasyBib AutoCite (http://content.easybib.com/autocite/)
Example user agent
Jaxified Bot (+http://www.jaxified.com/crawler/)Feedly
Example user agent
MetaFeedly/1.0 (http://www.feedly.com)
Example user agent
Mozilla/5.0 (compatible; Orangebot/2.0; support.orangebot@orange.com)
Example user agent
Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)Bing
Example user agent
msnbot/1.1 ( http://search.msn.com/msnbot.htm)
Example user agent
seebot/2.0 (+http://www.seegnify.com/bot)Sproose/1.0beta
Example user agent
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com)FAST Web Crawler
Example user agent
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
Example user agent
Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)Apache Nutch
Example user agent
MySpider/Nutch-2.2
Example user agent
MSR-ISRCCrawlergrapeFX/0.9
Example user agent
Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk
Example user agent
blackspider
Example user agent
Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/)
Example user agent
Mozilla/4.0 (compatible; Iplexx Spider/1.1)OpenWebSpider v0.1.4
Example user agent
OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
Example user agent
LEIA/3.01pr (LEIAcrawler; leia@gseek.com; http://www.gseek.com)Blogtrottr
Example user agent
Blogtrottr/2.0FurlBot/Furl Search 2.0
Example user agent
Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net)Bing
Example user agent
"msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)Amorank Spider
Example user agent
AmorankSpider/0.1; +http://amorank.com/webcrawler.html
Example user agent
Teragram/SAS Crawler
Example user agent
blog_crawler/1.0ZyBorg/1.0 - b
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)
Example user agent
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)
Example user agent
SimpleCrawler/0.1bixo
Example user agent
Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Example user agent
Mozilla/5.0 (compatible; mxbot/1.0; http://www.chainn.com/mxbot.html)PHP
Example user agent
GuzzleHttp/6.1.0 curl/7.38.0 PHP/7.0.0RC7cityreview
Example user agent
Cityreview Robot (+http://www.cityreview.org/crawler/)
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)
Example user agent
AppCodes crawler - looking for iOS app mentions. More info: support@appcodes.com. Robots.txt id: AppCodesCrawler
Example user agent
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)
Example user agent
Sangfor Spider/0.7MetamojiCrawler/1.0
Example user agent
Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.htmlFAST Web Crawler
Example user agent
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)YaCy Bot
Example user agent
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html
Example user agent
SheenBot/SheenBot-1.0.4 (Sheen web crawler)
Example user agent
Mozilla/5.0 (compatible; BeslistBot; nl; BeslistBot 1.0; http://www.beslist.nl/
Example user agent
HappyFunBot/1.1 ( http://www.happyfunsearch.com/bot.html)
Example user agent
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
Example user agent
Mozilla/4.0 (compatible; Inspyder-Crawler; http://www.inspyder.com)
Example user agent
SearchSpider/1.2.10
Example user agent
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Example user agent
Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)HubPages
Example user agent
HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)Google Web Preview
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13Peew/1.0
Example user agent
Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)PHP
Example user agent
PHP/5.2.13Crowsnest/0.5
Example user agent
Crowsnest/0.5 (+http://www.crowsnest.tv/)Yahoo! Seeker
Example user agent
YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
Example user agent
8bo Crawler Bot
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://shortr.at)TelegramBot
Example user agent
TelegramBot (like TwitterBot)
Example user agent
OmniExplorer_Bot/5.01 (+http://www.omni-explorer.com) WorldIndexerIlTrovatore/1.2
Example user agent
IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)
Example user agent
Moreoverbot/5.00 (+http://www.moreover.com; webmaster@moreover.com)CommaFeed
Example user agent
CommaFeed/1.0 (http://www.commafeed.com)
Example user agent
http://www.Syntryx.com/ ANT Chassis 9.27; Mozilla/4.0 compatible crawlerYandexBot/3.0
Example user agent
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)Kouio
Example user agent
kouio.com RSS readerMSNBot/2.0b + ._
Example user agent
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._pmoz.info ODP link checker
Example user agent
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
Example user agent
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Example user agent
Mozilla/5.0 (compatible; Hailoobot/1.2; http://www.hailoo.com/spider.html)
Example user agent
psbot-image (+http://www.picsearch.com/bot.html)W3C MobileOK Checker
Example user agent
W3C-mobileOK/DDC-1.0 (see http://www.w3.org/2006/07/mobileok-ddc)SilverReader
Example user agent
SilverReader/1.0; http://silverreader.com
Example user agent
Googlebot v2.1 (+http://www.google.com/bot.html) (http://www.openwebspider.org/)ia_archiver alexa
Example user agent
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)360spider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
Example user agent
Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)
Example user agent
WebCompanyCrawlerWIRE/0.11 b
Example user agent
WIRE/0.11 (Linux; i686; Robot,Spider,Crawler,aromano@cli.di.unipi.it)Steeler/3.5
Example user agent
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)360spider
Example user agent
360spider-image
Example user agent
NMG Spider/0.3 (szukanko.com)
Example user agent
Mozilla/5.0 (compatible; StreamScraper/1.0; +http://code.google.com/p/streamscraper/)SeznamBot/3.1-test
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)YioopBot
Example user agent
Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://sgfx.co)
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/v0.1.5ALPHA SVN rev58; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)
Example user agent
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)Googlebot
Example user agent
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://mrk.im)Baiduspider
Example user agent
Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)
Example user agent
mixi-crawler/2.00 (http://mixi.jp/)Yandex Newslinks
Example user agent
Mozilla/5.0 (compatible; YandexNewslinks; +http://yandex.com/bots)
Example user agent
PortalBSpider/2.0 (spider@portalb.com)
Example user agent
FyberSpider (+http://www.fybersearch.com/fyberspider.php)Bing
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)PHP
Example user agent
Guzzle/4.2.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.11Scrapy
Example user agent
Scrapy/0.24.6 (+http://scrapy.org)PHP
Example user agent
PHP/5.3.77Google Wireless Transcoder
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;)
Example user agent
Altresium/4.6 (+http://www.altresium.com/bot.html)Apache Nutch
Example user agent
your sipder name/Nutch-1.7
Example user agent
BlogRangerCrawler/1.0
Example user agent
NationalDirectory-WebSpider/1.3
Example user agent
DRKSpider - Website link validator - http://www.drk.com.ar/spider/,gzip(gfe),gzip(gfe),gzip(gfe)
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://www.easylink.mobi)Googlebot-Image/1.0
Example user agent
Googlebot-Image/1.0Seznam screenshot-generator 2.0
Example user agent
Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)Feedly
Example user agent
FeedlyApp/1.0 (http://www.feedly.com)
Example user agent
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touchvoyager/2.0
Example user agent
voyager/2.0 (http://www.kosmix.com/crawler.html)
Example user agent
ByWebSite Search Engine SpiderData Hotel Watchdog
Example user agent
Data-Hotel-Watchdog/1.1ADmantX
Example user agent
ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://fhort.com)
Example user agent
jrCrawler/1.0bApache Nutch
Example user agent
MSNBot/Nutch-1.5.1
Example user agent
Liquida.it-Crawler/1.0 ( crawler@liquida.it +http://www.liquida.it )proximic
Example user agent
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)Facebook
Example user agent
facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php)Googlebot
Example user agent
Googlebot/2.1(+http://www.googlebot.com/bot.html)
Example user agent
Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://z-url.us)
Example user agent
Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
Example user agent
Mozilla/5.0 GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)
Example user agent
Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)FeedValidator
Example user agent
FeedValidator/1.3StatusCake Error Detector
Example user agent
StatusCake Error Detector Crawler V0.01QuerySeekerSpider
Example user agent
QuerySeekerSpider ( http://queryseeker.com/bot.html )CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dla-marbach.de)
Example user agent
LinkStats Bot
Example user agent
CheckSite Verification Agent (+http://www.checksite.us)
Example user agent
fastlwspider/1.0Pinterest
Example user agent
Pinterest/0.2 (+http://www.pinterest.com/)
Example user agent
niki-bot200PleaseBot/1.0
Example user agent
Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)Baiduspider/2.0
Example user agent
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)FAST Web Crawler
Example user agent
FAST-WebCrawler/2.2.11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)PHP
Example user agent
CakePHPTwikle/1.0
Example user agent
Twikle/1.0 , http://twikle.com , contact@twikle.com
Example user agent
New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98)
Example user agent
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.rogerbot/1.0
Example user agent
Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)Google App Engine
Example user agent
AppEngine-Google
Example user agent
cis455crawlerButterfly/1.0 a
Example user agent
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8WordPress
Example user agent
WordPress/3.7.1; http://wordpress.comCareerBot/1.1
Example user agent
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
Example user agent
wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)Yeti/1.0
Example user agent
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)Ask Jeeves
Example user agent
Inne: Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html)
Example user agent
Mozilla/5.0 (compatible; zyklop; +http://www.seoratio.de/zyklop-crawler/)MSRBOT c
Example user agent
MSRBOT (http://research.microsoft.com/research/sv/msrbot)Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Example user agent
Aleksika Spider/1.0 ( http://www.aleksika.com/)ichiro/3.0
Example user agent
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)Google Ads Bot
Example user agent
AdsBot-Google ( http://www.google.com/adsbot.html)
Example user agent
Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)
Example user agent
agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com)
Example user agent
WeSEE:Ads/PageBot (http://www.wesee.com/bot/)Y!J-BRJ/YATS
Example user agent
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)Visbot/2.0
Example user agent
VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)
Example user agent
Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)
Example user agent
Screaming Frog SEO Spider/2,55IRLbot/2.0 b
Example user agent
IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)RssBar
Example user agent
RssBar/1.29 (RssBar for unDonut 1.35)
Example user agent
Mozilla/5.0 (compatible; VodpodCrawler/1.0; +http://vodpod.com/site/help)
Example user agent
IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler)MetaJobBot
Example user agent
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
Example user agent
Mozilla/5.0 (compatible; PiplBot; http://www.pipl.com/bot/)
Example user agent
MetaInspector/3.0.0 (+https://github.com/jaimeiniesta/metainspector)Bing
Example user agent
msnbot/2.0 (+http://search.msn.com/msnbot.htm) WASALive-Bot
Example user agent
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)Sosospider
Example user agent
Sosospider+(+http://help.soso.com/webspider.htm)
Example user agent
Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)Urlfilebot/2.2
Example user agent
Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
Example user agent
Mozilla/5.0 (compatible; ZemlyaCrawl/1.0; +http://zemlyaozer.com/bot)
Example user agent
Hatena::Crawler/0.01Ask Jeeves/Teoma - b
Example user agent
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)NimbleCrawler/2.0.1
Example user agent
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.comSnappy/1.1
Example user agent
Snappy/1.1 ( http://www.urltrends.com/ )
Example user agent
Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html)InternetArchive/0.8-dev
Example user agent
InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Example user agent
Mozilla/5.0 (compatible; Scrubby/2.2; +http://www.scrubtheweb.com/)
Example user agent
DotSpotsBot/0.2 (crawler; support at dotspots.com)
Example user agent
WSDL Crawler
Example user agent
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)
Example user agent
CowCrawler/CowCrawler-dev (+http://beta.cow.com),gzip(gfe),gzip(gfe)
Example user agent
Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawlerSpeedy Spider
Example user agent
Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)Apache Nutch
Example user agent
NutchSpider/Nutch-1.4
Example user agent
Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)Crawler4j
Example user agent
crawler4j (http://code.google.com/p/crawler4j/)
Example user agent
SensikaBot/x.33 (+http://sensika.com)Wotbox/2.01
Example user agent
Wotbox/2.01 (+http://www.wotbox.com/bot/)Bing
Example user agent
(Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.0WBSearchBot/1.1
Example user agent
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
Example user agent
TwitSpider
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://www.h-link.de)
Example user agent
woobot/2.0ADmantX
Example user agent
ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.comBlogram
Example user agent
BlogramCrawler/1.0.1(+http://blogram.jp/)WebbCrawler
Example user agent
WebbCrawler 1.0 ( http://badcheese.com/crawler.html )Sensis Web Crawler - b
Example user agent
Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au)
Example user agent
Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )
Example user agent
Mozilla/5.0 (compatible; fr-crawler/1.1)
Example user agent
igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)Baiduspider
Example user agent
Baiduspider ( http://www.baidu.com/search/spider.htm)
Example user agent
deepcrawler/3.1 (http://www.queusearch.com/whatis_deepcrawler.php),gzip(gfe),gzip(gfe)
Example user agent
YOURLS v1.0-rc +http://yourls.org/ (running on http://epi.my)Tiny Tiny RSS
Example user agent
Tiny Tiny RSS/1.11.4c63934 (http://tt-rss.org/)
Example user agent
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
Example user agent
haupia-crawler
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://jmp.su)
Example user agent
ProloCrawler/1.0 (http://www.prolo.com)WordPress
Example user agent
Pachacutec/0.5 (qpImageBuilders.com); WordPress/3.1; http://contacto-latino.com/newsYahoo! Japan
Example user agent
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
Example user agent
WNMbot/0.4.0
Example user agent
ConveraMultiMediaCrawler/0.1 ( http://www.authoritativeweb.com/crawl)
Example user agent
Mozilla/5.0 (compatible; kgbody/2.0; +http://kids.goo.ne.jp/tool/kgbody.php)discoverybot/2.0
Example user agent
Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)TwengaBot
Example user agent
TwengaBot
Example user agent
DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/botLijit
Example user agent
Lijit Crawler (+http://www.lijit.com/robot/crawler)Apache Nutch
Example user agent
Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)
Example user agent
Feedfetcher-feedraider; (+http://feedraider.com/help/feedfetcher.html; X subscribers; Allow like Gecko)
Example user agent
YOURLS v1.7.1 +http://yourls.org/ (running on https://ivyti.es)socialbm_bot/1.0
Example user agent
Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)
Example user agent
TAMU_CS_IRL_CRAWLER/1.0
Example user agent
Superarama.com - BOT/v.0.1
Example user agent
Facebot
Example user agent
Mozilla/5.0 (compatible; StoneSunSpider/1.1)
Example user agent
RankFlex.com Webspider
Example user agent
ScSpider/0.2Apache Nutch
Example user agent
CB/Nutch-1.7PostRank
Example user agent
PostRank/2.0 (postrank.com; 1 subscribers)Y!J-BRO/YFSJ
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)ShopWiki/1.0
Example user agent
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)Google News
Example user agent
Googlebot-News
Example user agent
Apexoo Spider (http://www.apexoo.com/spider/)
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html; zurukko1552995316;)
Example user agent
Adknowledge Engage Crawler
Example user agent
tCrawler/0.1,gzip(gfe),gzip(gfe),gzip(gfe)rogerbot/1.0
Example user agent
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)
Example user agent
YOURLS v1.7.1 +http://yourls.org/ (running on http://sresc.io)Automattic Analytics Crawler/0.1
Example user agent
Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/Apache Httpd
Example user agent
Apache-HttpClient/UNAVAILABLE (java 1.4)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5Wget
Example user agent
Wget/1.9.1CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.fi/eng/project.html)
Example user agent
Mozilla/5.0 (compatible; Squzer/0.0.1; +http://www.declum.com/)RedBot1.0
Example user agent
RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)PowerMapper
Example user agent
CrawlerProcess (http://www.PowerMapper.com) /5.5.678.0TinEye/1.1
Example user agent
TinEye/1.1 (http://tineye.com/crawler.html)
Example user agent
www2.apserver.net ASP-Ranker Feed Crawler
Example user agent
Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)
Example user agent
EasouSpiderThumbshots.ru
Example user agent
Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3silk/1.0 -a
Example user agent
Silk/1.0Bing Preview
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 (KHTML, like Gecko) BingPreview/1.0b
Example user agent
boitho crawler
Example user agent
Mozilla/5.0 (Android 3.0; YRSpider; +http://www.yunrang.com/yrspider.html)NetWhatCrawler/0.06-dev
Example user agent
NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com)
Example user agent
Mozilla/5.0 (compatible; PaxleFramework/0.1.1; http://www.paxle.net/en/bot)Mixi
Example user agent
mixi-check/1.0 (http://mixi.jp/)
Example user agent
SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com )Yahoo-MMCrawler/4.0
Example user agent
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Example user agent
Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.comBing Preview
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://urlmicro.me)LexxeBot/1.0
Example user agent
LexxeBot/1.0 (lexxebot@lexxe.com)
Example user agent
Autonomy Spider
Example user agent
Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)moba-crawler
Example user agent
DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
Example user agent
Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)AhrefsBot
Example user agent
AhrefsBot.Feeds v0.1; http://ahrefs.com/Mapian News Bot
Example user agent
"mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)"SeznamBot/3.0
Example user agent
SeznamBot/3.0 (+http://fulltext.sblog.cz/)PHP
Example user agent
PHP
Example user agent
Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
Example user agent
Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/)Mail.RU_Bot/2.0
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Example user agent
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
Example user agent
OmniExplorer_Bot/6.60 (+http://www.omni-explorer.com) WorldIndexerW3C Unified Validator
Example user agent
W3C_Unicorn/1.0 (http://validator.w3.org/services)
Example user agent
Mozilla/5.0 (compatible; X11; Linux x86_64; Google-StructuredDataTestingTool; +http://www.google.com/webmasters/tools/richsnippets)
Example user agent
Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com)
Example user agent
NerdyBotBlogPulse
Example user agent
BlogPulse (ISSpider-3.0)
Example user agent
Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com)
Example user agent
Mozilla/5.0 (compatible; SputnikBot/2.2)
Example user agent
Mozilla/5.0 (compatible; SRCCN!Spider/1.1; +http://site.srccn.com/spider.html)netEstate NE Crawler
Example user agent
netEstate NE Crawler (+http://www.sengine.info/)woriobot
Example user agent
Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)UnwindFetchor/1.0
Example user agent
UnwindFetchor/1.0 (+http://www.gnip.com/)Bing
Example user agent
Mozilla/5.0 (compatible; bingbot/2.1; +http://www.bing.com/bingbot.htm)Ad Muncher
Example user agent
Ad Muncher v4.93.33707/4957WinkBot/0.06
Example user agent
WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)Feedspot
Example user agent
Feedspot http://www.feedspot.comFacebook
Example user agent
FacebookSecurity/2.1 ([www.FacebookSecurity.com])
Example user agent
Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org)
Example user agent
IPv4Scan (+http://ipv4scan.com)Archive.org
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Example user agent
Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3
Example user agent
Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws)
Example user agent
Mozilla/5.0 (compatible; FunkyCrawler; )Googlebot-Mobile/2.1
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Example user agent
Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)
Example user agent
Locubot (compatible; Googlebot; msnbot)Archive.org
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
Example user agent
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)WinWebBot/1.0
Example user agent
WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;)Sogou Web Spider
Example user agent
Sogou inst spider/4.0(+http://www.sogou.com/docs/help/webmasters.htmZyBorg/1.0 Dead Link Checker
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)
Example user agent
stq_bot (+http://www.searchteq.de)Genieo/1.0
Example user agent
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
Example user agent
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Example user agent
Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)Baiduspider
Example user agent
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)
Example user agent
truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )
Example user agent
Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)
Example user agent
Google (+https://developers.google.com/+/web/snippet/)Domain Re-Animator
Example user agent
Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com
Example user agent
pr-cy.ru Screenshot Bot
Example user agent
Screaming Frog SEO Spider/1.10CommonCrawl
Example user agent
CCBot/2.0 (http://commoncrawl.org/faq/)Grub
Example user agent
grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)
Example user agent
NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)Bing
Example user agent
msnbot/0.9 (+http://search.msn.com/msnbot.htm)
Example user agent
BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com)
Example user agent
Mozilla/4.0 (compatible: FDSE robot)AddThis.com robot
Example user agent
AddThis.com robot tech.support@clearspring.comBing
Example user agent
msnbot/2.1
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)
Example user agent
Bimbot/1.0
Example user agent
YoonoCrawler/1.0 (crawler@yoono.com)
Example user agent
AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)suggybot/0.01a
Example user agent
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
Example user agent
Mozilla/5.0 (compatible; AbiLogicBot/1.0; +http://www.abilogic.com)Bing
Example user agent
msnbot/2.0b ( http://search.msn.com/msnbot.htm)Twisted PageGetter
Example user agent
Twisted PageGetter
Example user agent
Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)
Example user agent
AnomaliesBot/0.06-dev (The Anomalies Network Search Spider; http://www.anomalies.net; info@anomalies.net)
Example user agent
RSSIncludeBot/1.0 (http://www.rssinclude.com/spider)LTX71
Example user agent
ltx71 - (http://ltx71.com/)
Example user agent
Zookabot/2.4; +http://zookabot.com
Example user agent
Wotbox/2.01 ( http://www.wotbox.com/bot/)Mapian News Bot
Example user agent
mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)DotBot/1.1
Example user agent
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
Example user agent
CosmixCrawler/0.1R6_FeedFetcher
Example user agent
R6_FeedFetcher(www.radian6.com/crawler)WordPress
Example user agent
WordPress/3.8.1; http://www.samsung-blog-news.de
Example user agent
GematchCrawler/2.1 (http://www.gematch.com/crawler.html)Googlebot
Example user agent
Mozilla (Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html))
Example user agent
WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)PostPost/1.0
Example user agent
PostPost/1.0 (+http://postpo.st/crawlers)
Example user agent
MonitorBacklinksWP (+http://monitorbacklinks.com/blog/incoming-links/)
Example user agent
iaskspiderlinksmanager_bot
Example user agent
Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)
Example user agent
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)
Example user agent
Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
Example user agent
Slackbot (+https://api.slack.com/robots)Bing
Example user agent
msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)
Example user agent
compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com)
Example user agent
everyfeed-spider/2.0 (http://www.everyfeed.com)
Example user agent
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://ezurl.me)Ruby
Example user agent
Ruby, link_thumbnailerXML Sitemaps Generator/1.0
Example user agent
Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0Nutch/0.8 at asked.jp
Example user agent
asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)
Example user agent
NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com)Esribot/1.0
Example user agent
Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)Bing
Example user agent
msnbot-media/1.0 ( http://search.msn.com/msnbot.htm)Orbiter
Example user agent
Orbiter (+http://www.dailyorbit.com/bot.htm)
Example user agent
Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)Blekkobot
Example user agent
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
Example user agent
FFC Trap Door SpiderGirafabot c
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; Girafabot [girafa.com])
Example user agent
Mozilla/5.0 (compatible; TwitterCrawler)Analytics SEO
Example user agent
Curious George - www.analyticsseo.com/crawler
Example user agent
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)MJ12bot/v1.4.4
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)HTTPMon
Example user agent
HTTPMon/1.0b (http://www.httpmon.com)
Example user agent
IRLbot/2.0 ( http://irl.cs.tamu.edu/crawler)Pinterest
Example user agent
Pinterest/4.1.3 CFNetwork/672.1.14 Darwin/14.0.0
Example user agent
Mozilla/5.0 (compatible; +http://tweetedtimes.com)Ruby
Example user agent
RubyW3C Internationalization Checker
Example user agent
W3C_I18n-Checker/1.0 (http://validator.w3.org/services)
Example user agent
IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org)Apache Nutch
Example user agent
DiscoverEd/Nutch-1.7 (OER search crawler; http://wiki.creativecommons.org/DiscoverEd; webmaster@creativecommons.org)
Example user agent
googlebotYandexImages/3.0
Example user agent
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Example user agent
Zao-Crawler
Example user agent
Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)rogerbot/1.0
Example user agent
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/
Example user agent
dCrawlBot/1.0.1120Sogou web spider/4.0
Example user agent
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Girafabot b
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
Example user agent
Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about)
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)YandexMetrika/2.0
Example user agent
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)Accoona-AI-Agent/1.1.1
Example user agent
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters)GrapeshotCrawler/2.0
Example user agent
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)ICC-Crawler/2.0
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)80legs/0.71
Example user agent
Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
Example user agent
abot/0.1 (abot; http://www.abot.com; abot@abot.com)
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)MSRBOT b
Example user agent
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)
Example user agent
Websquash.com (Add url robot)ExactSeek
Example user agent
ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com)
Example user agent
Mozilla/5.0 (compatible; DomainSCAN/2.0; +http://domainscan.info/robot/)YodaoBot/1.0
Example user agent
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://www.vrbas.be/url)
Example user agent
CamontSpider/1.0 http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
Example user agent
QuickFinder CrawlerDaumoa/3.0
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0PaperLiBot/2.1
Example user agent
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
Example user agent
Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
Example user agent
MSNBOT/0.1 (http://search.msn.com/msnbot.htm)
Example user agent
Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/)
Example user agent
Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)
Example user agent
HPI FeedCrawler/0.1 (+http://www.hpi.uni-potsdam.de/meinel/bross/feedcrawler)SynooBot/0.7.1
Example user agent
SynooBot/0.7.1 (SynooBot; http://www.synoo.de/bot.html; webmaster@synoo.com)
Example user agent
Pete-Spider/1.1Wget
Example user agent
Wget/1.6
Example user agent
Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)ichiro/5.0
Example user agent
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
Example user agent
Mozilla/5.0 (compatible; loc-crawl/1.10.1 +http://www.google.com)
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)FacebookExternalHit/1.1
Example user agent
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)Hatena Pagetitle
Example user agent
Hatena Pagetitle Agent/1.0
Example user agent
Jaxified Crawler 1.0a (+http://www.jaxified.com/),gzip(gfe),gzip(gfe)
Example user agent
Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)OpenindexSpider
Example user agent
Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html)Radian 6
Example user agent
R6_FeedFetcher_(www.radian6.com/crawler)
Example user agent
WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler)
Example user agent
HttpSpider/0.91meanpathbot/1.0
Example user agent
Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)
Example user agent
JoobleBot (compatible; http://jooble.org/jooble-bot)
Example user agent
Mozilla/5.0 (compatible; SiteBot/0.1; http://www.sitebot.org/robot/)
Example user agent
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
Example user agent
Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html)
Example user agent
OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexerNaver Yeti
Example user agent
Yeti/1.0 (+http://help.naver.com/robots/)Pingdom
Example user agent
Pingdom.com_bot_version_1.4_(http://www.pingdom.com)
Example user agent
INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)bingbot/2.0
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)Wget
Example user agent
Wget/1.10.2
Example user agent
Slackbot-LinkExpanding (+https://api.slack.com/robots)Google Sitemaps
Example user agent
Google-Sitemaps/1.0Spinn3r 3.1
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19BecomeBot/3.0
Example user agent
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
Example user agent
MetaGeneratorCrawler/1.3.8 (www.metagenerator.info)
Example user agent
Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://u.vsil.org)Yahoo! Japan
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248)AdsBot-Google b
Example user agent
AdsBot-Google
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://xurl.co)
Example user agent
Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
Example user agent
CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com)
Example user agent
TECOMAC-Crawler/0.3AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
Example user agent
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)Sogou Web Spider
Example user agent
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htmExactSeek
Example user agent
ExactSeek Crawler/0.1YaCy Bot
Example user agent
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.html
Example user agent
1NET/Crawler (compatible; 1NET; robot; http://www.1NET.gr)envolk/1.7
Example user agent
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
Example user agent
FAST Enteprise Crawler/6 (www dot fastsearch dot com)Google App Engine
Example user agent
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: twitter-mirror),gzip(gfe),gzip(gfe),gzip(gfe)360spider
Example user agent
360Spider
Example user agent
Yeti-FeedItemCrawler/1.0 (NHN Corp.;+http://help.naver.com/robots/)MojeekBot/0.2
Example user agent
MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)Feedbin
Example user agent
Feedbin - 9 subscribers
Example user agent
OmniExplorer_Bot/4.80 (+http://www.omni-explorer.com) WorldIndexer
Example user agent
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)Wget
Example user agent
Wget/1.15 (linux-gnu)CatchBot/1.0
Example user agent
CatchBot/1.0; +http://www.catchbot.com
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)CiteSeerX
Example user agent
flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)
Example user agent
Sogou Push Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Example user agent
LargeSmall Crawlerbitlybot
Example user agent
bitlybotGooglebot
Example user agent
Googlebot/2.1 (+http://www.googlebot.com/bot.html)Bing
Example user agent
msnbot/0.11 ( http://search.msn.com/msnbot.htm)Scrapy
Example user agent
Scrapy/0.20.1 (+http://scrapy.org)Google FeedBurner
Example user agent
FeedBurner/1.0 (http://www.FeedBurner.com)
Example user agent
Comodo-Certificates-Spider
Example user agent
Mozilla/5.0 (compatible; Host-Spy Crawler/1.2; +http://www.host-spy.com/)
Example user agent
Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw)Kalooga
Example user agent
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)Apache Nutch
Example user agent
testnutch/Nutch-1.8SimplePie
Example user agent
freshrss/0.8-dev (Linux; http://freshrss.org) SimplePie/1.4-dev-FreshRSS
Example user agent
lb-spider
Example user agent
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)
Example user agent
MQbot http://metaquerier.cs.uiuc.edu/crawlerGooglebot
Example user agent
Googlebot/2.2
Example user agent
Mozilla/5.0 (compatible; MerchantCentricBot/1.0)
Example user agent
PArchiveCrawler/1.0
Example user agent
pipeLiner/0.10 (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://rlink.eu)Googlebot/2.1
Example user agent
Googlebot/2.1 (+http://www.google.com/bot.html)
Example user agent
BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)
Example user agent
Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co)
Example user agent
CSimpleSpider RobotW3C Link Checker
Example user agent
W3C-checklink/4.0 [4.4] libwww-perl/5.803
Example user agent
Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0
Example user agent
OmniExplorer_Bot/6.62 (+http://www.omni-explorer.com) WorldIndexerPHP
Example user agent
PEAR HTTP_Request classSpeedy Spider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Example user agent
Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)
Example user agent
BebopBot/2.5.1 (compatible; media crawler V1; http://www.apassion4jazz.net/bebopbot.html;)
Example user agent
woobot/1.1
Example user agent
SeekOn Spider 1.9(+http://www.seekon.com/spider.html)
Example user agent
Mozilla/5.0 (compatible; Linux i686; Yandex.Gazeta Bot/1.0; +http://gazeta.yandex.ru)
Example user agent
envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)
Example user agent
Gnam Gnam Spider
Example user agent
Yasaklibot/v1.2 (http://www.Yasakli.com/bot.php)
Example user agent
LB-Crawler/1.0
Example user agent
Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)
Example user agent
LSSRocketCrawler/1.0 LightspeedSystemsSWEBot/1.0
Example user agent
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)Wget
Example user agent
Wget/1.14 (linux-gnu)Netcraft Survey Bot
Example user agent
Mozilla/4.0 (compatible; Netcraft Web Server Survey)
Example user agent
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
Example user agent
Mozilla/5.0 (compatible; ptd-crawler; http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://webarhive.nlc.gov.cn)
Example user agent
Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)Scrapy
Example user agent
Scrapy/0.16.4 (+http://scrapy.org)
Example user agent
adidxbot/2.0 (+http://search.msn.com/msnbot.htm)Googlebot-Mobile/2.1
Example user agent
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Wget
Example user agent
Wget/1.11.4Baiduspider
Example user agent
Baiduspider-image ( http://www.baidu.com/search/spider.htm)
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://3uves.com)Infohelfer/1.2.0
Example user agent
Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.f
Example user agent
AntBot/1.0 (http://www.ant.com)sistrix
Example user agent
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
Example user agent
Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)
Example user agent
EricssonR320/R1A (Fast Wireless Crawler)
Example user agent
Iframely/0.7.1 (+http://iframe.ly/;)Facebook
Example user agent
facebookexternalhit/1.1
Example user agent
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )WordPress
Example user agent
SAE/fetchurl-22wy0njxnm WordPress/3.2.1; http://1.webfront.sinaapp.comPinterest
Example user agent
Pinterest/3.3.3 CFNetwork/609.1.4 Darwin/13.0.0RufusBot (Rufus Web Miner)
Example user agent
RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)Apercite
Example user agent
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
Example user agent
noxtrumbot/1.0 (crawler@noxtrum.com)
Example user agent
node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
Example user agent
Brekiri crawler/1.0
Example user agent
it2media-domain-crawler/1.0 on crawler-prod.it2media.de
Example user agent
Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler)Google Feedfetcher
Example user agent
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)Archive.org
Example user agent
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://revmkt.in)
Example user agent
hl_ftien_spider
Example user agent
JikeSpider; +http://shoulu.jike.com/spider.htmlZumBot/1.0
Example user agent
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
Example user agent
IXE Crawler
Example user agent
Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Example user agent
Mozilla/5.0 (compatible; BlinkaCrawler/1.0; http://www.blinka.jp/crawler/)
Example user agent
ExactSeekCrawler/1.0
Example user agent
JUST-CRAWLER( http://www.justsystems.com/jp/tech/crawler/)LinkWalker/2.0
Example user agent
LinkWalker/2.0
Example user agent
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)oBot/2.3.1
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)GurujiBot/1.0
Example user agent
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)Fever
Example user agent
Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)
Example user agent
OOZBOT/0.17 (--; http://www.setooz.com/oozbot.html; pvvpr at iiit dot ac dot in)ExB Language Crawler 2.1.5
Example user agent
ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)CompSpyBot/1.0
Example user agent
Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)
Example user agent
BuildCMS crawler (http://www.buildcms.com/crawler)
Example user agent
Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 http://www.archive.org/details/archive_crawler)Tiny Tiny RSS
Example user agent
Tiny Tiny RSS/1.10 (http://tt-rss.org/)Monitor.us
Example user agent
Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)
Example user agent
GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)Goo Blog Search
Example user agent
gooblogsearch/2.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)
Example user agent
TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)OrangeBot
Example user agent
Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com)
Example user agent
ssearch_bot (sSearch Crawler; http://www.semantissimo.de)
Example user agent
ZillaCrawlerSpeedy Spider
Example user agent
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)Facebook
Example user agent
facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php)WordPress
Example user agent
WordPress/3.7.1; http://samsung.hdblog.itFyberSpider/1.3
Example user agent
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)magpie-crawler/1.1
Example user agent
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
Example user agent
WocBot/Mozilla/5.0 (Wocodi Web Crawler 1.0; http://www.wocodi.com/crawler; crawler@wocodi.com)Apache Nutch
Example user agent
agent6/Nutch-1.1
Example user agent
gsa-crawler
Example user agent
Mozilla/4.0 (compatible: Shoula robot)FTRF: Friendly Robot
Example user agent
FTRF: Friendly robot/1.3
Example user agent
i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)
Example user agent
bingbot
Example user agent
BillyBobBot/1.0 (+http://www.billybobbot.com/crawler/)
Example user agent
HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko)
Example user agent
Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks)Feedly
Example user agent
FeedlyBot/1.0 (http://feedly.com)
Example user agent
Yahoo-Newscrawler/3.9 (news-search-crawler at yahoo-inc dot com)Hatena Diary
Example user agent
Hatena Diary RSS Module (http://d.hatena.ne.jp/help#rssmodule)Y!J-BRW/1.0
Example user agent
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)Apache Nutch
Example user agent
5acd734a-1ed3-4a17-ad9d-0b09ae7bc2d2/Nutch-1.7Apache Nutch
Example user agent
My Nutch Spider/Nutch-1.6YoudaoBot/1.0
Example user agent
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://jim-s.com)
Example user agent
Mozilla/5.0 (compatible; Muenster University of Applied Sciences; +http://fb02itsscan.fh-muenster.de)Baiduspider
Example user agent
Baiduspider-testbranchGigabot/3.0
Example user agent
Gigabot/3.0 (http://www.gigablast.com/spider.html)Googlebot-Video/1.0
Example user agent
Googlebot-Video/1.0Naver Yeti
Example user agent
Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)netEstate NE Crawler
Example user agent
netEstate NE Crawler (+http://www.website-datenbank.de/)MetaURI API/2.0
Example user agent
MetaURI API/2.0 +metauri.com
Example user agent
MixBot (+http://t.co/GSRLLKex24)Willow Internet Crawler 2.1
Example user agent
Willow Internet Crawler by Twotrees V2.1Apache Nutch
Example user agent
Innovazion Crawler/Nutch-1.7
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)Archive.org
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot)ProCogBot/1.0
Example user agent
Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)Wget
Example user agent
Wget/1.16.1 (darwin14.0.0)
Example user agent
Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)Bing
Example user agent
msnbot/2.0b v1394720947.6703
Example user agent
Mozilla/5.0 (compatible; VSMCrawler; http://www.visualsitemapper.com/crawler/)
Example user agent
ProoXiBotFeedWrangler
Example user agent
Feed Wrangler/1.0 (3 subscribers; feed-id=248559; http://feedwrangler.net; Allow like Gecko)
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://mv2.me)
Example user agent
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
Example user agent
Mozilla/5.0 (compatible; YRSpider; +http://www.yunyun.com/SiteInfo.php?r=about)
Example user agent
nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com),gzip(gfe),gzip(gfe)
Example user agent
Mozilla/5.0 (compatible; EtaoSpider/1.0; http://open.etao.com/dev/EtaoSpider)
Example user agent
Mozilla/5.0 (compatible; TridentSpider/3.1)GingerCrawler/1.0
Example user agent
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)Qseero 1.0.0
Example user agent
Qseero v1.0.0
Example user agent
PicSpider/1.1 (spider@217-20-118-26.internetserviceteam.com; http://www.bildkiste.de)
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://ecra.se)
Example user agent
persomm-spider/v1.0
Example user agent
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Example user agent
KIT webcrawler/0.2.4
Example user agent
Phantom.js botEzooms Crawler
Example user agent
Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
Example user agent
Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)
Example user agent
Mozilla/5.0 (compatible; Scrubby/2.1; +http://www.scrubtheweb.com/abs/meta-check.html)
Example user agent
Mozilla/5.0 (compatible; CrawlBot/1.0.0)
Example user agent
Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc)
Example user agent
TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, +http://tweetedtimes.com)PHP
Example user agent
GuzzleHttp/6.1.0 curl/7.35.0 PHP/5.5.9-1ubuntu4.14Googlebot/2.1
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Example user agent
Sosoblogspider+(+http://help.soso.com/soso-blog-spider.htm)
Example user agent
GrowerIdeas Crawler/GrowerIdeas-nutch-1.6 (Crawls URLs for indexing content for our new search startup which aims to provide simple and smart search across curated content. For more info please contact helloworld@growerideas.com. If you thinkpsbot/0.1
Example user agent
psbot/0.1 (+http://www.picsearch.com/bot.html)
Example user agent
Jomjaibot/1.0 Crawl (+http://www.jomjaibot.com/)PHP
Example user agent
Mozilla/5.0 (compatible; PEAR HTTP_Request class;)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp
Example user agent
bot
Example user agent
MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)Kouio
Example user agent
kouio.com RSS reader - 6 subscribers
Example user agent
YisouSpider
Example user agent
Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://loc.gov),gzip(gfe),gzip(gfe),gzip(gfe)
Example user agent
LivelapBot/0.2 (http://site.livelap.com/crawler)Apache Nutch
Example user agent
Nutch12/Nutch-1.2
Example user agent
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)
Example user agent
Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com)Ask Jeeves/Teoma - c
Example user agent
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Example user agent
YRL_ODP_CRAWLER
Example user agent
Mozilla/5.0 (compatible; SemanticBot/1.1)thumbshots-de-bot
Example user agent
thumbshots-de-bot (+http://www.thumbshots.de/)
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://linkcut.eu)
Example user agent
Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)CiteSeerX
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://netarkivet.dk/webcrawler/)
Example user agent
Thesis_php_crawlerSogou-Test-Spider/4.0
Example user agent
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)Yandex Blogs
Example user agent
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)
Example user agent
Mozilla/5.0 (compatible; SemrushBot/0.97sa; +http://www.semrush.com/bot.html)
Example user agent
Mozilla/5.0 (compatible; Sosoimagespider/2.0; +http://help.soso.com/soso-image-spider.htm)Aboundexbot/0.3
Example user agent
Aboundex/0.3 (http://www.aboundex.com/crawler/)
Example user agent
WoFindeIch Robot 1.0(+http://www.search.wofindeich.com/robot.php)ScreenerBot Crawler Beta 2.0
Example user agent
ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
Example user agent
omgilibot/0.4 +http://omgili.comLink Valet Online 1.1
Example user agent
Link Valet Online 1.1
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/0.1.5a; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)ownCloud Server Crawler
Example user agent
ownCloud Server Crawler
Example user agent
Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)Cliqzbot
Example user agent
Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)80legs
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620
Example user agent
EsperanzaBot(+http://www.esperanza.to/bot/)OmniExplorer_Bot/6.47
Example user agent
OmniExplorer_Bot/6.47 (+http://www.omni-explorer.com) WorldIndexer
Example user agent
EmeraldShield.com WebBot
Example user agent
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)
Example user agent
Arachnida Web CrawlerVoilaBot BETA 1.2
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)pingdom.com_bot 1.4
Example user agent
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
Example user agent
Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)AhrefsBot/4.0
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)Genieo Web Filter
Example user agent
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html
Example user agent
domainsbot (+http://www.domainsbot.com)
Example user agent
Spider/5.0
Example user agent
ConveraCrawler/0.9c ( http://www.authoritativeweb.com/crawl)
Example user agent
Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)Livedoor
Example user agent
livedoor FeedFetcher/0.01 (http://reader.livedoor.com/; 999 subscribers)
Example user agent
GarlikCrawler/1.2 (http://garlik.com/
Example user agent
Mozilla/5.0 (compatible; nbot/2.0)FollowSite Bot
Example user agent
FollowSite Bot ( http://www.followsite.com/bot.html )
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://onj.me)Ruby
Example user agent
Ruby/CurbCatchBot/2.0
Example user agent
CatchBot/2.0; +http://www.catchbot.comPeeplo Screenshot Bot/0.20
Example user agent
Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )YandexNews/3.0
Example user agent
Mozilla/5.0 (compatible; YandexNews/3.0; +http://yandex.com/bots)
Example user agent
Top10Ranking Spider/3.1 ( http://www.top10Ranking.nl/, Top10ranking.nl heeft op een aantal woorden uw posities in Google gecheckt)
Example user agent
BOT/0.1 (BOT for JCE)
Example user agent
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)
Example user agent
OpenLink Virtuoso RDF crawler
Example user agent
Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html)
Example user agent
Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)Naver Yeti
Example user agent
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)
Example user agent
HuaweiSymantecSpider/1.0 DSE-support@huaweisymantec.com (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)Accoona-AI-Agent/1.1.2
Example user agent
Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)YandexImageResizer/2.0
Example user agent
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)
Example user agent
Fast PartnerSite Crawler
Example user agent
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://tty.nu)WordPress
Example user agent
WordPress/3.2.1; http://www.painlog.jp
Example user agent
DoubleVerify Crawler
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://rutlink.info)
Example user agent
Mozilla/5.0 (compatible; YoudaoBot-rts/1.0; http://www.youdao.com/help/webmaster/spider/; )
Example user agent
FyberSpiderBecomeBot/2.3
Example user agent
Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)ZumBot/1.0
Example user agent
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)
Example user agent
Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net)
Example user agent
Googlebot
Example user agent
Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
Example user agent
Mozilla/5.0 (compatible; ADJUSTbot/2.0; +http://www.ad-just.jp/)YandexDirect/3.0
Example user agent
Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)Pinterest
Example user agent
Pinterest/3356 CFNetwork/711.0.6 Darwin/14.0.0PHP
Example user agent
GuzzleHttp/6.1.0 curl/7.45.0 PHP/5.6.14
Example user agent
OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexerYahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0 ; http://help.yahoo.com/help/us/ysearch/slurp)rogerbot/1.0
Example user agent
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)wscheck.com/1.0.0
Example user agent
wscheck.com/1.0.0 (+http://wscheck.com/)holmes/3.12.4 - morfeo
Example user agent
holmes/3.12.4 (http://morfeo.centrum.cz/bot)
Example user agent
visaduhoc.info Crawler
Example user agent
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!
Example user agent
omgilibot/0.3 http://www.omgili.com/Crawler.html
Example user agent
yrspider (Mozilla/5.0 (compatible; YRSpider; http://www.yunrang.com/yrspider.html))BacklinkCrawler
Example user agent
BacklinkCrawler (http://www.backlinktest.com/crawler.html)
Example user agent
1470.net crawler
Example user agent
dtSearchSpiderScrapy
Example user agent
Scrapy/0.22.2 (+http://scrapy.org)
Example user agent
X-Crawler
Example user agent
Mozilla/5.0 (compatible; .None-Crawler/0.1 +http://domains.ericbinek.None/)
Example user agent
PsSpider/0.7 (spider@bildkiste.de)
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)Zookabot/2.5
Example user agent
Zookabot/2.5;++http://zookabot.com
Example user agent
Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)W3C Markup Validator
Example user agent
W3C_Validator/1.3 http://validator.w3.org/services
Example user agent
audioCrawlerBot/1.0 (http://www.audiocrawler.com/)AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxyfile1-hrd)
Example user agent
Webspider/1.0 (web spider; )MetaHeadersBot
Example user agent
MetaHeadersBot (+http://www.metaheaders.com/bot.html)W3C CSS Validator
Example user agent
Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See <http://validator.w3.org/services>)BUbiNG
Example user agent
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
Example user agent
Mozilla/4.0 (compatible; BlitzBot)
Example user agent
Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)
Example user agent
Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)
Example user agent
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
Example user agent
Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)
Example user agent
Mozilla/5.0 (Votay bot/4.0; http://www.votay.com/arts/comics/)OrgbyBot/1.2
Example user agent
Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)SEOENGBot/1.0
Example user agent
SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
Example user agent
flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)Exabot-Thumbnails
Example user agent
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)Apache Nutch
Example user agent
emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)CazoodleBot b
Example user agent
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com)
Example user agent
Mozilla/5.0 (compatible; AnyApexBot/1.0; +http://www.anyapex.com/bot.html)VocusBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; aggregator VocusBot 0.4; +http://www.vocus.com/vnhs.html)Pinterest
Example user agent
Pinterest/3.2 CFNetwork/672.0.8 Darwin/14.0.0Qualidator.com Bot 1.0
Example user agent
Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)Wget
Example user agent
Wget/1.12 (linux-gnu)
Example user agent
Zspider (+http://www.zhanzhangsou.com/index.htm)
Example user agent
DotBot/1.0.1 (http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://www.bv-xa.com)MSNBot/2.0b
Example user agent
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
Example user agent
Mozilla/4.0, VM-Crawler/cs version info ofni noisrev sc
Example user agent
TheSuBot/0.2 (www.thesubot.de)ShowyouBot
Example user agent
ShowyouBot (http://showyou.com/crawler)ROI Hunter
Example user agent
ROI Hunter; https://api-dev.roihunter.com
Example user agent
Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)discobot/1.0
Example user agent
Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)
Example user agent
Mozilla/4.0 (compatible; bmdspider; windows 5.1)Y!J-BRI/0.0.1
Example user agent
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )bot-pge.chlooe.com/1.0.0
Example user agent
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
Example user agent
Mozilla/5.0 (compatible; WebMasterAid/1.0; +http://wmaid.com/explore/)
Example user agent
PKU Student SpiderBloglovin
Example user agent
Bloglovin/1.0 (http://www.bloglovin.com; 1 subscribers)
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)Twitterbot
Example user agent
Twitterbot/1.0
Example user agent
FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)Linkdex Bot
Example user agent
linkdex.com/v2.0 and linkdex.com/v2.1
Example user agent
Liquida Spider 1.0 +http://liquida.com/
Example user agent
TailsweepBlogCrawler/Tailsweep-2.9-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)
Example user agent
rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+pr1-crawler-14@moz.com)Archive.org
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Example user agent
Mozilla/4.0 (compatible; focuseekbot)
Example user agent
Slackbot-LinkExpanding+(+https://api.slack.com/robots)
Example user agent
RSpider/1.0
Example user agent
Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)Backlink-Check
Example user agent
Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)
Example user agent
TheSuBot/0.1 (www.thesubot.de)
Example user agent
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://SiteExplorer.Info/)WordPress
Example user agent
WordPress/1.2.1 PHP/4.3.9-1WeSEE:Search/0.1 (Alpha)
Example user agent
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
Example user agent
DNSGroup/0.1 (DNS Group Crawler; http://www.dnsgroup.com/; crawler@dnsgroup.com)
Comparison created
2016-05-17 06:51:27 | by
ThaDafinser