Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2.22Aboundexbot
Example user agent
Aboundex/0.2 (http://www.aboundex.com/crawler/)
Example user agent
Mozilla/5.0 (Java; U; en-us; htc s740) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/405/UCWEB Mobile UNTRUSTED/1.0GIDBot
Example user agent
GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php)NaverBot
Example user agent
NaverCafe/3.1.6 (Android OS 4.0.4; samsung SHW-M380S; NaverCafe; 3.1.6)lwp
Example user agent
lwp-trivial/1.41Kraken
Example user agent
Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)CommonCrawler Node
Example user agent
Mozilla/5.0 CommonCrawler Node MNATI3RZFBD6ZTQNRVQGXBDCQOEXIA4CIDEUKNYZDF3MJSH5G7BCQ5EMADE5TLF.MHCZ3LLTWGHIHMYLM.YFBJBHE6R2YQTHOTJ3N5BGFP457JRX4AJRXGAJYGDAANRY24.cdn0.common.crawl.zoneCrawlboy
Example user agent
Crawllybot/0.1/0.1 (Crawllybot/0.1; http://www.crawlly.com; crawler@crawlly.com)Gluten Free Crawler
Example user agent
Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)UnisterBot
Example user agent
Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)MSN
Example user agent
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)Heritrix
Example user agent
mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)Exabot
Example user agent
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)WinHttp
Example user agent
WinHTTPCoccoc bot
Example user agent
Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)libwww
Example user agent
libwww-perl/5.64Baiduspider
Example user agent
BaiDuSpider
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-gs170 browser) UCBrowser8.4.0.159/70/352/UCWEB Mobile
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-a887) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-b7300) UCBrowser8.2.0.132/69/352/UCWEB Mobile, SAMSUNG-GT-B7300/1.0 UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)Blogshares Spiders
Example user agent
Blogshares Spiders (Renewed V1.65)Vinti
Example user agent
Vinti/0.1A6-Indexer
Example user agent
A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)Anonymizied
Example user agent
Mozilla/5.0 (Randomized by FreeSafeIP.com/upgrade-to-remove; compatible; MSIE 9.0; Windows NT 5.0WOW64) Chrome/18.0.1229.79Shim Crawler
Example user agent
Shim-CrawlerYodaoBot-Image
Example user agent
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)NaverBot
Example user agent
NANaverNDrive/3.0.0(Android OS 4.2.2;samsung SHV-E300S)W3C-checklink
Example user agent
W3C-checklink/4.81 libwww-perl/5.836Feeddler RSS Reader
Example user agent
FeeddlerRSS 2.4 (iPad; iPhone OS 7.1; en_GB)PluckItCrawler
Example user agent
PluckItCrawler/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com;)Embed PHP Library
Example user agent
Embed PHP Library
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-a817) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0Netchart Adv Crawler
Example user agent
Netchart Adv Crawler 1.0jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; N02A)Baiduspider
Example user agent
'Mozilla/5.0 (compatible; Baiduspider/2.0; +ht'BLEXBot
Example user agent
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)NewsGator
Example user agent
NewsGator/3.0,gzip(gfe),gzip(gfe)WebTarantula
Example user agent
WebTarantula.com CrawlerNutch
Example user agent
Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)Google Bot
Example user agent
GoogleEarth/7.1.0002.2011(Android;Android (HTC Desire HD A9191-ace-user-2.3.5);de-DE;kml:2.2;client:Free;type:default)NetLyzer FastProbe
Example user agent
NetLyzer FastProbeSwiss Search Engine
Example user agent
search.ch/2.8 Android/4.0.3; htc_europe:HTC One V; Mozilla/5.0 (Linux; U; Android-universal)Google Bot
Example user agent
Googlebot/Nutch-1.7Google Bot
Example user agent
Googlebot-Mobile (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Automated Security Analyser
Example user agent
asafaweb.comMnoGoSearch
Example user agent
MnoGoSearch/3.3.15Y!J-AGENT
Example user agent
Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)Sogou Develop Spider
Example user agent
Sogou develop spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)Crawlboy
Example user agent
CrawlFire - you can disable this Robot: http://pastebin.de/25277
Example user agent
Mozilla/5.0 (Java; U; en-in; samsung-gt-c5130) UCBrowser8.3.0.154/69/405/UCWEB Mobile UNTRUSTED/1.0Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2,01AuditMyPC Webmaster Tool
Example user agent
AuditMyPC Webmaster ToolInternet Archive
Example user agent
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.1-x http://pandora.nla.gov.au/crawl.html)Bloglovin Bot
Example user agent
Bloglovin/1.0 (http://www.bloglovin.com; 0 subscribers)msnbot
Example user agent
msnbot/1.1 (+http://search.msn.com/msnbot.htm)ScollSpider
Example user agent
ScollSpider/2.0 ( http://www.webwobot.com/ScollSpider.php)Android Download Manager
Example user agent
AndroidDownloadManagerFeedDemon
Example user agent
FeedDemon/4.5 (http://www.feeddemon.com/; Microsoft Windows XP)Google Bot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ASProxy/5.5b5WhatWeb Web Scanner
Example user agent
WhatWeb/0.4.8-devGoogle Bot
Example user agent
GOOG
Example user agent
Mozilla/5.0 (Java; U; Id; lg-t310) UCBrowser8.3.0.154/69/405/UCWEB Mobile UNTRUSTED/1.0Microsoft Bing Mobile SocialStreams Bot
Example user agent
Microsoft Bing Mobile SocialStreams BotFriendica
Example user agent
FriendicaScrubby
Example user agent
Scrubby/2.1 (http://www.scrubtheweb.com/)SEOkicks Robot
Example user agent
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)Nutch
Example user agent
Mozilla/5.0 (compatible; summers;)/Nutch-1.7magpie-crawler
Example user agent
magpie-crawler/1.1 (U; Linux amd64; en-GB; http://www.brandwatch.net)Nutch
Example user agent
nutch crawler/Nutch-2.2CatchBot
Example user agent
CatchBot/1.0; http://www.catchbot.comAdsBot Google
Example user agent
AdsBot-Google (+http://www.google.com/adsbot.html)Liferea
Example user agent
Liferea/1.10.6 (Linux; en_US.UTF8; http://liferea.sf.net/) AppleWebKit (KHTML, like Gecko)
Example user agent
CTC/HUAWEIC5720 Java/HWJa/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0Blackboard Safeassign
Example user agent
Blackboard SafeassignCC Metadata Scaper
Example user agent
CC Metadata Scaper http://wiki.creativecommons.org/Metadata_ScrapercURL
Example user agent
curl/7.19.5 (i586-pc-mingw32msvc) libcurl/7.19.5 zlib/1.2.3ht://Dig
Example user agent
htdig/3.1.5 (infosys@storm.rmi.org)Entireweb
Example user agent
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)BingBot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0 http://www.bing.com/bingbot.htm)Gigabot
Example user agent
Gigabot/2.0/gigablast.com/spider.html
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-gt-s5350l) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0Asynchronous HTTP Client
Example user agent
AHC/1.0Univention Corporate Server
Example user agent
UCS (ESX) - 4.0-3 errata302 - 28d414cc-2dac-4c0e-a34a-734020b8af66 - 00000000-0000-0000-0000-000000000000 -YOURLS
Example user agent
YOURLS v1.7.1 +http://yourls.org/ (running on http://bp7.org)msnbot-NewsBlogs
Example user agent
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)Google Bot
Example user agent
Googlebot/2.1 (http://www.googlebot.com/bot.html)Java Standard Library
Example user agent
Java/1.6.0_43CCBot
Example user agent
CCBot/1.0 ( http://www.commoncrawl.org/bot.html)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-s5350) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/405/UCWEB MobileAndroid Download Manager
Example user agent
AndroidDownloadManager/4.1.1 (Linux; U; Android 4.1.1; POV_TAB-P1325(V1.1) Build/JRO03H)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-lg900g) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0Research Projects
Example user agent
CMS crawler ( http://buytaert.net/crawler/)Baidu Image Search
Example user agent
BaiduImagespider ( http://help.baidu.jp/system/05.html)cURL
Example user agent
curl/7.16.0 (i386-unknown-openbsd4.1) libcurl/7.16.0 OpenSSL/0.9.7j zlib/1.2.3Baiduspider
Example user agent
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s5333) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0CATExplorador
Example user agent
CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)wget
Example user agent
Wget/1.11.4 Red Hat modifiedScreaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2,22Yahoo-Test
Example user agent
Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com)WiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-s7350i) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile UNTRUSTED/1.0nineconnections
Example user agent
Kimengi/nineconnections.com
Example user agent
Mozilla/5.0 (Java; U; en-id; samsung gt-m3310) UCBrowser8.2.1.144/69/444/UCWEB MobileFriendica
Example user agent
Friendica 'Lily of the valley' 3.4.0-1182; http://social.feder8.rujig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; 706P)Dispatch
Example user agent
Dispatch/0.11.1-SNAPSHOTURL Grabber
Example user agent
urlgrabber/3.9.1 yum/3.2.29Rippers
Example user agent
Mozilla/3.01 (compatible;)SetLinks.ru Crawler
Example user agent
Mozilla/5.0 (compatible; SetLinks bot 1.2)
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-b3410w) UCBrowser8.2.0.132/70/452/UCWEB Mobile UNTRUSTED/1.0Amiga
Example user agent
Mozilla/5.0 (compatible; Origyn Web Browser; U; AmigaOS 4.1; xx) AppleWebKit/534.0 (KHTML, like Gecko, Safari/534.0) OWB/3.32Internet Archive
Example user agent
Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)Theme Spider
Example user agent
Theme Spider ( http://www.themespider.com/spider.html)Semantic Discovery
Example user agent
semanticdiscovery/2.0(http://www.semanticdiscovery.com/robot.html)Baidu Mobile Search
Example user agent
DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0; http://www.baidu.jp/spider/)Windmaker Bot
Example user agent
Stewdio Windmaker 2.2Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.2-1188; https://friendika.openmindspace.orgRogerbot
Example user agent
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)Gregarius
Example user agent
Gregarius/0.5.2 (+http://devlog.gregarius.net/docs/ua)ZmEu
Example user agent
ZmEuEzooms
Example user agent
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-sgh-a597) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0NetNewsWire
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.74.9 (KHTML, like Gecko) Version/6.0 NetNewsWire/4.0.0Browsershots
Example user agent
BrowsershotsInternet Archive
Example user agent
Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)Cliqzbot
Example user agent
Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)redbot
Example user agent
RED/1 (http://redbot.org/)GurujiBot
Example user agent
GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)ArchitextSpider
Example user agent
ArchitextSpiderDuckDuck Favicons Bot
Example user agent
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5560i) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.2-1188; http://mozartweg.dyndns.org/friendica
Example user agent
Mozilla/5.0 (Java; U; es-la; htc touch hd t8282) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-c3500) UCBrowser8.2.0.132/69/352/UCWEB MobileW3C_Validator
Example user agent
W3C_Validator/1.767visionutils
Example user agent
visionutils/0.2OffByOne
Example user agent
Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b7320) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile, SAMSUNG-GT-B7320/1.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/4.0 UNTRUSTED/1.0krakenjs
Example user agent
kraken/0.6.0Boitho
Example user agent
boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html )360Spider
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)BingBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)Sogou Orion spider
Example user agent
Sogou Orion spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)Europe Internet Portal
Example user agent
EuripBot/0.5 (+http://www.eurip.com) PreCheckFlipboardProxy
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)SophosAgent
Example user agent
SophosAgent/1.0 (type= spa )BlogLines Web
Example user agent
Bloglines/3.0-rho (http://www.bloglines.com; 3 subscribers)SiteCon
Example user agent
Mozilla/5.0 (Linux; U; en-gb) AppleWebKit/418.9.1 (KHTML, like Gecko) SiteCon/8.10.22
Example user agent
Apache-HttpClient/4.0-beta2 (java 1.5)PageFreezer
Example user agent
PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)WordPress
Example user agent
WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 95.215.52.150Sogou Image Crawler
Example user agent
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htmMJ12bot
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; aplix jblend) UCBrowser8.3.0.154/69/444/UCWEB Mobile, MOT-A3100/1.0 Release/09.09.2008 Profile/MIDP-2.0 Configuration/CLDC-1.1 Software/WM6.1 UNTRUSTED/1.0isc header collector handlers
Example user agent
isc header collector handlers@isc.sans.edu
Example user agent
ProjectWF-java-test-crawler
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s8600) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0GoGuidesBot
Example user agent
GoGuidesBot/0.0.1 (GoGuides Indexing Spider; http://www.goguides.org/spider.html)SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.2-1188; https://karl.marx.pm
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-i6230) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0Crazywebcrawler
Example user agent
CRAZYWEBCRAWLER 0.9.0, http://www.crazywebcrawler.comwonderbot
Example user agent
wonderbot/JS 1.0HubSpot Webcrawler
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot WebcrawlerXenus Link Sleuth
Example user agent
Xenu's Link Sleuth 1.0pNutch
Example user agent
Spider/Nutch-2.3-SNAPSHOT (Webcrawler)libwww
Example user agent
libwww-perl/5.814GarlikCrawler
Example user agent
GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)NewsGator FetchLinks
Example user agent
NewsGator FetchLinks extension/0.2.0 (http://graemef.com)python-requests
Example user agent
python-requests/2.2.1 CPython/2.7.3 Linux/3.2.0-56-genericTestCrawler
Example user agent
ldspider (BTC 2011 crawl, harth@kit.edu, http://code.google.com/p/ldspider/wiki/Robots)GenieKnows
Example user agent
genieBot (http://64.5.245.11/faq/faq.html)Nutch
Example user agent
NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)
Example user agent
Research spider - ak1835@albany.eduYellowPages
Example user agent
yp-crawl@attinteractive.comOpenWebSpider
Example user agent
OpenWebSpider/0.6 (http://www.openwebspider.org)Scrubby
Example user agent
Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)Vagabondo
Example user agent
Mozilla/5.0 (compatible; Vagabondo/Wapspider; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)NewsGatorOnline
Example user agent
NewsGatorOnline/2.0 (http://www.newsgator.com)
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-sgh-f400) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0Feedfetcher-Google
Example user agent
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 19 subscribers; feed-id=13965549748850348809)DotBot
Example user agent
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)hivaBot
Example user agent
hivaBot/hivaBot-0.2 (info@gorgor.ir)Adbeat Bot
Example user agent
Mozilla/5.0 (X11; U; Linux x86; de-de) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s7233e) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0ICC-Crawler
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)TestCrawler
Example user agent
MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.htmlInternetSeer.com
Example user agent
InternetSeer.comAmiga
Example user agent
Mozilla/5.0 (AmigaOS; AmigaOS 53.26 PowerPC; rv:2.0.1) Gecko/20120327 Firefox/4.0.1Mail.Ru
Example user agent
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)Tweetmeme Bot
Example user agent
Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)E-Mail Siphon
Example user agent
EmailSiphoneZ Publish Link Validator
Example user agent
eZ Publish Link ValidatorXenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2cYOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://vaiqui.com)
Example user agent
Mozilla/5.0 (Java; U; en-us; htc star trek) UCBrowser8.3.0.154/69/352/UCWEB MobileJava Standard Library
Example user agent
Java/1.5.0_17ContextAd Bot
Example user agent
ContextAd Bot 1.0Picsearch Bot
Example user agent
psbot-page (+http://www.picsearch.com/bot.html)Amiga
Example user agent
Mozilla/5.0 (compatible; Origyn Web Browser; AmigaOS 4; PPC; U) AppleWebKit/532.0+ (KHTML, like Gecko, Safari/532.0+)lwp
Example user agent
lwp-trivial/1.35CrawlRobot
Example user agent
Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php)AdvBot
Example user agent
Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)Larbin
Example user agent
larbin_2.6.3 larbin2.6.3@unspecified.mailWada.vn Search Bot
Example user agent
Mozilla/5.0 (compatible; Wada.vn Vietnamese Image Search)Entireweb
Example user agent
Speedy Spider (http://www.entireweb.com)woriobot
Example user agent
Mozilla/5.0 (compatible; woriobot +http://worio.com)Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130207.001528 +http://webarchiv.cz/kontakty/)Crawlboy
Example user agent
CRAWL-E/0.6.4ROR Sitemap Generator
Example user agent
ROR Sitemap Generator (http://www.rorweb.com)
Example user agent
Mozilla/5.0 (Java; U; en-in; samsung-sgh-t528g) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0WI Job Roboter
Example user agent
Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)
Example user agent
Mozilla/5.0 (Java; U; en-in; UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0 SonyEricssonP1i/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1zitebot
Example user agent
Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.2-1188; https://myfriendica.net
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-c3510t) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.run4dom.com)SEOprofiler
Example user agent
Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )mbot
Example user agent
mbot v.1.16Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.3-dev-1191; https://friendica.xyzOWASP_SECRET_BROWSER
Example user agent
OWASP_SECRET_BROWSER/3.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b6520) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile, SAMSUNG-GT-B6520/1.0 Mozilla/4.0 UNTRUSTED/1.0Google Bot Mobile
Example user agent
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)DAWINCI ANTIPLAG SPIDER
Example user agent
Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)Amiga
Example user agent
Mozilla/5.0 (compatible; IBrowse 3.0; AmigaOS4.0)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-u800) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0Forum Poster
Example user agent
Forum Poster V3IIS Site Analysis Web Crawler
Example user agent
iisbot/1.0 (+http://www.iis.net/iisbot.html)Xenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2dgrub
Example user agent
Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-i637) UCBrowser8.3.0.154/69/444/UCWEB Mobile, SAMSUNG-SGH-I637/1.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/4.0 UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java 1.6.0_01; Windows XP 5.1 x86; en) ICEbrowser/v6_1_2SiteExplorer
Example user agent
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)Facebot
Example user agent
Facebot/1.0Steeler
Example user agent
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)RssBandit
Example user agent
RssBandit/1.9.0.1002
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-c300) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0Scopia Crawler
Example user agent
Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; F02A)BingPreview
Example user agent
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/538.14 (KHTML, like Gecko) BingPreview/1.0b360Spider
Example user agent
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider
Example user agent
Squrl Java/1.6.0_22WinHttp
Example user agent
Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)YOURLS
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://shelby.im)
Example user agent
IE6.0,Java(TM) 2 Runtime Environment, Standard EditionThe Old Reader
Example user agent
Mozilla/5.0 (compatible; theoldreader.com; 1 subscribers; feed-id=51413de5bd9279ea5002d75b)libwww
Example user agent
libwww-perl/5.76libwww
Example user agent
libwww-perl/5.805SocialSpider-Finder
Example user agent
SocialSpider-Finder/0.2
Example user agent
Mozilla/5.0 (Java; U; Id; lg-gt360 teleca) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0Echoping
Example user agent
Echoping/6.0.2RSSOwl
Example user agent
RSSOwl/2.2.1.201312301316 (X11; U; en)Northern Light Web Search
Example user agent
Mozilla/5.0 (compatible; NLCrawler/2.0.15; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko)PycURL
Example user agent
PycURL/7.22.0Baidu Union
Example user agent
Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)Google Bot
Example user agent
Googlebot (gocrawl v0.4)lwp
Example user agent
LWP LDMusicNews::LDNewsAPIR6 CommentReader
Example user agent
R6_CommentReader(www.radian6.com/crawler)bl.uk_lddc_bot
Example user agent
bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)FeedBlitz
Example user agent
FeedBlitz/1.0Crawler
Example user agent
Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-lgc300) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0Lipperhey SEO Service
Example user agent
Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)yoozBot
Example user agent
Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsungsgh-b7330) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile, SAMSUNG-GT-B7330/1.0 UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5330) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; xx; sonyericssonw150i) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0YOURLS
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://oerk.at)Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)Sogou Spider
Example user agent
sogou spiderpython-requests
Example user agent
python-requests/1.2.3 CPython/2.7.3 Linux/3.5.0-23-genericAboutUs Bot Johnny5
Example user agent
Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)Google Web Preview
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
Example user agent
Mozilla/5.0 (Java; U; xx; samsung-gt-e2202) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0msnbot-media
Example user agent
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)OpmlViewer network fetcher
Example user agent
OpmlViewer network fetcher version 1.0.0.0YOURLS
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://u.i4x3.tk)webnumbr Fetcher
Example user agent
webnumbrFetcher/1.0 (http://webnumbr.com/)fastbot crawler
Example user agent
fastbot crawler beta 4.0 (+http://www.fastbot.de)
Example user agent
Epic Empire/1.80 pgid=454af7cb350c2a1b tag=454af7cb350c2a1b cv=180011 hw=ASUS ME371MG; ME371MG; gzip; JavaHttp
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-a927) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile UNTRUSTED/1.0Google App Engine
Example user agent
AppEngine-Google; ( http://code.google.com/appengine)Google Bot
Example user agent
Googlebot/1.3Veooz Bot
Example user agent
Mozilla/5.0 ( compatible ; Veooz/1.0 ; +http://www.veooz.com/veoozbot.html )Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://www.wise-guys.nl/)Simpy
Example user agent
Simpy/1.1 (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)Friendica
Example user agent
Friendica 'Ginger' 3.3.1-1174; http://ccc.serpens.uberspace.de
Example user agent
Mozilla/5.0 (Java; U; es-la; lg-gs505) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB MobileYahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)IRLbot
Example user agent
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)Heritrix
Example user agent
Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)Accoona Bot
Example user agent
Accoona-Biz-Agent/1.1.1 crawler@accoona.com
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-gm210) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB MobileBingBot
Example user agent
Mozilla/5.0 (compatible;bingbot/2.0;+http://www.bing.com/bingbot.htm)IRLbot
Example user agent
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)AdSense Bot
Example user agent
Mediapartners-GoogleFeedDemon
Example user agent
FeedDemon/4.5 (http://www.feeddemon.com/; Microsoft Windows)Google Bot
Example user agent
GoogleEarth/7.1.0003.1255(Android;Android (HTC Sensation XE with Beats Audio Z715e-pyramid-user-4.0.3);de-DE;kml:2.2;client:Free;type:default)Google Feedfetcher
Example user agent
Mozilla/5.0 (compatible) Feedfetcher-Google;(+http://www.google.com/feedfetcher.html)Amiga
Example user agent
Mozilla/5.0 (AmigaOS; U; AmigaOS 1.3; en-US; rv:1.8.1.21) Gecko/20090303 SeaMonkey/1.1.15Google Bot
Example user agent
GoogleEarth/7.1.0002.2011(Android;Android (HTC One X-endeavoru-user-4.2.2);de-DE;kml:2.2;client:Free;type:default)pdrlabs Bot
Example user agent
Cloud mapping experiment. Contact research@pdrlabs.net
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru)ARC Reader
Example user agent
ARC Reader (http://arc.semsol.org/)BLEXBot
Example user agent
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)memoryBot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)libwww
Example user agent
libwww-perl/5.36Yahoo! Slurp
Example user agent
Mozilla/3.0 (Slurp.so/1.0; slurp@inktomi.com; http://www.inktomi.com/slurp.html)YOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://2u4.us)WEB.DE MailCheck
Example user agent
WEB.DE MailCheck/2.4.0Google PageSpeed Insights
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/4.0.5 Mobile/8A306 Safari/6531.22.7BabalooSpider
Example user agent
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)libwww
Example user agent
libwww-perl/5.75SiteSell
Example user agent
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)python-requests
Example user agent
python-requests/2.0.0 CPython/2.6.6 Linux/2.6.32-431.el6.x86_64Yahoo Pipes
Example user agent
Yahoo Pipes 2.0Zollard Worm
Example user agent
Mozilla/5.0 (compatible; Zollard; Linux)PagePeeker Screenshot Maker
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) domain.com Safari/538.1CMS Crawler
Example user agent
Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)NaverBot
Example user agent
NANaverNDrive/3.0.0(Android OS 4.1.2;PANTECH IM-A860S)GetLinkInfo Bot
Example user agent
Mozilla/5.0 (GetLinkInfo.com - http://www.getlinkinfo.com)Crawler
Example user agent
Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)Google Adwords DisplayAds WebRender
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36Nutch
Example user agent
Your Nutch Spider/Nutch-2.2.1google images relinker
Example user agent
greasemonkey – google images relinker (v 2.0)GenieKnows
Example user agent
genieBot ((http://64.5.245.11/faq/faq.html))Gigablast Search Engine
Example user agent
GigablastOpenSource/1.0inbound.li parser
Example user agent
inbound.li parserWebAlta Crawler
Example user agent
WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)HTTP Kit
Example user agent
http-kit/2.012345
Example user agent
12345CloudFlare AlwaysOnline
Example user agent
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34Adbeat Bot
Example user agent
Mozilla/5.0 (X11; U; Linux x86; nl-NL) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4libwww
Example user agent
libwww-perl/5.816uMBot
Example user agent
Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)Mail.Ru
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)Exalead CloudView
Example user agent
Mozilla/5.0 (compatible; ExaleadCloudView/5;)Werbefreie Deutsche Suchmaschine
Example user agent
Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html)YOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://clickand.co)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-gx200) UCBrowser8.2.0.132/69/352/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-gm750) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0YOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://pagn.info)FAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/6 (www.fastsearch.com)Scrapy
Example user agent
Scrapy/1.0.3.post6+g2d688cd (+http://scrapy.org)
Example user agent
Mozilla/5.0 (LG-C510 AppleWebKit/531 MMS/LG-MMS-V1.0/1.2 MediaPlayer/LGPlayer/1.0 Java/ASVM/1.1 Profile/MIDP-2.1),gzip(gfe),gzip(gfe)Internet Archive
Example user agent
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)Google Web Preview
Example user agent
Mozilla/5.0 (iPad; CPU OS 5_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/5.1 Mobile/9A334 Safari/7534.48.3ca-crawler
Example user agent
Mozilla/5.0 (compatible; ca-crawler/1.0)Website Explorer
Example user agent
Website Explorer/0.9.9.9Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)WebThumb
Example user agent
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0lbot
Example user agent
Mozilla/5.0 (lbot/2.6.4) lbot@mailme.not
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b7722) UCBrowser8.2.1.144/69/405/UCWEB Mobile UNTRUSTED/1.0Baiduspider
Example user agent
Baiduspider
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5380b) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0Tweetmeme Bot
Example user agent
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
Example user agent
Mozilla/5.0 (Java; U; En-us; Alcatel-ot-806d) UCBrowser8.3.1.161/70/465/UCWEB Mobile UNTRUSTED/1.0Sosospider
Example user agent
Sosospider+(+help.soso.com/webspider.htm)Short URL Checker
Example user agent
Short URL Checker ( http://www.kurld.com/ )Uipbot
Example user agent
uipbot/1.0 (uipbot@semasio.net)TurnitinBot
Example user agent
TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)metager2-verification-bot
Example user agent
Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)Squid
Example user agent
BREW-Applet/0x20068888 (BREW/3.1.5.20; DeviceId: 90086; Lang: zhcn) ucweb-squidProspectB2B Bot
Example user agent
Mozilla/5.0 (compatible; ProspectB2B/2.0; http://prospectb2b.com)SMTBot
Example user agent
Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-ks360 teleca) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s3370l) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0JobRoboter
Example user agent
JobRoboter/1.1Comodo Spider
Example user agent
Comodo Spider 1.1Teoma
Example user agent
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Example user agent
SEC-SGHE900/10 SAMSUNG JAVA CLIENT
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-a130) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0Google Web Snippet
Example user agent
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
Example user agent
Mozilla/5.0 (Java; U; ru; samsung-gt-s5380g) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0HyperCrawl
Example user agent
Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)Virtuoso
Example user agent
Mozilla/4.0 (compatible; Virtuoso)80Legs
Example user agent
Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620BOTW Spider
Example user agent
Mozilla/4.0 (compatible; BOTW Spider; http://botw.org)NewsGator
Example user agent
NewsGator/2.0 Bot (http://www.newsgator.com)Lycos
Example user agent
Lycos_Spider_(modspider)ObjectsSearch
Example user agent
ObjectsSearch/0.2 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)PagesInventory Bot
Example user agent
PagesInventory (robot +http://www.pagesinventory.com)Validator.nu/LV
Example user agent
Validator.nu/LV http://validator.w3.org/servicesNaverBot
Example user agent
nApps (LG-F160S; 16; naversearchapp; 5.0.1)akregator
Example user agent
Mozilla/5.0 (X11;) AppleWebKit/534.34 (KHTML, like Gecko) akregator/4.14.1 Safari/534.34CazoodleBot
Example user agent
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.jp/spider/)MJ12bot
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)Apple-PubSub
Example user agent
Apple-PubSub/65.28ht://Dig
Example user agent
htdig/3.1.5 (webmaster@online-medien.de)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; V702NK)MonoBot
Example user agent
Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)Amiga
Example user agent
Mozilla/5.0 (Amiga; PowerPC AmigaOS 4.1; Odyssey Web Browser; rv:1.23) AppleWebKit/538.1 (KHTML, like Gecko) OWB/1.23 Safari/538.1NetNewsWire
Example user agent
NetNewsWire/4.0.0 (Mac OS X; http://netnewswireapp.com/mac/; gzip-happy)Nutch
Example user agent
spider/Nutch-1.5.1 (spider; http://www.xxx.com)Google Image Proxy
Example user agent
Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-m5650) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB Mobile UNTRUSTED/1.0goose-extractor
Example user agent
Goose/1.0.25Web Glance
Example user agent
Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1FacebookExternalHit
Example user agent
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)lbot
Example user agent
Mozilla/5.0 (lbot/2.6.4) (lbot@mailme.not)
Example user agent
SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net),gzip(gfe),gzip(gfe)Yahoo Ad Monitoring
Example user agent
Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2.40Xenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2h
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-t669mkb) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0Slinky
Example user agent
Slinky v1.0 +http://dentedreality.com.au/projects/slinky/Googlebot-News
Example user agent
Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))Smartsite HTTPClient
Example user agent
Smartsite HTTPClient - Mozilla/4.0 (compatible; MSIE 6.0)Google App Engine
Example user agent
AppEngine-Google; (+http://code.google.com/appengine; appid: xbmacx)libwww
Example user agent
WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803TweetedTimes Bot
Example user agent
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0 (resolver); +http://tweetedtimes.com)Exabot
Example user agent
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)cURL
Example user agent
curl/7.8 (i386-redhat-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6b) (ipv6 enabled)HTTrack
Example user agent
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)Facebook
Example user agent
facebookplatform/1.0 (+http://developers.facebook.com)TestCrawler
Example user agent
Mozilla/5.0 (compatible; TestCrawler)
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s3653) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0Python-urllib
Example user agent
Python-urllib/2.4Y!J-AGENT
Example user agent
Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)Google Bot
Example user agent
GoogleProducer; (+http://goo.gl/7y4SX)80Legs
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620Sogou Web Spider
Example user agent
Sogou web spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)Scrubby
Example user agent
Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Example user agent
Mozilla/5.0 (Java; U; en-us; UCBrowser8.4.0.159/70/405/UCWEB Mobile SonyEricssonW950i/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1Google Bot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)libwww
Example user agent
libwww-perl/5.53Updated!
Example user agent
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)Zeus
Example user agent
Zeusbot/0.8.1 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)Nutch
Example user agent
SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)BingBot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0; )"
Example user agent
LG-GT540 Mozilla/5.0 (Linux; U; Android 1.6; xx; GT540; Build/Donut) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 Java/Jbed/7.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 MMS/LG-Android-MMS-V1.0/1.2
Example user agent
Mozilla/5.0 (Java; U; en-us; htc snap s521) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0FAST MetaWeb Crawler
Example user agent
FAST MetaWeb Crawler (helpdesk at fastsearch dot com)Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)reddit pic scraper
Example user agent
Mozilla / reddit pic scraper v0.8 (bklug@tyx.net)CyberPatrol
Example user agent
CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)Ichiro Bot
Example user agent
ichiro/1.0 (ichiro@nttr.co.jp)NetSeer Crawler
Example user agent
Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-d608) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0infoX-WISG
Example user agent
infoX-WISGEntireweb
Example user agent
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/)DomainAppender Bot
Example user agent
Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)Yahoo Slingstone
Example user agent
Yahoo:LinkExpander:SlingstoneIRLbot
Example user agent
IRLbot/2.0 (+http://irl.cs.tamu.edu/crawler)VoilaBot
Example user agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)Genderanalyzer
Example user agent
Genderanalyzer/1.0Amiga
Example user agent
Mozilla/5.0 (AmigaOS 4; PPC; InternetSurfboard; U; xx) AppleWebKit/533.3 (KHTML, like Gecko) InternetSurfboard/0.002 Safari/533.3
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5550) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0WordPress
Example user agent
WordPress.com; https://ramaransonvsthecult.wordpress.comPECL HTTP
Example user agent
PECL::HTTP/1.7.4 (PHP/5.4.4)BabalooSpider
Example user agent
BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.2-1188; https://f.tschlotfeldt.deWindows Phone Search
Example user agent
Windows Phone Search (Windows Phone OS 8.0;NOKIA;Nokia 520;8.0;10328)Y!J-AGENT
Example user agent
Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80)Norbert the Spider
Example user agent
Norbert the Spider(Burf.com)HiddenMarket
Example user agent
HiddenMarket-1.0-beta (www.hiddenmarket.net/crawler.php)Java Standard Library
Example user agent
Java1.2.2-JDeveloperYahoo-MMCrawler
Example user agent
Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)ScoutJet
Example user agent
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung sgh-d980) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobilepubliclibraryarchive Bot
Example user agent
Mozilla/5.0 (compatible; publiclibraryarchive.org/1.0; +crawl@publiclibraryarchive.org)websitepulse checker
Example user agent
websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)
Example user agent
Mozilla/5.0 (LG-GD880-Orange/V09d AppleWebKit/531 MMS/LG-MMS-V1.0/1.2 MediaPlayer/LGPlayer/1.0 Java/ASVM/1.1 Profile/MIDP-2.1 Configuration/CLDC-1.1)Windows Phone Search
Example user agent
Windows Phone Search (Windows Phone OS 8.0;NOKIA;Lumia 1520;8.0;10521)msnbot-media
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/534+ (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 MsnBot-Media /1.0bInternet Archive
Example user agent
ia_archiver/8.7 (Windows NT 5.0; )XSpider
Example user agent
XSpider 0.01;http://uncool.oicp.net/spider.htmlGoogle Wireless Transcoder
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14Adbeat Bot
Example user agent
adbeat_botMegaIndex Bot
Example user agent
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)newspaper
Example user agent
newspaper/0.0.9.8AcoonBot
Example user agent
Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)YahooCacheSystem
Example user agent
YahooCacheSystemLinkedInBot
Example user agent
LIAuthLibrary:0.0.2 com.linkedin.android:3.4.2 TCL_ALCATEL ONE TOUCH 4037T:android_4.4.2Squid
Example user agent
BREW-Applet/0x20068888 (BREW/3.1.5.20; DeviceId: 32056; Lang: zhcn) ucweb-squidYandexFavicons
Example user agent
Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)
Example user agent
Computer_and_Automation_Research_Institute_Crawler (crawler@info.ilab.sztaki.hu)Java Standard Library
Example user agent
Java1.3.1_04libwww
Example user agent
libwww-perl/5.835bnf.fr Bot
Example user agent
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)Amfibi
Example user agent
Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)Hatena Antenna
Example user agent
Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)alyze Bot
Example user agent
Mozilla/5.0 (compatible; http://alyze.info)FAST Enterprise Crawler
Example user agent
FAST Enterprise Crawler/5.3.4 (crawler@fast.no)Yahoo! Slurp China
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)EasyBib AutoCite
Example user agent
EasyBib AutoCite (http://content.easybib.com/autocite/)Google HTTP Client Library for Java
Example user agent
Google-HTTP-Java-Client/1.13.1-beta (gzip)Vagabondo
Example user agent
Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)Google Bot
Example user agent
GoogleEarth/7.1.0003.1255(Android;Android (HTC Desire X-protou-user-4.1.1);de-DE;kml:2.2;client:Free;type:default)SPIP
Example user agent
SPIP-3.0.20 (http://www.spip.net)linkCheck
Example user agent
Mozilla/5.0 (compatible; linkCheckV3.0)Python-urllib
Example user agent
Python-urllib/2.6test certificate info
Example user agent
test certificate infoOrangebot
Example user agent
Mozilla/5.0 (compatible; Orangebot/2.0; support.orangebot@orange.com)WDG HTML Validator
Example user agent
WDG_Validator/1.6.5Daumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;)Seznam Screenshot Generator
Example user agent
Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)Amazon CloudFront
Example user agent
Amazon CloudFrontmsnbot
Example user agent
msnbot/1.1 ( http://search.msn.com/msnbot.htm)SeeBot
Example user agent
seebot/2.0 (+http://www.seegnify.com/bot)Sproose
Example user agent
sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; htc touch 3g t3232) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/4.0 LG-GM750h Browser/IE8.12 MMS/LGMMSv1.0/1.2 Java/LGVM/1.1 Profile/MIDP-2.1 Configuration/CLDC-1.1 (compatible; MSIE 6.0; Windows CE; IEMobile 8.12; MSIEMobile 6.0)FAST-WebCrawler
Example user agent
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)Xerka
Example user agent
Xerka WebBot v1.0.0 [AIDO_CREA_ab]WbSrch Web Search
Example user agent
Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)cURL
Example user agent
curl/7.14.1 (i386-portbld-freebsd4.7) libcurl/7.14.1 OpenSSL/0.9.8 zlib/1.1.3Leadify Bot
Example user agent
Leadify/1.01 Page Staleness Detector
Example user agent
Mozilla/5.0 (Java; U; En-us; Samsung-sgh-a877) AppleWebKit/530.13 (KHTML, Like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB MobileSquid
Example user agent
BREW-Applet/0x20068888 (BREW/3.1.5.20; DeviceId: 90197; Lang: zhcn) ucweb-squidNutch
Example user agent
MySpider/Nutch-2.2libwww
Example user agent
amaya/10 libwww/5.4.0Crawlboy
Example user agent
Crawler 0.1,gzip(gfe),gzip(gfe)Ichiro Mobile Bot
Example user agent
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)InfegyAtlas
Example user agent
Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; collection@infegy.com)MSR-ISRCCrawler
Example user agent
MSR-ISRCCrawlergrapeFX
Example user agent
Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uksReleaseBot
Example user agent
Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/)OpenWebSpider
Example user agent
OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
Example user agent
SonyEricssonw760i/R4EA Java/SEMC-Java/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0, sonyericssonw760i
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5260) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0GreatNews
Example user agent
GreatNews/1.0
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s5380k) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0CrawlConvera
Example user agent
CrawlConvera0.1 (CrawlConvera@yahoo.com)Experibot
Example user agent
Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)akregator
Example user agent
Akregator/1.2.9; librss/remnantsWiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)LoadTimeBot
Example user agent
Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)Diffbot
Example user agent
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15Guzzle Http Client
Example user agent
GuzzleHttp/6.1.0 curl/7.38.0 PHP/7.0.0RC7NewsGatorOnline
Example user agent
NewsGatorOnline/2.0 (http://www.newsgator.com; 2 subscribers)Yahoo Mobile App
Example user agent
YahooMobile/1.0 (Yahoo! Search; 3.1.3) (Apple; iPhone; iPhone OS/7.1.2); YHOO_Search_App/3.1.3SeznamBot
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)TestCrawler
Example user agent
AppCodes crawler - looking for iOS app mentions. More info: support@appcodes.com. Robots.txt id: AppCodesCrawlerThumbnailAgent
Example user agent
ThumbnailAgentHTTP Service
Example user agent
HTTP Servicegvfs
Example user agent
gvfs/1.24.1007AC9 Crawler
Example user agent
Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)Powermarks
Example user agent
Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT)FAST-WebCrawler
Example user agent
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)FindLinks
Example user agent
findlinks/2.6 ( http://wortschatz.uni-leipzig.de/findlinks/)Amiga
Example user agent
Amiga-AWeb/3.4.167SETYPO3 Linkvalidator
Example user agent
Mozilla/5.0 TYPO3-linkvalidatorYaCy Bot
Example user agent
yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.htmlArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36
Example user agent
Mozilla/5.0 (Java; U; es-la; samsungsgh-i627) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile, SAMSUNG-SGH-i780/1.0 UNTRUSTED/1.0Google PageSpeed Insights
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25SiteBar
Example user agent
SiteBar/3.3.5 (Bookmark Server; http://sitebar.org/)GO HttpClient
Example user agent
Go 1.1 package httpLinkpadBot
Example user agent
Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-kc550 teleca) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/405/UCWEB Mobile UNTRUSTED/1.0semantic-visions.com crawler
Example user agent
Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)Quora Link Preview Bot
Example user agent
Quora Link Preview/1.0 (http://www.quora.com)gettor
Example user agent
gettor/0.5Prlog
Example user agent
Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)HubPages Bot
Example user agent
HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)Google
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-s3370b) --XIOZ-->8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; en-in; samsung-gt-b2710d) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0PHP
Example user agent
PHP/5.2.13Python-urllib
Example user agent
Python-urllib/2.5msnbot-media
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0bYahooSeeker
Example user agent
YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)Google Bot Mobile
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Example user agent
Mozilla/5.0 (Java; U; en-us; AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0 SonyEricssonG900/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1Crawlboy
Example user agent
CrawlerBoy Pinpoint.comMagpieRSS
Example user agent
MagpieRSS/0.7 ( http://magpierss.sf.net)Digg Bot
Example user agent
Digg Deeper/v1 (http://digg.com/about)YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://shortr.at)NewsBlur Feed Fetcher
Example user agent
NewsBlur Feed Fetcher - 0 subscribers - http://www.newsblur.com/site/1719739/nouvelles-cinema (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)Xaldon WebSpider
Example user agent
Xaldon WebSpider 2.7.b6YandexBot
Example user agent
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)AdSense Bot
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)libwww
Example user agent
Mozilla/4.04_(X11;_I;_OSF_3.2d)/1.0 libwww/4.0DXyleme
Example user agent
cosmos/0.9_(robot@xyleme.com)msnbot
Example user agent
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._pmoz.info ODP link checker
Example user agent
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)NewsBlur Feed Fetcher
Example user agent
NewsBlur Feed Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)Picsearch Bot
Example user agent
psbot-image (+http://www.picsearch.com/bot.html)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; SA3C)W3C-mobileOK/DDC
Example user agent
W3C-mobileOK/DDC-1.0 (see http://www.w3.org/2006/07/mobileok-ddc)Google Bot
Example user agent
Googlebot v2.1 (+http://www.google.com/bot.html) (http://www.openwebspider.org/)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5620) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0RedCarpet
Example user agent
RedCarpet/1.0 (http://www.redcarpet-inc.com/robots.html)Internet Archive
Example user agent
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b5310l) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0360Spider
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360SpiderSteeler
Example user agent
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)Open Vulnerability Assessment System
Example user agent
Mozilla/4.75 [en] (X11, U; OpenVAS)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-e2222l) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-sgh-a657) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0SimplePie
Example user agent
SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111118194710Zao
Example user agent
Zao/0.1 (http://www.kototoi.org/zao/)Integrity
Example user agent
integrity/4SeznamBot
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)URLChecker
Example user agent
URLCheckerSophora Linkchecker
Example user agent
Mozilla/5.0 (compatible; Sophora Linkchecker; http://www.subshell.com)PDFBot
Example user agent
PDFBot (crawler@pdfind.com)YioopBot
Example user agent
Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)YOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://sgfx.co)WiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)Java Standard Library
Example user agent
Java1.3.1_06libwww
Example user agent
libwww-perl/5.808
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-gd880) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0Twingly Recon
Example user agent
Mozilla/5.0 (compatible; Twingly Recon; twingly.com)
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/v0.1.5ALPHA SVN rev58; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.0; SN3F)DomainSigmaCrawler
Example user agent
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)Google Bot
Example user agent
Googlebot/2.1 ( http://www.googlebot.com/bot.html)YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://mrk.im)Baiduspider
Example user agent
Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)FyberSpider
Example user agent
FyberSpider (+http://www.fybersearch.com/fyberspider.php)BingBot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)Securepoint Content Filter
Example user agent
securepoint cfGuzzle Http Client
Example user agent
Guzzle/4.2.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.11Scrapy
Example user agent
Scrapy/0.24.6 (+http://scrapy.org)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-c3300k) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile, SAMSUNG-GT-C3300K/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0VSE Link Tester
Example user agent
VSE/1.0 (vsecrawler@hotmail.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-gb230) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0PHP
Example user agent
PHP/5.3.77WebZIP
Example user agent
WebZIP/7.0 (http://www.spidersoft.com)WebClient
Example user agent
WebClient/1.0Nutch
Example user agent
your sipder name/Nutch-1.7YOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://www.easylink.mobi)Google Adwords DisplayAds WebRender
Example user agent
Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; 001SC)Android Download Manager
Example user agent
AndroidDownloadManager/4.1.1 (Linux; U; Android 4.1.1; MID0714 Build/JRO03H)Google Image Search
Example user agent
Googlebot-Image/1.0Second Live Client
Example user agent
Mozilla/5.0 (Windows; U; ; en-US) AppleWebKit/533.3 (KHTML, like Gecko) SecondLife/3.7.23.297296 (Second Life Release; default skin) Safari/533.3YahooSeeker-Mobile
Example user agent
LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)Google PageSpeed Insights
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1Seznam Screenshot Generator
Example user agent
Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)BCKLINKS
Example user agent
BCKLINKS 1.0feedly App
Example user agent
FeedlyApp/1.0 (http://www.feedly.com)Superfeedr Bot
Example user agent
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touchTestCrawler
Example user agent
WEBB Crawler - see: http://badcheese.com/robots.htmlAdbeat Bot
Example user agent
Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21Voyager Bot
Example user agent
voyager/2.0 (http://www.kosmix.com/crawler.html)WiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)Liferea
Example user agent
Liferea/1.6.4 (Linux; en_US.UTF-8; http://liferea.sf.net/)WiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0ADmantX Platform Semantic Analyzer
Example user agent
ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.comYOURLS
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://fhort.com)Nutch
Example user agent
Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)libcurl
Example user agent
libcurl-agent/1.0MSN
Example user agent
MSNBot/Nutch-1.5.1BingPreview
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML; like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 BingPreview/1.0blibwww
Example user agent
NCSA_Mosaic/2.7b5 (X11;Linux 2.6.7 i686) libwww/2.12 modifiedjig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; 830CA)SimplePie
Example user agent
SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111015034325proximic
Example user agent
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)Add Catalog
Example user agent
Mozilla/5.0 (compatible; Add Catalog/2.1;)Facebookscraper
Example user agent
facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php)Google Bot
Example user agent
Googlebot/2.1(+http://www.googlebot.com/bot.html)
Example user agent
Mozilla/5.0 (Java; U; en-us; j2me) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile, SAMSUNG-SGH-I607/I607FG1 Mozilla/4.0/UNTRUSTED/1.0Owler Bot
Example user agent
Mozilla/5.0 (compatible; Owler/0.4; +; )YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://z-url.us)acapbot
Example user agent
Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
Example user agent
Mozilla/5.0 (Java; U; ru; samsung-gt-s5510) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UN
Example user agent
Mozilla/5.0 (Java; U; En-us; Samsung-gt-m8910) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0GurujiBot
Example user agent
Mozilla/5.0 GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)
Example user agent
Mozilla/5.0 (Java; U; ru; samsung-gt-b7722i) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; Id; lg-kf510 obigo) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0BoardReader Favicon Fetcher
Example user agent
BoardReader Favicon Fetcher /1.0 info@boardreader.comFeedValidator
Example user agent
FeedValidator/1.3QuerySeekerSpider
Example user agent
QuerySeekerSpider ( http://queryseeker.com/bot.html )Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dla-marbach.de)Nagios
Example user agent
check_http/v2.0 (nagios-plugins 2.0)FAST Enterprise Crawler
Example user agent
RB2B-bot v0.1 (Using Fast Enterprise Crawler 6 search@reedbusiness.com)LinkStats Bot
Example user agent
LinkStats BotHTTP Request
Example user agent
raynette_httprequest/1.0Feedfetcher-Google
Example user agent
FeedFetcher-Google; ( http://www.google.com/feedfetcher.html)Typhoeus
Example user agent
Typhoeus - https://github.com/typhoeus/typhoeus
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-te365 teleca) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0CheckSite Verification Agent
Example user agent
CheckSite Verification Agent (+http://www.checksite.us)Short URL Checker
Example user agent
GentleSource Short URL Checker ( http://www.gentlesource.com/ )Google HTTP Client Library for Java
Example user agent
Google-HTTP-Java-Client/1.17.0-rc (gzip)Google Bot
Example user agent
GoogleEarth/7.1.0003.1255(Android;Android (LG-E986-geefhd-user-4.1.2);de-DE;kml:2.2;client:Free;type:default)Comodo SSL Checker
Example user agent
Comodo SSL CheckerBrowseX
Example user agent
Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows)WiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)NaverBot
Example user agent
NANaverNDrive/3.0.0(Android OS 4.2.2;samsung SHV-E300L)NikiBot
Example user agent
niki-bot200PleaseBot
Example user agent
Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)Baiduspider
Example user agent
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)FAST-WebCrawler
Example user agent
FAST-WebCrawler/2.2.11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)Baidu Mobile Search
Example user agent
Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html)
Example user agent
ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86)Optimizer Bot
Example user agent
Mozilla/5.0 (compatible; Optimizer)CakePHP
Example user agent
CakePHPXenus Link Sleuth
Example user agent
Xenu Link Sleuth/1.3.8Twikle Bot
Example user agent
Twikle/1.0 , http://twikle.com , contact@twikle.comSogou Spider
Example user agent
New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98)Digincore Bot
Example user agent
Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.Google App Engine
Example user agent
AppEngine-GoogleDuckDuck App
Example user agent
DDG-Android-3.0.11Butterfly
Example user agent
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8WordPress
Example user agent
WordPress/3.7.1; http://wordpress.comNaverBot
Example user agent
NaverNDrive/3.2.1 (Android OS 4.3; samsung SHV-E210S)CareerBot
Example user agent
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)WordPress
Example user agent
WordPress 4.0-alpha Feed Client Mozilla/5.0 CompatibleNaverBot
Example user agent
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)MSRBOT
Example user agent
MSRBOT (http://research.microsoft.com/research/sv/msrbot)Baiduspider
Example user agent
Baiduspider+(+http://www.baidu.com/search/spider.htm)Ichiro Bot
Example user agent
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)EasyBib AutoCite
Example user agent
EasyBib AutoCite (http://autocite-info.citation-api.com/)AdsBot Google
Example user agent
AdsBot-Google ( http://www.google.com/adsbot.html)Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.0-1182; http://frndc.saschaschroeder.euMoreover
Example user agent
Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)GWPImages
Example user agent
Mozilla/5.0 (compatible; GWPImages/1.0)Nutch
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7RMSnapKit
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) RMSnapKit/1.0Java Standard Library
Example user agent
Java1.4.0_03Rippers
Example user agent
Mozilla/3.0 (compatible; Indy Library)WeSEE:Ads
Example user agent
WeSEE:Ads/PageBot (http://www.wesee.com/bot/)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-a797) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0libwww
Example user agent
libwww-perl/5.821Y!J-AGENT
Example user agent
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)Qwantify
Example user agent
Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/2,55IRLbot
Example user agent
IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)Sprinklr
Example user agent
Sprinklr 2.0RssBar
Example user agent
RssBar/1.29 (RssBar for unDonut 1.35)
Example user agent
Mozilla/4.5 (Screen=240x320x64K; InputMethod=PEN; Page=1M; Product=CASIO/CASSIOPEIA BE; HTML-level=3.2; Category=PDA; JavaScript=yes) (WorldTALK/2.2.24)IRLbot
Example user agent
IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler)MetaJobBot
Example user agent
Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)
Example user agent
JNLP/1.7.0 javaws/10.17.2.02 (<internal>) Java/1.7.0_17PiplBot
Example user agent
Mozilla/5.0 (compatible; PiplBot; http://www.pipl.com/bot/)MetaInspector
Example user agent
MetaInspector/3.0.0 (+https://github.com/jaimeiniesta/metainspector)FeedBlitz
Example user agent
FeedBlitz/1.0 (1 subscribers)T-Online Browser
Example user agent
T-Online BrowserBlog Search
Example user agent
Mozilla/4.0 (compatible; Blog Search;)
Example user agent
Mozilla/5.0 (Java; U; Id; lg-t300) UCBrowser8.2.1.144/69/444/UCWEB MobileVSE Link Tester
Example user agent
VSE/1.0 (testcrawler@hotmail.com)MSN
Example user agent
msnbot/2.0 (+http://search.msn.com/msnbot.htm)
Example user agent
SonyEricssonW20a/R7CA Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (Java; U; Id; sonyericssonw20a) UCBrowser8.3.0.154/70/444/UCWEB MobileUserAgent Bot
Example user agent
UserAgent/1.0Lucidworks Bot
Example user agent
Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com)Google Bot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Example user agent
Mozilla/5.0 (Java; U; Id; UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0 SonyEricssonW960i/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1WASALive Bot
Example user agent
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5250) UCBrowser8.2.0.132/69/352/UCWEB Mobile UNTRUSTED/1.0Sosospider
Example user agent
Sosospider+(+http://help.soso.com/webspider.htm)Nutch
Example user agent
NutchOSUOSL/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)nuSearch
Example user agent
NuSearch Spider www.nusearch.comgosquared-thumbnailer
Example user agent
gosquared-thumbnailer/1.0ArchiveBot
Example user agent
ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36Friendica
Example user agent
Friendica 'Ginger' 3.3.3-RC-1178; https://anthron.octans.uberspace.de/friendicaHatena Crawler
Example user agent
Hatena::Crawler/0.01Teoma
Example user agent
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)http_requester
Example user agent
http_requester/0.1AjaxSnapBot
Example user agent
Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html)Internet Archive
Example user agent
InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)Faraday
Example user agent
Faraday v0.9.2Scrubby
Example user agent
Mozilla/5.0 (compatible; Scrubby/2.2; +http://www.scrubtheweb.com/)ImplisenseBot
Example user agent
ImplisenseBot 1.0Hatena::Bookmark
Example user agent
Hatena::Bookmark/2.00 (Hatena::Bookmark; master;)WeSEE:Search
Example user agent
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)BingPreview
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)Entireweb
Example user agent
Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)Premium URL Shortener
Example user agent
Premium URL Shortener 4.0
Example user agent
HotJava/1.1.2 FCSNutch
Example user agent
NutchSpider/Nutch-1.4EventMachine HttpClient
Example user agent
EventMachine HttpClient
Example user agent
Mozilla/5.0 (Java; U; en-us; htc touch2 t3333) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile UNTRUSTED/1.0Microsoft-WebDAV
Example user agent
DavClnt
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-lg840g) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)crawler4j
Example user agent
crawler4j (http://code.google.com/p/crawler4j/)Wotbox
Example user agent
Wotbox/2.01 (+http://www.wotbox.com/bot/)WBSearchBot
Example user agent
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)Raven Link Checker
Example user agent
Raven Link CheckerNagios
Example user agent
check_http/v1.4.16 (nagios-plugins 1.4.16)YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://www.h-link.de)Getintent Crawler
Example user agent
GetintentCrawler getintent.com
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-c395) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB MobileWooRank
Example user agent
woobot/2.0ADmantX Platform Semantic Analyzer
Example user agent
ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.comWebRingChecker
Example user agent
WebRingChecker/1.71 LWP/5.79gooblog
Example user agent
gooblog/2.0 (http://help.goo.ne.jp/contact/)OpenBSD ftp
Example user agent
OpenBSD ftpSimplePie
Example user agent
SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20121030175911SEOprofiler
Example user agent
Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )fr-crawler
Example user agent
Mozilla/5.0 (compatible; fr-crawler/1.1)XmlSitemapGenerator
Example user agent
XmlSitemapGenerator - http://xmlsitemapgenerator.orgLinksManager
Example user agent
Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html)Baiduspider
Example user agent
Baiduspider ( http://www.baidu.com/search/spider.htm)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-a697) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.0; 940N)YOURLS
Example user agent
YOURLS v1.0-rc +http://yourls.org/ (running on http://epi.my)Tiny Tiny RSS
Example user agent
Tiny Tiny RSS/1.11.4c63934 (http://tt-rss.org/)
Example user agent
Mozilla/5.0 (Java; U; es-la; htc_s620) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0WebAlta Crawler
Example user agent
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
Example user agent
Mozilla/5.0 (Java; U; Id; lg-kf600) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0Proxy Gear Pro
Example user agent
Proxy Gear Pro/2.1 (Windows; N; Windows NT 6.1; en)newspaper
Example user agent
newspaper/0.0.7YOURLS
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://jmp.su)Gomez Site Monitor
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1PlantyNet
Example user agent
PlantyNet_WebRobot_V1.9 babo@plantynet.comVagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)Y!J-AGENT
Example user agent
Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)Xenus Link Sleuth
Example user agent
Xenu Link Sleuth/1.3.7WNMbot
Example user agent
WNMbot/0.4.0jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; HI3E)ConveraMultiMediaCrawler
Example user agent
ConveraMultiMediaCrawler/0.1 ( http://www.authoritativeweb.com/crawl)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung sgh-zx20) UCBrowser8.4.0.159/70/352/UCWEB Mobilekgbody
Example user agent
Mozilla/5.0 (compatible; kgbody/2.0; +http://kids.goo.ne.jp/tool/kgbody.php)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-s390g) UCBrowser8.3.1.161/69/405/UCWEB Mobile UNTRUSTED/1.0Discovery Bot
Example user agent
Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)libwww
Example user agent
NCSA Mosaic/2.6b1 (X11;UNIX_System_V 4.2 R4000) libwww/2.12 modified
Example user agent
Mozilla/5.0 (Java; U; Id; lg-c330) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0DomainCrawler
Example user agent
DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/botcURL
Example user agent
curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5Domnutch Bot
Example user agent
Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)YOURLS
Example user agent
YOURLS v1.7.1 +http://yourls.org/ (running on https://ivyti.es)
Example user agent
Mozilla/5.0 (Java; U; en-us; UCBrowser8.2.1.144/70/444/UCWEB Mobile SonyEricssonG700/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-c3262) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile, SAMSUNG-GT-C3262/C3262DDLHA Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0MailChimp.com
Example user agent
MailChimp.comJava Standard Library
Example user agent
Java1.4.0_01HooWWWer
Example user agent
HooWWWer/2.1.0 ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>hiit.fi)ShortLinkTranslate
Example user agent
ShortLinkTranslateSuperarama.com - BOT
Example user agent
Superarama.com - BOT/v.0.1REL Link Checker
Example user agent
REL Link Checker Lite 1.0Zend_Http_Client
Example user agent
Zend_Http_ClientRankFlex
Example user agent
RankFlex.com WebspiderNutch
Example user agent
CB/Nutch-1.7WWW-Mechanize
Example user agent
WWW-Mechanize/1.64Y!J-AGENT
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)ShopWiki
Example user agent
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)Googlebot-News
Example user agent
Googlebot-NewsICC-Crawler
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html; zurukko1552995316;)libwww
Example user agent
libwww-perl/5.69Bot Revolt
Example user agent
revoltRogerbot
Example user agent
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)Instant Payment Notification
Example user agent
PayPal IPN ( https://www.paypal.com/ipn )YOURLS
Example user agent
YOURLS v1.7.1 +http://yourls.org/ (running on http://sresc.io)
Example user agent
HuaweiC6100 Java/HWJa/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0Wayback Archive Bot
Example user agent
Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org)
Example user agent
Apache-HttpClient/UNAVAILABLE (java 1.4)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5wget
Example user agent
Wget/1.9.1Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.fi/eng/project.html)Squzer
Example user agent
Mozilla/5.0 (compatible; Squzer/0.0.1; +http://www.declum.com/)Crawlboy
Example user agent
CrawlerProcess (http://www.PowerMapper.com) /5.5.678.0Java Standard Library
Example user agent
Java/1.17TinEye Bot
Example user agent
TinEye/1.1 (http://tineye.com/crawler.html)Entireweb
Example user agent
Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)ayna-crawler
Example user agent
Mozilla/5.0 (compatible; ayna-crawler http://www.ayna.com)nuSearch
Example user agent
Nusearch Spider (compatible; MSIE 6.0)online link validator
Example user agent
online link validator (http://www.dead-links.com/)
Example user agent
SonyEricssonJ108i/R7EA Profile/MIDP-2.1 Configuration/CLDC-1.1 UNTRUSTED/1.0 UCWEB/2.0(Java; U; MIDP-2.0; xx; sonyericssonj108i) U2/1.0.0 UCBrowser/8.8.0.227 U2/1.0.0 MobileSvven Summarizer Bot
Example user agent
Svven-SummarizerBingPreview
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 (KHTML, like Gecko) BingPreview/1.0bLongURL Bot
Example user agent
LongURL APIQuickiWiki Bot
Example user agent
QuickiWiki/1.0 (http://www.quickiwiki.com; admin@quickiwiki.com) BasedOnNodeJs/0.10.23lwp
Example user agent
lwp-trivial/1.38Safari RSS
Example user agent
AppleSyndication/56.1
Example user agent
Mozilla/5.0 (Java; U; en-in; samsung-gt-s7250d) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0Google Toolbar
Example user agent
Mozilla/4.0 (compatible; GoogleToolbar 7.5.5111.1712; Windows XP 5.1; MSIE 8.0.6001.18702)Daumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0Save n Keep Screenshot Bot
Example user agent
SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com )Google Image Proxy
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com)Yahoo-MMCrawler
Example user agent
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; N905i)AboutTheDomain Bot
Example user agent
aboutthedomain
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung m2710) UCBrowser8.2.1.144/69/352/UCWEB MobileExploratodo Bot
Example user agent
Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.comBingPreview
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0bYOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://urlmicro.me)Feedfetcher-Google
Example user agent
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 2 subscribers; feed-id=17860707833818568603)Friendica
Example user agent
Mozilla/5.0 (compatible; Friendica)Begun Advertising Bot
Example user agent
Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)YahooCacheSystem
Example user agent
YahooCacheSystem; YahooWebServiceClient
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-sgh-t669) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobilebl.uk_lddc_bot
Example user agent
Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)BlueDragon
Example user agent
BlueDragoncURL
Example user agent
curl/7.19.6 (i686-pc-cygwin) libcurl/7.19.6 OpenSSL/0.9.8n zlib/1.2.3 libidn/1.18 libssh2/1.2SeznamBot
Example user agent
SeznamBot/3.0 (+http://fulltext.sblog.cz/)PHP
Example user agent
PHPNett.io bot
Example user agent
Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/)Mail.Ru
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)Linkdex Bot
Example user agent
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)LinkedInBot
Example user agent
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 http://www.linkedin.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-u900) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB MobileW3C Unicorn
Example user agent
W3C_Unicorn/1.0 (http://validator.w3.org/services)Google Structured-Data TestingTool
Example user agent
Mozilla/5.0 (compatible; X11; Linux x86_64; Google-StructuredDataTestingTool; +http://www.google.com/webmasters/tools/richsnippets)NerdyBot
Example user agent
NerdyBotSiteBar
Example user agent
SiteBar/3.3.8 (Bookmark Server; http://sitebar.org/)BlogPulse
Example user agent
BlogPulse (ISSpider-3.0)Larbin
Example user agent
LarbinWebCrawler (internet@bredband.net)Amiga
Example user agent
OWB/3.12 (AmigaOS 4.1; ppc; U)Python
Example user agent
pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp)DBot
Example user agent
Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com)Baiduspider
Example user agent
baiduspider-mobile-gatenetEstate NE Crawler
Example user agent
netEstate NE Crawler (+http://www.sengine.info/)woriobot
Example user agent
Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)Traackr
Example user agent
Traackr.comMS OPD
Example user agent
Microsoft Office Protocol DiscoveryBingBot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.1; +http://www.bing.com/bingbot.htm)Ad Muncher
Example user agent
Ad Muncher v4.93.33707/4957WordPress
Example user agent
WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 23.19.176.193WinkBot
Example user agent
WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)Amiga
Example user agent
Mozilla/5.0 (compatible; Origyn Web Browser; U; AmigaOS 4.0; xx) AppleWebKit/534.0 (KHTML, like Gecko, Safari/534.0) OWB/3.32OpenWebSpider
Example user agent
OpenWebSpider (link collector; http://links.port30.se/cia.html) v (http://www.openwebspider.org/)BingPreview
Example user agent
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0) BingPreview/1.0bIPv4Scan
Example user agent
IPv4Scan (+http://ipv4scan.com)Internet Archive Special Archiver
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)DBot
Example user agent
Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws)WEBDAV Client
Example user agent
WEBDAV ClientGoogle Bot Mobile
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)IstellaBot
Example user agent
Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)Locubot
Example user agent
Locubot (compatible; Googlebot; msnbot)Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.3-dev-1191; http://toktan.orgWindows-Update-Agent
Example user agent
Windows-Update-Agent/7.9.9600.17729 Client-Protocol/1.21Internet Archive Special Archiver
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)Charlotte
Example user agent
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@beta.spider.com)SMTBot
Example user agent
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Example user agent
Mozilla/5.0 (Java; U; en-in; lg-s310) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0vBulletin SEO Bot
Example user agent
Mozilla/4.0 (vBSEO; http://www.vbseo.com)WiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)ZIBB Crawler
Example user agent
ZIBB Crawler (email address / WWW address)Searchteq Bot
Example user agent
stq_bot (+http://www.searchteq.de)Genieo Web Filter
Example user agent
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)Icarus6j
Example user agent
Icarus6j - (contact: phil@icarus6.com)Texis Webscript
Example user agent
Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)RedCarpet
Example user agent
RedCarpet/2.1 CFNetwork/672.0.2 Darwin/14.0.0TLSProber
Example user agent
TLSProber/0.8libwww
Example user agent
NEC MosaicView/1.0 libwww/2.15_SpyglassInAGist URL Resolver
Example user agent
InAGist URL Resolver (http://inagist.com)Sogou Image Crawler
Example user agent
Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)Google Bot
Example user agent
Google-YouTubeSample/1.0 Google-HTTP-Java-Client/1.7.0-beta (gzip) Google-HTTP-Java-Client/1.7.0-beta (gzip)Apt HTTP Transport
Example user agent
Debian APT-HTTP/1.3 (1.0.1ubuntu2)Google Keyword Suggestion
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16GurujiBot
Example user agent
Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)Baidu Image Search
Example user agent
Baiduspider-image+(+http://www.baidu.com/search/spider.htm)Liferea
Example user agent
Liferea/1.10-RC1 (Linux; en_GB.UTF-8; http://liferea.sf.net/)Friendica
Example user agent
Friendica 'Ginger' 3.3.1-1174; http://friendica.offerman.comlibwww
Example user agent
libwww-perl/5.820Alexabot
Example user agent
Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)Google Bot
Example user agent
Google (+https://developers.google.com/+/web/snippet/)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; DM003SH)Screenshot Bot
Example user agent
pr-cy.ru Screenshot Botlwp
Example user agent
LWP::Simple/5.800Screaming Frog SEO Spider
Example user agent
Screaming Frog SEO Spider/1.10CCBot
Example user agent
CCBot/2.0 (http://commoncrawl.org/faq/)Python-urllib
Example user agent
Python-urllib/3.1Grammarly
Example user agent
Grammarly/1.0 (http://www.grammarly.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s3370) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0Amiga
Example user agent
IBrowse/2.4oem (AmigaOS V53; PPC; 68K build)Windows-RSS-Platform
Example user agent
Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 6.3)grub
Example user agent
grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)msnbot
Example user agent
msnbot/0.9 (+http://search.msn.com/msnbot.htm)WordPress
Example user agent
WordPress.com; http://fleurdelyz.compython-requests
Example user agent
python-requests/2.5.1 CPython/2.7.6 Linux/3.13.0-36-genericRSSOwl
Example user agent
RSSOwl/2.2.1.201312301314 (Windows; U; en)Vagabondo
Example user agent
Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)
Example user agent
Mozilla/5.0 (Java; U; ru; samsung-gt-s5780d) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0BuiBui-Bot
Example user agent
BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com)FDSE Robot
Example user agent
Mozilla/4.0 (compatible: FDSE robot)
Example user agent
Mozilla/5.0 (Java; U; en-in; lg-gu220) UCBrowser8.2.0.132/69/352/UCWEB Mobile UNTRUSTED/1.0Yahoo Mobile App
Example user agent
YahooMobile/1.0 (Yahoo! Search; 3.1.2) (Apple; iPod touch; iPhone OS/7.1.1); YHOO_Search_App/3.1.2
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s5253) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0AddThis.com robot
Example user agent
AddThis.com robot tech.support@clearspring.comjig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; PT34)
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s3653w) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0 UNTRUSTED/1.0msnbot
Example user agent
msnbot/2.1Yahoo Link Preview
Example user agent
Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)Mail.Ru
Example user agent
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)AccServer
Example user agent
AccServer[HOGANMIDSCHOOL]/5.0.2.20(24517) 64-bit HTTP-Agent
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-m8910b) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB MobileAboutUs Bot
Example user agent
AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; 823P)
Example user agent
Mozilla/5.0 (Java; U; Id; lg-t310i) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0msnbot
Example user agent
msnbot/2.0b ( http://search.msn.com/msnbot.htm)Twisted PageGetter
Example user agent
Twisted PageGetterLipperhey Kaus Australis
Example user agent
Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)Wotbox
Example user agent
Wotbox/0.7-alpha (bot@wotbox.com; http://www.wotbox.com)ht://Dig
Example user agent
htdig/3.1.5 (root@localhost)WhatWeb Web Scanner
Example user agent
WhatWeb/0.4.8
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s3572) UCBrowser8.3.0.154/69/352/UCWEB Mobile, SAMSUNG-GT-S3572/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0BigBozz - Financial Search
Example user agent
BigBozz/BigBozz-2.2.1WordPress
Example user agent
WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 93.61.114.253ltx71 Bot
Example user agent
ltx71 - (http://ltx71.com/)Zookabot
Example user agent
Zookabot/2.4; +http://zookabot.comWotbox
Example user agent
Wotbox/2.01 ( http://www.wotbox.com/bot/)DotBot
Example user agent
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-a767) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0NoteTextView
Example user agent
NoteTextViewXenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2iR6 FeedFetcher
Example user agent
R6_FeedFetcher(www.radian6.com/crawler)Python-urllib
Example user agent
Python-urllib/1.16Riddler
Example user agent
Riddler (http://riddler.io/about)WordPress
Example user agent
WordPress/3.8.1; http://www.samsung-blog-news.deDigg Feed Fetcher
Example user agent
Digg Feed Fetcher 1.0 (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-s5628) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0SoftMaker Fetch
Example user agent
SoftMaker Fetch/1.0Google Bot
Example user agent
Mozilla (Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html))WeSEE Bot
Example user agent
WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)Google Image Proxy
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com GoogleImageProxy)Nutch
Example user agent
NutchCVS/0.06-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)SMTBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)Xenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2eIncoming Links - WordPress Plugin
Example user agent
MonitorBacklinksWP (+http://monitorbacklinks.com/blog/incoming-links/)LinksManager
Example user agent
Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)Ichiro Mobile Bot
Example user agent
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)Entireweb
Example user agent
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)Tweetmeme Bot
Example user agent
Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)Slackbot
Example user agent
Slackbot (+https://api.slack.com/robots)Pompos
Example user agent
Pompos/1.2 http://pompos.iliad.frmsnbot-media
Example user agent
msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)Cyberduck
Example user agent
Cyberduck/4.6.4 (16610) (Windows XP/5.1) (x86)ZE.bz
Example user agent
ZeBot_www.ze.bz (ze.bz@hotmail.com)Socialradar Bot
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com)TweetedTimes Bot
Example user agent
Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://ezurl.me)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-b3410) UCBrowser8.3.0.154/69/444/UCWEB Mobilelink_thumbnailer
Example user agent
Ruby, link_thumbnailerht://Dig
Example user agent
htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)Netvibes
Example user agent
Netvibes (http://www.netvibes.com)XML Sitemaps Generator
Example user agent
Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0Nutch
Example user agent
asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)NetzCheckBot
Example user agent
NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com)
Example user agent
SonyEricssonJ108i/R7EA Profile/MIDP-2.1 Configuration/CLDC-1.1 UNTRUSTED/1.0 UCWEB/2.0 (Java; U; MIDP-2.0; xx; SonyEricssonJ108i) U2/1.0.0 UCBrowser/9.4.0.342 U2/1.0.0 MobileYahoo-Test
Example user agent
Mozilla/5.0 (Yahoo-Test/4.0 ysm-keystone-crawl-support@yahoo-inc.com)NetSurf
Example user agent
NetSurf/2.0 (RISC OS; armv5l)Esribot
Example user agent
Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)Windows-Update-Agent
Example user agent
Windows-Update-Agent/7.9.9600.18094 Client-Protocol/1.21Scrubby
Example user agent
Scrubby/2.2 (http://www.scrubtheweb.com/)msnbot-media
Example user agent
msnbot-media/1.0 ( http://search.msn.com/msnbot.htm)Baidu Mobile Search
Example user agent
Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spi_der.html)Sistrix Crawler
Example user agent
SistrixDolphin smalltalk http client
Example user agent
Dolphin http client/10.3.0(280) (Android)optivo NetHelper
Example user agent
optivo(R) NetHelperpub-crawler
Example user agent
Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)BlekkoBot
Example user agent
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)Pearltrees Bot
Example user agent
PearltreesLinks4US-Crawler
Example user agent
Links4US-Crawler, ( http://links4us.com/)
Example user agent
Mozilla/5.0 (Java; U; Id; lg-gt365) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0Spiderlytics
Example user agent
Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com)Java Standard Library
Example user agent
Java1.2.1Girafabot
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; Girafabot [girafa.com])Python-urllib
Example user agent
Python-urllib/2.0a1MarketwireBot
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com)NaverBot
Example user agent
NaverCafe/3.1.4 (Android OS 4.1.2; PANTECH IM-A850S; NaverCafe; 3.1.4)Java Standard Library
Example user agent
Java/1.7Athena Site Analyzer
Example user agent
Athena Site Analyzer (bestsiteanalyzer.com)Entireweb
Example user agent
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)MJ12bot
Example user agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b7320l) UCBrowser8.2.1.144/69/405/UCWEB Mobile, SAMSUNG-GT-B7320L/1.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/4.0 UNTRUSTED/1.0SEOstats
Example user agent
SEOstats 2.1.0 https://github.com/eyecatchup/SEOstatsDaumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0IRLbot
Example user agent
IRLbot/2.0 ( http://irl.cs.tamu.edu/crawler)Vagabondo
Example user agent
Mozilla/4.0 (compatible; Vagabondo/2.3; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)LOOQ
Example user agent
LOOQ/0.1 alfa (LOOQ Crawler for european sites; http://looq.eu; root (at) looq dot eu)TweetedTimes Bot
Example user agent
Mozilla/5.0 (compatible; +http://tweetedtimes.com)
Example user agent
RubyW3C I18n Checker
Example user agent
W3C_I18n-Checker/1.0 (http://validator.w3.org/services)Google Bot
Example user agent
GoogleEarth/7.1.0002.2011(Android;Android (LG-P875-l1e-user-4.1.2);de-DE;kml:2.2;client:Free;type:default)Shrook
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/434lDiscoverEd
Example user agent
DiscoverEd/Nutch-1.7 (OER search crawler; http://wiki.creativecommons.org/DiscoverEd; webmaster@creativecommons.org)Google Bot
Example user agent
googlebot
Example user agent
UCWEB/2.0 (Java; U; MIDP-2.0; 11; GT-S5230) U2/1.0.0 UCBrowser UCVN.MOBI/9.4.1.377 U2/1.0.0 Mobile UNTRUSTED/1.0Unity Web Player
Example user agent
UnityPlayer/4.3.2f1 (http://unity3d.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-s3100) UCBrowser8.3.0.154/69/444/UCWEB Mobile
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-s5780) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0 UNTRUSTED/1.0YandexImages
Example user agent
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)hivaBot
Example user agent
hivaBot(http://gorgor.ir and info@gorgor.ir)/hivaBot-0.2NaverBot
Example user agent
NaverWebToon/1.4.4(Android OS 4.0.4;samsung YP-GP1)Zao-Crawler
Example user agent
Zao-CrawlerElmer
Example user agent
Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)libwww
Example user agent
libwww-perl/5.48Rogerbot
Example user agent
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung c5212) UCBrowser8.3.0.154/69/352/UCWEB MobileIconSurf
Example user agent
IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)libwww
Example user agent
libwww-perl/5.63Envolk
Example user agent
envolk/1.7 ( http://www.envolk.com/envolkspider.html)Friendica
Example user agent
Friendica 'Lily of the valley' 3.4.2-1188; https://friends.tausys.deAhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/Sogou Web Spider
Example user agent
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Example user agent
Mozilla/5.0 (Java; U; en-id; samsung gt-c3303i) UCBrowser8.2.1.144/69/444/UCWEB MobileGirafabot
Example user agent
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)WebmasterCoffee
Example user agent
Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about)
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s3850) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0memoryBot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)Lipperhey SEO Service
Example user agent
Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)YandexMetrika
Example user agent
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)Accoona-AI-Agent
Example user agent
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)Heritrix
Example user agent
Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters)GrapeshotCrawler
Example user agent
Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-gs390) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0ICC-Crawler
Example user agent
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)80Legs
Example user agent
Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620NameProtect
Example user agent
abot/0.1 (abot; http://www.abot.com; abot@abot.com)SeznamBot
Example user agent
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)MSRBOT
Example user agent
MSRBOT (http://research.microsoft.com/research/sv/msrbot/)cars-app-browser
Example user agent
cars-app-browserXING Contenttabreceiver
Example user agent
XING-contenttabreceiver/2.0Nutch
Example user agent
ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com)DomainScan Server Monitoring
Example user agent
Mozilla/5.0 (compatible; DomainSCAN/2.0; +http://domainscan.info/robot/)YodaoBot
Example user agent
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )vBulletin Bot
Example user agent
vBulletin via PHPYOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://www.vrbas.be/url)QuickFinder
Example user agent
QuickFinder CrawlerDaumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0Paper.li Bot
Example user agent
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)MojeekBot
Example user agent
Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)Java Standard Library
Example user agent
Java1.4.0Astute Social
Example user agent
Astute SRMmsnbot
Example user agent
MSNBOT/0.1 (http://search.msn.com/msnbot.htm)Google Web Preview
Example user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)PagesInventory Bot
Example user agent
PagesInventory (robot http://www.pagesinvenotry.com)AOLbot
Example user agent
Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/)WebCorp
Example user agent
Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)Omea Reader
Example user agent
JetBrains Omea Reader 2.2 (http://www.jetbrains.com/omea/reader/)XmlSitemapGenerator
Example user agent
XmlSitemapGenerator - http://XmlSitemapGenerator.orgSymfony RSS reader
Example user agent
sfFeedReader/0.9
Example user agent
UNTRUSTED/1.0 Mozilla/5.0 (Java; U; es-la; samsung gt-s3770) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB MobileLarbin
Example user agent
08_800w_web1 (larbin2.6.3@unspecified.mail)newspaper
Example user agent
newspaper/0.0.6Pete-Spider
Example user agent
Pete-Spider/1.1n4p_bot
Example user agent
n4p_bot (crawler@n4p.com)wget
Example user agent
Wget/1.6Amiga
Example user agent
Mozilla/4.0 (compatible; MSIE 5.5; AmigaOS4.0) IBrowse 2.3ZnajdzFoto/ImageBot
Example user agent
ZnajdzFoto/Image 2.0MojeekBot
Example user agent
Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)EBSCO EJS Content Server
Example user agent
EBSCO EJS Content ServerIchiro Bot
Example user agent
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)Another Web Mining Tool
Example user agent
Mozilla/5.0 (compatible; Another Web Mining Tool 1.0; +none; awmt).NET Framework CLR
Example user agent
MS Web Services Client Protocol 1.0.3705.0newsme
Example user agent
newsme/1.0; feedback@news.meAdaxas Spider
Example user agent
Adaxas Spider (http://www.adaxas.net/)Website Thumbnail Generator
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)SPDYCheck SPDY Protocol Tester
Example user agent
SPDYCheck SPDY Protocol Tester, see http://spdycheck.org/about.htmlSafeSearch microdata crawler
Example user agent
SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)FacebookExternalHit
Example user agent
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)MagpieRSS
Example user agent
MagpieRSS/0.72 \(+http://magpierss.sf.net\)Embedly
Example user agent
Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)Shrook
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/1290ymemoryBot
Example user agent
Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html)R6 FeedFetcher
Example user agent
R6_FeedFetcher_(www.radian6.com/crawler)Butterfly Robot
Example user agent
Mozilla/5.0 (Macintosh; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8WIRE
Example user agent
WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler)Enigma Browser
Example user agent
Enigma Browsermeanpathbot
Example user agent
Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)Google PageSpeed Insights
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36
Example user agent
Mozilla/5.0 (Java; U; ru; samsung-s5610) UCBrowser8.3.0.154/69/444/UCWEB MobileJoobleBot
Example user agent
JoobleBot (compatible; http://jooble.org/jooble-bot)webceo Bot
Example user agent
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; htc touch diamond p3700) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-lg420g[tf012125009345399008080015075895332] obigo) UCBrowser8.3.0.154/70/352/UCWEB MobileNaverBot
Example user agent
Yeti/1.0 (+http://help.naver.com/robots/)Pingdom
Example user agent
Pingdom.com_bot_version_1.4_(http://www.pingdom.com)Ilse
Example user agent
INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)BingBot
Example user agent
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)wget
Example user agent
Wget/1.10.2TinEye Bot
Example user agent
TinEyeSlackbot-Link-Expanding
Example user agent
Slackbot-LinkExpanding (+https://api.slack.com/robots)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-i900) UCBrowser8.2.0.132/69/452/UCWEB Mobile, SAMSUNG-SGH-i900/1.0 UNTRUSTED/1.0Google-Sitemaps
Example user agent
Google-Sitemaps/1.0Spinn3r RSS Aggregator
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19Python-urllib
Example user agent
Python-urllib/1.17BecomeBot
Example user agent
Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)MetaGeneratorCrawler
Example user agent
MetaGeneratorCrawler/1.3.8 (www.metagenerator.info)DMCA Page Protection Crawling Service
Example user agent
DMCA.com Page Protection Crawling ServiceKomodiaBot
Example user agent
Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0libwww
Example user agent
NCSA Mosaic/2.6b1 [en-gb] (X11;UNIX_System_V 4.2 R4000) libwww/2.12 modifiedYOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://u.vsil.org)Y!J-AGENT
Example user agent
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248)Google Bot
Example user agent
mozilla/5.0 (iphone; cpu iphone os 8_3 like mac os x) applewebkit/600.1.4 (khtml, like gecko) version/8.0 mobile/12f70 safari/600.1.4 (compatible; googlebot/2.1; +http://www.google.com/bot.html)LinksCrawler
Example user agent
LinksCrawler 0.1betaAdsBot Google
Example user agent
AdsBot-GoogleJava Standard Library
Example user agent
Java1.3.1YOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://xurl.co)URLAppendBot
Example user agent
Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)wsr-agent
Example user agent
wsr-agent/1.0Cydral Web Image Search
Example user agent
CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com)AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)EveryoneSocialBot
Example user agent
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)
Example user agent
Mozilla/4.0 LG-GM750Q Browser/IE8.12 MMS/LGMMSv1.0/1.2 Java/LGVM/1.1 Profile/MIDP-2.1 Configuration/CLDC-1.1 (compatible; MSIE 6.0; Windows CE; IEMobile 8.12; MSIEMobile 6.0)Sogou Web Spider
Example user agent
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
Example user agent
Mozilla/5.0 (Java; U; Id; lg-a230) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0Apache Synapse
Example user agent
Mozilla/4.0 (compatible; Synapse)Google Bot
Example user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36ExactSeek Crawler
Example user agent
ExactSeek Crawler/0.1YaCy Bot
Example user agent
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.html
Example user agent
Mozilla/5.0 (Java; U; en-in; samsung-gt-c3300k-orange) UCBrowser8.4.0.159/69/352/UCWEB Mobile, SAMSUNG-GT-C3300K-ORANGE/C3300KBVJG3 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0AdsBot Google-Mobile
Example user agent
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36Envolk
Example user agent
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)YandexBot
Example user agent
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: twitter-mirror),gzip(gfe),gzip(gfe),gzip(gfe)Crawlboy
Example user agent
crawl/0.4 langcrawl/0.1TestCrawler
Example user agent
Yeti-FeedItemCrawler/1.0 (NHN Corp.;+http://help.naver.com/robots/)Google Bot Mobile
Example user agent
DoCoMo//2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)Marketing Grader
Example user agent
HubSpot Marketing GraderWebbot.ru
Example user agent
Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-a667) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0wget
Example user agent
Wget/1.15 (linux-gnu)Jakarta Commons HttpClient
Example user agent
Jakarta Commons-HttpClient/3.0CatchBot
Example user agent
CatchBot/1.0; +http://www.catchbot.com
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-m3710) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0oBot
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)libwww
Example user agent
libwww-perl/6.05Heritrix
Example user agent
flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5750e) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0Sogou Push Spider
Example user agent
Sogou Push Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)BitlyBot
Example user agent
bitlybotAmiga
Example user agent
Mozilla/5.0 (Macintosh; PowerPC AmigaOS 4.1; Odyssey Web Browser; rv:1.16) AppleWebKit/535.14 (KHTML, like Gecko) OWB/1.16 Safari/535.14Google Bot
Example user agent
Googlebot/2.1 (+http://www.googlebot.com/bot.html)Website Thumbnail Generator
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generatmsnbot
Example user agent
msnbot/0.11 ( http://search.msn.com/msnbot.htm)akregator
Example user agent
Akregator/4.12.3; syndication SUSEScrapy
Example user agent
Scrapy/0.20.1 (+http://scrapy.org)FeedBurner
Example user agent
FeedBurner/1.0 (http://www.FeedBurner.com)RSS2HTML
Example user agent
FeedForAll rss2html.php v2PWBot
Example user agent
Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw)Nutch
Example user agent
testnutch/Nutch-1.8
Example user agent
Mozilla/5.0 (Java; U; en-in; samsung-gt-s8530) UCBrowser8.2.1.144/70/352/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-sgh-a897) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-gr700) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0Mignify Bot
Example user agent
Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)MQbot
Example user agent
MQbot http://metaquerier.cs.uiuc.edu/crawlerGoogle Bot
Example user agent
Googlebot/2.2libwww
Example user agent
libwww-perl/5.50MonkeyCrawl
Example user agent
MonkeyCrawl/0.05 (MonkeyCrawl; http://www.monkeymethods.org; )MerchantCentricBot
Example user agent
Mozilla/5.0 (compatible; MerchantCentricBot/1.0)DomainScan Server Monitoring
Example user agent
DomainSCANGoogle Bot Mobile
Example user agent
KDDI-CA34 UP.Browser/6.2.0.10.2.2(GUI)MMP/2.0 (compatible; KDDI-Googlebot-Mobile/2.1; http://www.google.com/bot.html)DomainsDB
Example user agent
DomainsDB.net MetaCrawler v.0.9.7b (http://domainsdb.net/)YOURLS
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://rlink.eu)Crawlboy
Example user agent
CrawlWave/1.2 (crawlwave[at]circular.gr http://www.spiderwave.aueb.gr/Google Bot
Example user agent
Googlebot/2.1 (+http://www.google.com/bot.html)GLBot
Example user agent
Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co)WordPress
Example user agent
WordPress.com; https://positivecups.wordpress.comNetvibes
Example user agent
Netvibes (http://www.netvibes.com/; 8 subscribers; feedID: 2244192)
Example user agent
Mozilla/5.0 (Java; U; es-la; lg-gu295) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0W3C-checklink
Example user agent
W3C-checklink/4.0 [4.4] libwww-perl/5.803
Example user agent
Mozilla/5.0 (Java; U; en-us; htc touch viva t2223) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0Yahoo! Slurp
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)libwww
Example user agent
NCSA_Mosaic/2.6L10N+ (X11;SunOS 5.6 sun4u) libwww/2.12 modifiedPhoton
Example user agent
Photon/1.0URLfilterDB Crawler
Example user agent
Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0PEAR-PHP
Example user agent
PEAR HTTP_Request classsuperagent
Example user agent
node-superagent/1.4.0Entireweb
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)WooRank
Example user agent
woobot/1.1Envolk
Example user agent
envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)picmole Bot
Example user agent
Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)AdsBot Google-Mobile
Example user agent
AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile SafariReverse IP Lookup
Example user agent
Mozilla/5.0+(compatible; RevIP.info site analyzer v4.00; http://poweredby.revip.info)Lightspeed Systems RocketCrawler
Example user agent
LSSRocketCrawler/1.0 LightspeedSystemswget
Example user agent
Wget/1.14 (linux-gnu)vSentry
Example user agent
Mozilla/5.0 (Windows; chromeframe/2.4.8.5746) AppleWebKit/1.0 (KHTML, like Gecko) Bromium Safari/1.0PagesInventory Bot
Example user agent
PagesInventory (robot http://www.pagesinventory.com)Safe-t.net Web Filtering Service
Example user agent
FANGCrawl/0.01VoilaBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)nuSearch
Example user agent
nuSearch Spider <a href='http://www.nusearch.com'>www.nusearch.com</a> (compatible; MSIE 4.01; Windows NT)Netcraft Webserver Survey
Example user agent
Mozilla/4.0 (compatible; Netcraft Web Server Survey)uMBot
Example user agent
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)NetNewsWire
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.74.9 (KHTML, like Gecko) NetNewsWire/3.3.2Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://webarhive.nlc.gov.cn)Sitemap Generator
Example user agent
Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)Jupdate
Example user agent
jupdateScrapy
Example user agent
Scrapy/0.16.4 (+http://scrapy.org)adidxbot
Example user agent
adidxbot/2.0 (+http://search.msn.com/msnbot.htm)Google Bot Mobile
Example user agent
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)wget
Example user agent
Wget/1.11.4URL Grabber
Example user agent
urlgrabber/3.10 yum/3.4.3Baidu Image Search
Example user agent
Baiduspider-image ( http://www.baidu.com/search/spider.htm)YOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://3uves.com)RemoteGet
Example user agent
gpEasy RemoteGetdead link checker
Example user agent
www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36Jayde Crawler
Example user agent
Jayde Crawler. http://www.jayde.comHeritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.f360Spider
Example user agent
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360SpiderAntBot
Example user agent
AntBot/1.0 (http://www.ant.com)Sistrix Crawler
Example user agent
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)SEOdiver Bot
Example user agent
Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)Iframely Bot
Example user agent
Iframely/0.7.1 (+http://iframe.ly/;)FacebookExternalHit
Example user agent
facebookexternalhit/1.1CSE HTML Validator
Example user agent
CSE HTML Validator Online (http://www.OnlineWebCheck.com) via 23.19.188.134BrokenLinkCheck
Example user agent
Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1Alexa Site Audit
Example user agent
Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )NetSurf
Example user agent
NetSurf/2.0 (Linux; i686)CSE HTML Validator
Example user agent
CSE HTML Validator Online (http://www.OnlineWebCheck.com) via 108.62.151.31
Example user agent
Mozilla/5.0 (Java; U; ru; samsung-gt-s5380f) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0RssBandit
Example user agent
RssBandit/1.9.0.1002 (.NET CLR 2.0.50727.7512; WinNT 6.2.9200.0; http://www.rssbandit.org)Apercite
Example user agent
Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)node-fetch
Example user agent
node-fetch/1.0 (+https://github.com/bitinn/node-fetch)Internet Archive
Example user agent
Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler)NewsBlur Favicon Fetcher
Example user agent
NewsBlur Page Fetcher - 7 subscribers - http://www.newsblur.com/site/3966817/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)BingPreview
Example user agent
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Geckofeedly Feed Fetcher
Example user agent
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s3650) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0Internet Archive
Example user agent
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)asynchttp
Example user agent
asynchttpYOURLS
Example user agent
YOURLS v1.6 +http://yourls.org/ (running on http://revmkt.in)ZumBot
Example user agent
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)GG PeekBot
Example user agent
GG PeekBot 2.0 ( http://gg.pl/ http://info.gadu-gadu.pl/praca )Steeler
Example user agent
Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)JUST-CRAWLER
Example user agent
JUST-CRAWLER( http://www.justsystems.com/jp/tech/crawler/)Rival IQ Bot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)LinkWalker
Example user agent
LinkWalker/2.0EasouSpider
Example user agent
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)oBot
Example user agent
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)Amiga
Example user agent
AmigaVoyager/3.4.4 (MorphOS/PPC native) user-agentGurujiBot
Example user agent
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)BuSecurityProject
Example user agent
Mozilla,BuSecurityProject
Example user agent
Mozilla/5.0 (Java; U; En-in; Samsung-gt-s7070) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0RamblerMail Bot
Example user agent
RamblerMail/6.0 (incompatible; ImageProxy/6.0)Tiny Tiny RSS
Example user agent
Tiny Tiny RSS/1.10 (http://tt-rss.org/)GurujiBot
Example user agent
GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)bot
Example user agent
Bot Blocker Crawler 1.0 (btw, IncrediBILL says "HI!")Stratagems Kumo
Example user agent
Stratagems Kumohttpclient
Example user agent
HTTPClient/1.0 (2.4.0, ruby 1.9.3 (2013-06-27))Ilse
Example user agent
INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)Xenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.1fTinEye Bot
Example user agent
TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)
Example user agent
Mozilla/5.0 (Java; U; en-in; lenovo-p50) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0SimplePie
Example user agent
SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20130514092120
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-c6712) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0masidani_bot
Example user agent
masidani_bot_v0.3 (info@masidani.com)Android Download Manager
Example user agent
AndroidDownloadManager/4.3 (Linux; U; Android 4.3; ME302C Build/JSS15Q)Pcore::HTTP::UA
Example user agent
Mozilla/5.0 (compatible; U; Pcore-HTTP-UA/v0.12.4Python-urllib
Example user agent
Python-urllib/3.4sSearch Crawler
Example user agent
ssearch_bot (sSearch Crawler; http://www.semantissimo.de)Larbin
Example user agent
binlar_2.6.3 (test@mgmt.mic)
Example user agent
Mozilla/5.0 (Java; U; ru; lg-gb270) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0Entireweb
Example user agent
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)FacebookExternalHit
Example user agent
facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php)SurveyBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)Python-urllib
Example user agent
Python-urllib/1.15WordPress
Example user agent
WordPress/3.7.1; http://samsung.hdblog.it
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-s3650w) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0FyberSpider
Example user agent
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)magpie-crawler
Example user agent
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)ThumbShotsBot
Example user agent
Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010091808 ThumbShotsBot (KFSW 3.0.6-3)Nutch
Example user agent
agent6/Nutch-1.1Google Search Appliance
Example user agent
gsa-crawler
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-lg800g) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0Semantic Discovery
Example user agent
semanticdiscovery/0.2(http://www.semanticdiscovery.com/sd/robot.html)NewsBlur Favicon Fetcher
Example user agent
NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)i1searchbot
Example user agent
i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)HRCrawler
Example user agent
HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko)CheckLinks
Example user agent
Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks)
Example user agent
JNLP/6.0 javaws/1.6.0_14 (b08) Java/1.6.0_14FeedlyBot
Example user agent
FeedlyBot/1.0 (http://feedly.com)open-webkit-sharp
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) OpenWebKitSharp
Example user agent
Mozilla/5.0 (Java; U; Id; lg-t500) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung gt-b5722c) UCBrowser8.3.0.154/69/444/UCWEB MobileY!J-AGENT
Example user agent
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)CSE HTML Validator
Example user agent
CSE HTML Validator Online (http://www.OnlineWebCheck.com) via 108.62.195.142Amiga
Example user agent
Mozilla/4.0 (compatible; IBrowse 2.3; AmigaOS4.0)Nutch
Example user agent
5acd734a-1ed3-4a17-ad9d-0b09ae7bc2d2/Nutch-1.7python-requests
Example user agent
python-requests/0.14 CPython/2.6 Linux/2.6-43-serverNutch
Example user agent
My Nutch Spider/Nutch-1.6YoudaoBot
Example user agent
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )Entireweb
Example user agent
Speedy Spider (Beta/1.0; www.entireweb.com)YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://jim-s.com)itsscan
Example user agent
Mozilla/5.0 (compatible; Muenster University of Applied Sciences; +http://fb02itsscan.fh-muenster.de)Baiduspider
Example user agent
Baiduspider-testbranchWindows-RSS-Platform
Example user agent
Windows-RSS-Platform/2.0 (MSIE 9.0; Windows NT 6.0)Thumbor
Example user agent
Thumbor/3.14.6Gigabot
Example user agent
Gigabot/3.0 (http://www.gigablast.com/spider.html)Googlebot-Video
Example user agent
Googlebot-Video/1.0WeSEE Bot
Example user agent
WeSEENaverBot
Example user agent
Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)netEstate NE Crawler
Example user agent
netEstate NE Crawler (+http://www.website-datenbank.de/)Larbin
Example user agent
binlar_2.6.3 test@mgmt.mic
Example user agent
Mozilla/5.0 (Java; U; en-in; lg-gm360) UCBrowser8.2.1.144/69/405/UCWEB Mobile UNTRUSTED/1.0Taproot Bot
Example user agent
Taproot (Guzzle) http://taprootproject.comEvernote Clip Resolver
Example user agent
Evernote Clip Resolverphantomas
Example user agent
phantomas/1.8.0 (PhantomJS/1.9.8; linux x64)FHScan Core
Example user agent
Mozilla/5.0 (FHScan Core 1.1)MetaURI Bot
Example user agent
MetaURI API/2.0 +metauri.comlibwww
Example user agent
libwww-perl/5.800Microsoft CryptoAPI
Example user agent
Microsoft-CryptoAPI/6.1MixBot
Example user agent
MixBot (+http://t.co/GSRLLKex24)Willow Internet Crawler
Example user agent
Willow Internet Crawler by Twotrees V2.1Nutch
Example user agent
Innovazion Crawler/Nutch-1.7Wada.vn Search Bot
Example user agent
Wada.vn Vietnamese Search
Example user agent
Mozilla/5.0 (Java) outbrainjig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; CA3C)Website Thumbnail Generator
Example user agent
Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot)
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-gb175) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobilewget
Example user agent
Wget/1.16.1 (darwin14.0.0)Moozilla
Example user agent
MoozillaSogou Image Crawler
Example user agent
Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)msnbot
Example user agent
msnbot/2.0b v1394720947.6703Melvil Bot
Example user agent
Melvil/1.0
Example user agent
HuaweiU1250/B002 Browser/Access-Browser/Q05A MMS/Obigo-MMS/Q05A SyncML/HW-SyncML/1.0 Java/HWJa/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Player/QTV-Player/5.3YOURLS
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://mv2.me)OpenHoseBot
Example user agent
Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)libwww
Example user agent
libwww-perl/5.803WebIndex
Example user agent
WebIndexTURLValidator
Example user agent
Mozilla/4.0 (compatible; TURLValidator/1.0)Microsoft CryptoAPI
Example user agent
Microsoft-CryptoAPI/6.3
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-i6712) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0Lightspeed Systems Crawler
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0;Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727;.NET CLR 3.0.04506.30) LightspeedsystemsNetSurf
Example user agent
NetSurf/3.0 (RISC OS)Java Standard Library
Example user agent
Java/1.8.0_25profiller
Example user agent
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) profiller Safari/537.21jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; F903i)WordPress
Example user agent
WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 23.19.127.211Linkdex Bot
Example user agent
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/)YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://ecra.se)Werbefreie Deutsche Suchmaschine
Example user agent
Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)TrendWinHttp
Example user agent
TrendWinHttp/Java Standard Library
Example user agent
Java1.3.0Phantom.js bot
Example user agent
Phantom.js botEzooms
Example user agent
Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)Google PageSpeed Insights
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1
Example user agent
Mozilla/5.0 (Java; U; Id; samsung-gt-s5753e) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-t505) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0Scrubby
Example user agent
Mozilla/5.0 (compatible; Scrubby/2.1; +http://www.scrubtheweb.com/abs/meta-check.html)Mozilla Crawler
Example user agent
Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc)Amiga
Example user agent
Mozilla/5.0 (compatible; Origyn Web Browser; AmigaOS 4.1; PPC; U) AppleWebKit/525.1+ (KHTML, like Gecko, Safari/525.1+)Guzzle Http Client
Example user agent
GuzzleHttp/6.1.0 curl/7.35.0 PHP/5.5.9-1ubuntu4.14Android Download Manager
Example user agent
AndroidDownloadManager/4.1.1 (Linux; U; Android 4.1.1; MB886 Build/9.8.0Q-97_MB886_FFW-20)Jeode
Example user agent
Jeode/1.x.xGoogle Bot
Example user agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; aplix jblend) UCBrowser8.4.0.159/69/352/UCWEB Mobile, SAMSUNG-SGH-I617/1.0 Mozilla/4.0 UNTRUSTED/1.0Nutch
Example user agent
GrowerIdeas Crawler/GrowerIdeas-nutch-1.6 (Crawls URLs for indexing content for our new search startup which aims to provide simple and smart search across curated content. For more info please contact helloworld@growerideas.com. If you thinkSetLinks.ru Crawler
Example user agent
SetLinks bot 1.0
Example user agent
Mozilla/5.0 (Java; U; kau; samsung-c6625) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0, UNTRUSTED/1.0Picsearch Bot
Example user agent
psbot/0.1 (+http://www.picsearch.com/bot.html)city-map screenshot service
Example user agent
city-map screenshot servicehitcrawler
Example user agent
hitcrawler_0.1 (ringken@gmail.com)ht://Dig
Example user agent
htdig/3.1.5Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurpbot
Example user agent
botDocWeb Link Crawler
Example user agent
DocWeb Link Crawler (http://doc.php.net)MetaGeneratorCrawler
Example user agent
MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)Page Analyzer
Example user agent
Page Analyzer v4.0 ( http://www.ranks.nl/ )Mozdex
Example user agent
mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)YisouSpider
Example user agent
YisouSpideriZSearch Bot
Example user agent
iZSearch.comWebsense
Example user agent
Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)Livelap Crawler
Example user agent
LivelapBot/0.2 (http://site.livelap.com/crawler)Nutch
Example user agent
Nutch12/Nutch-1.2cURL
Example user agent
curl/7.29.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-kd876) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0TurnitinBot
Example user agent
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com)Teoma
Example user agent
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)YRL_ODP_CRAWLER
Example user agent
YRL_ODP_CRAWLERSkypeUriPreview
Example user agent
Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5SemanticBot
Example user agent
Mozilla/5.0 (compatible; SemanticBot/1.1)bandscraper
Example user agent
bandscraperthumbshots-de-bot
Example user agent
thumbshots-de-bot (+http://www.thumbshots.de/)YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://linkcut.eu)Sogou Head Spider
Example user agent
Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)python-requests
Example user agent
python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-24-genericJava Standard Library
Example user agent
Java/1.6.0_22Heritrix
Example user agent
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://netarkivet.dk/webcrawler/)
Example user agent
LG-GW620 Mozilla/5.0 (Linux; U; Android 1.5) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 Java/Jbed/7.0 Profile/MIDP-2.MOSBookmarks Link Checker
Example user agent
Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker)Sogou-Test-Spider
Example user agent
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)MagpieRSS
Example user agent
MagpieRSS/0.61 (+http://magpierss.sf.net)YandexBlogs
Example user agent
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)Windows Phone Search
Example user agent
Windows Phone Search (Windows Phone OS 7.10;NOKIA;Lumia 900;7.10;8862)SemrushBot
Example user agent
Mozilla/5.0 (compatible; SemrushBot/0.97sa; +http://www.semrush.com/bot.html)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; 810T)Aboundexbot
Example user agent
Aboundex/0.3 (http://www.aboundex.com/crawler/)NING
Example user agent
NING/1.0AISearchBot
Example user agent
AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.)libwww
Example user agent
libwww-perl/5.800 (+http://passoire.afraid.org/mylittlewebsurvey/index.html)360Spider
Example user agent
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360SpiderXenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2bGlobel Bot
Example user agent
Mozilla/4.0 (compatible; Globel; Traffic Sent From: http://addictive.me.uk)ScreenerBot
Example user agent
ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-e2220) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile, SAMSUNG-GT-E2220/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0Google Bot
Example user agent
GoogleEarth/7.1.0002.2011(Android;Android (HTC One mini-m4-user-4.2.2);de-DE;kml:2.2;client:Free;type:default)Link Valet
Example user agent
Link Valet Online 1.1
Example user agent
Mozilla/5.0 (compatible; Ex-Crawler/0.1.5a; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)Google Bot
Example user agent
GoogleEarth/7.1.0003.1255(Android;Android (HTC One-m7-user-4.3);de-DE;kml:2.2;client:Free;type:default)Cliqzbot
Example user agent
Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)Applebot
Example user agent
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)80Legs
Example user agent
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620Riddler
Example user agent
Riddler (http://riddler.io/about.html)jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; F906i)m65bot
Example user agent
DoCoMo/2.0 P904i( m65bot/0.1; c; http://m65.jp/bot.html )Goldfire Server
Example user agent
Goldfire ServerGroupHigh Bot
Example user agent
Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)Arachnida Web Crawler
Example user agent
Arachnida Web CrawlerVoilaBot
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)Pingdom
Example user agent
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)Python-urllib
Example user agent
Python-urllib/3.0Camo Asset Proxy
Example user agent
Camo Asset Proxy 2.2.0vkShare
Example user agent
Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)Java Standard Library
Example user agent
Java1.4.0_02inbound.li link checker
Example user agent
inboundli link checkerGenieo Web Filter
Example user agent
Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.htmlDomainsBot
Example user agent
domainsbot (+http://www.domainsbot.com)BingBot
Example user agent
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)Convera
Example user agent
ConveraCrawler/0.9c ( http://www.authoritativeweb.com/crawl)Python-urllib
Example user agent
Python-urllib/2.7memoryBot
Example user agent
Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)Google Bot
Example user agent
Googlebot-richsnippetsECCP
Example user agent
ECCP/1.0 (search@eniro.com)TestCrawler
Example user agent
robotsFeedFetcher
Example user agent
livedoor FeedFetcher/0.01 (http://reader.livedoor.com/; 999 subscribers)GarlikCrawler
Example user agent
GarlikCrawler/1.2 (http://garlik.com/nbot
Example user agent
Mozilla/5.0 (compatible; nbot/2.0)YOURLS
Example user agent
YOURLS v1.5 +http://yourls.org/ (running on http://onj.me)Curb
Example user agent
Ruby/CurbCatchBot
Example user agent
CatchBot/2.0; +http://www.catchbot.comBoitho
Example user agent
boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )Peeplo Screenshot Bot
Example user agent
Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )YandexNews
Example user agent
Mozilla/5.0 (compatible; YandexNews/3.0; +http://yandex.com/bots)lwp
Example user agent
lwp-trivial/1.36BOT for JCE
Example user agent
BOT/0.1 (BOT for JCE)Tutorial Crawler
Example user agent
Tutorial Crawler 1.4Google Structured-Data TestingTool
Example user agent
Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)A6-Indexer
Example user agent
A6-Indexer
Example user agent
Samsung/910XXIG2/sw_platform=S60;sw_platform_version=5.0;java_build_version=1.3.4 Profile/MIDP-2.1 Configuration/CLDC-1.1 UNTRUSTED/1.0AskPeterBot
Example user agent
Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html)Reddit
Example user agent
Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)OffByOne
Example user agent
Mozilla/4.7 (compatible; OffByOne; Windows )NaverBot
Example user agent
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)WebFuck
Example user agent
WebFuck V2.1 T0PHackTeam www.t0p.xyzHuaweiSymantecSpider
Example user agent
HuaweiSymantecSpider/1.0 DSE-support@huaweisymantec.com (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-b5722) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB MobileAdSense Snapshot Bot
Example user agent
Adsense-Snapshot-GoogleCrawlboy
Example user agent
Crawlzilla/1.0 (Crawlzilla; http://www.crawlzilla.com; crawler@crawlzilla.com)
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-sgh-a777) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0Accoona-AI-Agent
Example user agent
Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-gu230) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0YandexImageResizer
Example user agent
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)FAST PartnerSite
Example user agent
Fast PartnerSite Crawler
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5380d) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0Linkdex Bot
Example user agent
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)Abonti WebSearch
Example user agent
Mozilla/5.0 (compatible; Abonti/0.92 - http://www.abonti.com)YOURLS
Example user agent
YOURLS v1.5.1 +http://yourls.org/ (running on http://tty.nu)WordPress
Example user agent
WordPress/3.2.1; http://www.painlog.jp
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b5310) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0viralvideochart Bot
Example user agent
viralvideochart.unrulymedia.comComodo Spider
Example user agent
Comodo Spider 1.2YOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://rutlink.info)HggH Screenshot System with PhantomJS
Example user agent
HggH PhantomJS Screenshoter
Example user agent
Mozilla/5.0 (Java; U; es-la; lg-gm360i) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0Xenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2g
Example user agent
Mozilla/5.0 (Java; U; ru; samsung-sgh-d880) UCBrowser8.3.0.154/69/444/UCWEB MobileDownload Accelerator
Example user agent
masscan/1.0 (https://github.com/robertdavidgraham/masscan)Second Live Commerce Client
Example user agent
Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0 ; .NET CLR 2.0.50215; SL Commerce Client v1.0; Tablet PC 2.0Web Glance
Example user agent
Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1Java Standard Library
Example user agent
Java1.4.2_03FyberSpider
Example user agent
FyberSpiderCrawlboy
Example user agent
Crawl/0.1 langcrawl/0.1BecomeBot
Example user agent
Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-a133) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0PEAR HTTP_Request2
Example user agent
HTTP_Request2/2.1.1 (http://pear.php.net/package/http_request2) PHP/5.3.10-1ubuntu3.6cURL
Example user agent
curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3ZumBot
Example user agent
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)Amiga
Example user agent
Mozilla/6.0 (Macintosh; U; Amiga-AWeb) Safari 3.1Plus5Bot
Example user agent
Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net)TestCrawler
Example user agent
BoardReader/1.0 (http://boardreader.com/info/robots.htm)-Machine1Google Bot
Example user agent
GooglebotFeedfetcher-Google
Example user agent
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 375 subscribers; feed-id=15381863289700640853)MixrankBot
Example user agent
Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)YandexDirect-Dyatel
Example user agent
Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)Casterly
Example user agent
CasterlyGuzzle Http Client
Example user agent
GuzzleHttp/6.1.0 curl/7.45.0 PHP/5.6.14Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0 ; http://help.yahoo.com/help/us/ysearch/slurp)Python-urllib
Example user agent
Python-urllib/2.1Rogerbot
Example user agent
rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)WSCheck Bot
Example user agent
wscheck.com/1.0.0 (+http://wscheck.com/)Holmes
Example user agent
holmes/3.12.4 (http://morfeo.centrum.cz/bot)
Example user agent
Mozilla/5.0 (Java; U; Id; lg-c305) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0Xenus Link Sleuth
Example user agent
Xenu Link Sleuth 1.2fSuperfeedr Bot
Example user agent
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!SynHttpClient
Example user agent
SynHttpClient-Built/5759
Example user agent
SonyEricssonJ20i/R7CA Profile/MIDP-2.1 Configuration/CLDC-1.1 UCWEB/2.0 (Java; U; MIDP-2.0; xx; SonyEricssonJ20i) U2/1.0.0 UCBrowser/9.4.1.377 U2/1.0.0 MobileCookie Reports Bot
Example user agent
CookieReports.com/0.1BacklinkCrawler
Example user agent
BacklinkCrawler (http://www.backlinktest.com/crawler.html)Java Standard Library
Example user agent
Java/1.7.0_71
Example user agent
Mozilla/5.0 (Java; U; en-us; lg-lg500g) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0
Example user agent
Mozilla/5.0 (Java; U; en-in; samsung-sgh-a687) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0Scrapy
Example user agent
Scrapy/0.22.2 (+http://scrapy.org)Java Standard Library
Example user agent
Java1.2.2
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung gt-c3520) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-gt-b2710) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/84/352/UCWEB Mobile UNTRUSTED/1.0libwww
Example user agent
libwww-perl/5.65
Example user agent
Mozilla/5.0 (Java; U; es-la; samsung-sgh-a847) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0Google Bot
Example user agent
GoogleEarth/7.1.0002.2011(Android;Android (HTC Sensation XL with Beats Audio X315e-runnymede-user-4.0.3);de-DE;kml:2.2;client:Free;type:default)Daumoa
Example user agent
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0;UniversalFeedParser
Example user agent
UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/WordPress
Example user agent
WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 23.19.127.108Zookabot
Example user agent
Zookabot/2.5;++http://zookabot.comOrangebot
Example user agent
Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)worldwebheritage.org Bot
Example user agent
Mozilla/5.0 (compatible; worldwebheritage.org/1.0; +crawl@worldwebheritage.org)W3C_Validator
Example user agent
W3C_Validator/1.3 http://validator.w3.org/servicesWiseNut search engine crawler
Example user agent
Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)AhrefsBot
Example user agent
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxyfile1-hrd)ThumbSniper
Example user agent
ThumbSniper (http://thumbsniper.com)MetaHeadersBot
Example user agent
MetaHeadersBot (+http://www.metaheaders.com/bot.html)Jigsaw_W3C_CSS_Validator
Example user agent
Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See <http://validator.w3.org/services>)BUbiNG Bot
Example user agent
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)NetNewsWire
Example user agent
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; de-de) AppleWebKit/531.22.7 (KHTML, like Gecko) NetNewsWire/3.2.7
Example user agent
LinkStar Research Crawler (http://linkstar.com/),gzip(gfe),gzip(gfe)SoftListBot
Example user agent
Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)Webmaster Tips Bot
Example user agent
Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)USyd-NLP-Spider
Example user agent
USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)XoviBot
Example user agent
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)DueDil Crawler
Example user agent
Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)SophosUpdateManager
Example user agent
SophosUpdateManager/1.5.7.50 SDDS/2.0 (u= EO2ANA123G c= 6342da15-f351-4ab7-9656-3f5f2d50885d )Bot.AraTurka.com
Example user agent
Bot.AraTurka.com/0.0.1Melvil Favicon Bot
Example user agent
Melvil Favicon/1.0 (Favicon fetcher for Melvil)Holmes
Example user agent
holmes/2.3Google-Site-Verification
Example user agent
Mozilla/5.0 (compatible; Google-Site-Verification/1.0)Flatland
Example user agent
flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)Accoona-AI-Agent
Example user agent
Accoona-AI-Agent/1.1.2Exalead
Example user agent
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)Nutch
Example user agent
emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)CazoodleBot
Example user agent
CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com)Amiga
Example user agent
IBrowse/2.5beta (AmigaOS 4.1; PPC)Apt HTTP Transport
Example user agent
Ubuntu APT-HTTP/1.3 (0.7.20.2ubuntu6)BlogLines Web
Example user agent
Bloglines/3.1 (http://www.bloglines.com)Feedfetcher-Google
Example user agent
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=000000000000000000)Qualidator.com Bot
Example user agent
Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)WordPress
Example user agent
WordPress.com; https://longboardrules.wordpress.comwget
Example user agent
Wget/1.12 (linux-gnu)DotBot
Example user agent
DotBot/1.0.1 (http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)Tweetminster Bot
Example user agent
Mozilla/5.0 TweetminsterYOURLS
Example user agent
YOURLS v1.7 +http://yourls.org/ (running on http://www.bv-xa.com)SiteCon
Example user agent
Mozilla/5.0 (Linux; U; en-gb) AppleWebKit/418.9.1 (KHTML, like Gecko) SiteCon/8.10.9Google Bot
Example user agent
Mozilla/5.0 AppleWebKit/537.4 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.4restify
Example user agent
restify/2.6.3 (x64-darwin; v8/3.14.5.9; OpenSSL/1.0.1i) node/0.10.32msnbot
Example user agent
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
Example user agent
Mozilla/5.0 (Java; U; en-us; samsung-gt-c5510) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0JamesBOT
Example user agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html
Example user agent
Mozilla/5.0 (Java; U; en-in; lg-gt350) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0ShowyouBot
Example user agent
ShowyouBot (http://showyou.com/crawler)IBM Connections
Example user agent
IC OpenGraph Crawler 4.5 (proprietary)kulturarw3
Example user agent
Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)AddThis.com robot
Example user agent
AddThis.com (http://support.addthis.com/)Finderlein Research Crawler
Example user agent
Finderlein Research Crawler 1.0jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; KC38)Ichiro Mobile Bot
Example user agent
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)GO HttpClient
Example user agent
Go-http-client/1.1WorldLingo
Example user agent
WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com)Y!J-AGENT
Example user agent
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
Example user agent
SonyEricssonW666i/R2BA Java/SEMC-Java/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0chlooe Bot
Example user agent
bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)JobBoerse Bot
Example user agent
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0WebMasterAid
Example user agent
Mozilla/5.0 (compatible; WebMasterAid/1.0; +http://wmaid.com/explore/)XML Sitemaps Generator
Example user agent
Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0Digg Bot
Example user agent
Mozilla/5.0 (Digg/1.0; support@digg.com)Bloglovin Bot
Example user agent
Bloglovin/1.0 (http://www.bloglovin.com; 1 subscribers)Socialradar Bot
Example user agent
Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com)Yahoo! Slurp
Example user agent
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)Twitterbot
Example user agent
Twitterbot/1.0FreeWebMonitoring SiteChecker
Example user agent
FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)MagpieRSS
Example user agent
MagpieRSS/0.72 (+http://magpierss.sf.net)cURL
Example user agent
curl/7.35.0ht://Dig
Example user agent
htdig/3.1.6 (romieu@bastide-medical.fr)SMRF URL Expander
Example user agent
SMRF URL ExpanderRogerbot
Example user agent
rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+pr1-crawler-14@moz.com)Internet Archive Special Archiver
Example user agent
Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)focuseekbot
Example user agent
Mozilla/4.0 (compatible; focuseekbot)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; lg-p525) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0jig browser web
Example user agent
Mozilla/4.0 (jig browser web; 1.0.4; 705NK)Slackbot-Link-Expanding
Example user agent
Slackbot-LinkExpanding+(+https://api.slack.com/robots)ht://Dig
Example user agent
htdig/3.1.6 (mathieu.peltier@inrialpes.fr)Trove Bot
Example user agent
Trove (help@trove.com, http://www.trove.com)Patwebbot
Example user agent
Patwebbot (http://www.herz-power.de/technik.html)
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung gt-c6112) UCBrowser8.2.1.144/69/444/UCWEB MobileLinkedInBot
Example user agent
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)JobdiggerSpider
Example user agent
Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)Voyager Bot
Example user agent
voyager/1.0
Example user agent
Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s7230b) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0SiteExplorer
Example user agent
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://SiteExplorer.Info/)WordPress
Example user agent
WordPress/1.2.1 PHP/4.3.9-1WeSEE:Search
Example user agent
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)DNS Group Crawler
Example user agent
DNSGroup/0.1 (DNS Group Crawler; http://www.dnsgroup.com/; crawler@dnsgroup.com)
Comparison created
2016-05-17 06:45:57 | by
ThaDafinser