Detected as bot - BrowscapPhp 6012

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-gb230) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• Applebot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

• Google Bot

  Example user agent
  Googlebot-richsnippets

• wget

  Example user agent
  Wget/1.15 (linux-gnu)

• Crawlboy

  Example user agent
  Crawllybot/0.1/0.1 (Crawllybot/0.1; http://www.crawlly.com; crawler@crawlly.com)

• Fake Browser

  Example user agent
  http://www.almaden.ibm.com/cs/crawler

• NameProtect

  Example user agent
  abot/0.1 (abot; http://www.abot.com; abot@abot.com)

• InfegyAtlas

  Example user agent
  Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; collection@infegy.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b5310) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0

• SEOprofiler

  Example user agent
  Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )

• RSSOwl

  Example user agent
  RSSOwl/2.2.1.201312301316 (X11; U; en)

• Google Bot Mobile

  Example user agent
  KDDI-CA34 UP.Browser/6.2.0.10.2.2(GUI)MMP/2.0 (compatible; KDDI-Googlebot-Mobile/2.1; http://www.google.com/bot.html)

• RedCarpet

  Example user agent
  RedCarpet/1.0 (http://www.redcarpet-inc.com/robots.html)

• Amiga

  Example user agent
  Mozilla/5.0 (AmigaOS 4; PPC; InternetSurfboard; U; xx) AppleWebKit/533.3 (KHTML, like Gecko) InternetSurfboard/0.002 Safari/533.3

• General Crawlers

  Example user agent
  SonyEricssonJ108i/R7EA Profile/MIDP-2.1 Configuration/CLDC-1.1 UNTRUSTED/1.0 UCWEB/2.0(Java; U; MIDP-2.0; xx; sonyericssonj108i) U2/1.0.0 UCBrowser/8.8.0.227 U2/1.0.0 Mobile

• MetaGeneratorCrawler

  Example user agent
  MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)

• Nutch

  Example user agent
  MySpider/Nutch-2.2

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-gr700) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; LNV-Lenovo_A560e Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; HTC T328w Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• PHP

  Example user agent
  PHP/5.2.13

• Fake IE

  Example user agent
  ( ; MSIE 9.0; Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko

• NewsGator

  Example user agent
  NewsGator/2.0 Bot (http://www.newsgator.com)

• Google Bot

  Example user agent
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5560i) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0

• Fake IE

  Example user agent
  Mozilla/5.0 [ja] (compatible; MSIE 6.0; Linux 2.6.11-1.1369_FC4; Playstation2)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; SCH-I699 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Unity Web Player

  Example user agent
  UnityPlayer/4.3.2f1 (http://unity3d.com)

• Sogou Web Spider

  Example user agent
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-c305) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• ExactSeek Crawler

  Example user agent
  ExactSeek Crawler/0.1

• Cydral Web Image Search

  Example user agent
  CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com)

• MetaURI Bot

  Example user agent
  MetaURI API/2.0 +metauri.com

• Fake Browser

  Example user agent
  Mozzila/5.0(Linux; U; Android 4.2.2; ru-ru; Flylife Connect 7 3G Build/JDQ39) AppleWebKit/534.30(KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

• Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2,22

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-c3500) UCBrowser8.2.0.132/69/352/UCWEB Mobile

• Squzer

  Example user agent
  Mozilla/5.0 (compatible; Squzer/0.0.1; +http://www.declum.com/)

• WinHttp

  Example user agent
  Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)

• 80Legs

  Example user agent
  Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/538.8 HbbTV/1.1.1 (+DL+PVR; selevision; Selevision EMC1000i; 1.0; 1.0;) CE-HTML/1.0

• Ezooms

  Example user agent
  Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)

• msnbot

  Example user agent
  msnbot/1.1 ( http://search.msn.com/msnbot.htm)

• libwww

  Example user agent
  NEC MosaicView/1.0 libwww/2.15_Spyglass

• Google Bot

  Example user agent
  GoogleEarth/7.1.0003.1255(Android;Android (HTC One-m7-user-4.3);de-DE;kml:2.2;client:Free;type:default)

• Butterfly Robot

  Example user agent
  Mozilla/5.0 (Macintosh; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8

• Amiga

  Example user agent
  Mozilla/5.0 (Amiga; PowerPC AmigaOS 4.1; Odyssey Web Browser; rv:1.23) AppleWebKit/538.1 (KHTML, like Gecko) OWB/1.23 Safari/538.1

• General Crawlers

  Example user agent
  ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86)

• Website Explorer

  Example user agent
  Website Explorer/0.9.9.9

• bot

  Example user agent
  Bot Blocker Crawler 1.0 (btw, IncrediBILL says "HI!")

• Second Live Client

  Example user agent
  Mozilla/5.0 (Windows; U; ; en-US) AppleWebKit/533.3 (KHTML, like Gecko) SecondLife/3.7.23.297296 (Second Life Release; default skin) Safari/533.3

• Baidu Union

  Example user agent
  Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)

• Amiga

  Example user agent
  IBrowse/2.5beta (AmigaOS 4.1; PPC)

• YodaoBot-Image

  Example user agent
  Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)

• Google PageSpeed Insights

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1

• Gigablast Search Engine

  Example user agent
  GigablastOpenSource/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s5253) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible; MSIE 8.0; Win32)

• Internet Archive Special Archiver

  Example user agent
  Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)

• Apache Synapse

  Example user agent
  Mozilla/4.0 (compatible; Synapse)

• nuSearch

  Example user agent
  Nusearch Spider (compatible; MSIE 6.0)

• SemrushBot

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.1; zh-cn; GT-N7108 Build JRO03C) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• Apt HTTP Transport

  Example user agent
  Ubuntu APT-HTTP/1.3 (0.7.20.2ubuntu6)

• Google Bot

  Example user agent
  Googlebot/2.1 (+http://www.google.com/bot.html)

• URL Grabber

  Example user agent
  urlgrabber/3.9.1 yum/3.2.29

• Generic Ruby Crawler

  Example user agent
  Ruby

• Genderanalyzer

  Example user agent
  Genderanalyzer/1.0

• NewsGatorOnline

  Example user agent
  NewsGatorOnline/2.0 (http://www.newsgator.com)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)

• SMTBot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; lg-gu295) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• Zookabot

  Example user agent
  Zookabot/2.4; +http://zookabot.com

• General Crawlers

  Example user agent
  LG-GT540 Mozilla/5.0 (Linux; U; Android 1.6; xx; GT540; Build/Donut) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 Java/Jbed/7.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 MMS/LG-Android-MMS-V1.0/1.2

• FyberSpider

  Example user agent
  FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)

• RssBandit

  Example user agent
  RssBandit/1.9.0.1002

• Rogerbot

  Example user agent
  rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+pr1-crawler-14@moz.com)

• Google Web Preview

  Example user agent
  Mozilla/5.0 (iPad; CPU OS 5_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/5.1 Mobile/9A334 Safari/7534.48.3

• NetSeer Crawler

  Example user agent
  Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)

• msnbot

  Example user agent
  msnbot/2.0b v1394720947.6703

• Y!J-AGENT

  Example user agent
  Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80)

• Feedfetcher-Google

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 2 subscribers; feed-id=17860707833818568603)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-b3410w) UCBrowser8.2.0.132/70/452/UCWEB Mobile UNTRUSTED/1.0

• Googlebot-Video

  Example user agent
  Googlebot-Video/1.0

• Fake IE

  Example user agent
  IE6.0,Java(TM) 2 Runtime Environment, Standard Edition

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-i6230) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0

• Liferea

  Example user agent
  Liferea/1.10-RC1 (Linux; en_GB.UTF-8; http://liferea.sf.net/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; lenovo-p50) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/2.3; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• SEOdiver Bot

  Example user agent
  Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-c6712) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0

• BoardReader Favicon Fetcher

  Example user agent
  BoardReader Favicon Fetcher /1.0 info@boardreader.com

• Crawler

  Example user agent
  Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/)

• Bot Revolt

  Example user agent
  revolt

• YoudaoBot

  Example user agent
  Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )

• WhatWeb Web Scanner

  Example user agent
  WhatWeb/0.4.8

• Amiga

  Example user agent
  Mozilla/5.0 (Macintosh; PowerPC AmigaOS 4.1; Odyssey Web Browser; rv:1.16) AppleWebKit/535.14 (KHTML, like Gecko) OWB/1.16 Safari/535.14

• NaverBot

  Example user agent
  NaverNDrive/3.2.1 (Android OS 4.3; samsung SHV-E210S)

• kgbody

  Example user agent
  Mozilla/5.0 (compatible; kgbody/2.0; +http://kids.goo.ne.jp/tool/kgbody.php)

• NewsGatorOnline

  Example user agent
  NewsGatorOnline/2.0 (http://www.newsgator.com; 2 subscribers)

• wget

  Example user agent
  Wget/1.16.1 (darwin14.0.0)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.2; en-us; HUAWEI G610-C00 Build HuaweiG610-C00) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-s5628) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0

• FAST PartnerSite

  Example user agent
  Fast PartnerSite Crawler

• Yahoo Ad Monitoring

  Example user agent
  Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)

• Apple-PubSub

  Example user agent
  Apple-PubSub/65.28

• SEOprofiler

  Example user agent
  Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )

• BaiDu

  Example user agent
  Baiduspider

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-gt-s5350l) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0

• BOT for JCE

  Example user agent
  BOT/0.1 (BOT for JCE)

• thumbshots-de-bot

  Example user agent
  thumbshots-de-bot (+http://www.thumbshots.de/)

• Microsoft CryptoAPI

  Example user agent
  Microsoft-CryptoAPI/6.3

• WiseNut search engine crawler

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)

• Generic Research Crawler

  Example user agent
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)

• ZmEu

  Example user agent
  ZmEu

• NetNewsWire

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.74.9 (KHTML, like Gecko) Version/6.0 NetNewsWire/4.0.0

• woriobot

  Example user agent
  Mozilla/5.0 (compatible; woriobot +http://worio.com)

• WiseNut search engine crawler

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0

• PlantyNet

  Example user agent
  PlantyNet_WebRobot_V1.9 babo@plantynet.com

• GoGuidesBot

  Example user agent
  GoGuidesBot/0.0.1 (GoGuides Indexing Spider; http://www.goguides.org/spider.html)

• GarlikCrawler

  Example user agent
  GarlikCrawler/1.2 (http://garlik.com/

• AdsBot Google

  Example user agent
  AdsBot-Google (+http://www.google.com/adsbot.html)

• W3C-checklink

  Example user agent
  W3C-checklink/4.81 libwww-perl/5.836

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; ZTE U793 Build MocorDroid2.3.5) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• AskPeterBot

  Example user agent
  Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html)

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130207.001528 +http://webarchiv.cz/kontakty/)

• Lightspeed Systems RocketCrawler

  Example user agent
  LSSRocketCrawler/1.0 LightspeedSystems

• Fake Chrome

  Example user agent
  Mozilla/4.0 (Windows NT 6.2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.70 Safari/537.17

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-sgh-a657) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0

• hitcrawler

  Example user agent
  hitcrawler_0.1 (ringken@gmail.com)

• ZIBB Crawler

  Example user agent
  ZIBB Crawler (email address / WWW address)

• NewsBlur Favicon Fetcher

  Example user agent
  NewsBlur Feed Fetcher - 0 subscribers - http://www.newsblur.com/site/1719739/nouvelles-cinema (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.f

• Pingdom

  Example user agent
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com)

• msnbot-media

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/534+ (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 MsnBot-Media /1.0b

• libwww

  Example user agent
  Mozilla/4.04_(X11;_I;_OSF_3.2d)/1.0 libwww/4.0D

• HyperCrawl

  Example user agent
  Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)

• Adbeat Bot

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21

• Website Thumbnail Generator

  Example user agent
  Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)

• DotBot

  Example user agent
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)

• GO HttpClient

  Example user agent
  Go 1.1 package http

• JamesBOT

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.1; zh-cn; SCH-i559 Build FROYO) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• USyd-NLP-Spider

  Example user agent
  USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)

• Fake Safari

  Example user agent
  Mozilla/5.0 (iPad; CPU iPhone OS 501 like Mac OS X) AppleWebKit/534.46 (KHTML like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-sgh-t669) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile

• WebTarantula

  Example user agent
  WebTarantula.com Crawler

• Hatena Antenna

  Example user agent
  Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)

• SophosAgent

  Example user agent
  SophosAgent/1.0 (type= spa )

• UniversalFeedParser

  Example user agent
  UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; Lenovo S720 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• viralvideochart Bot

  Example user agent
  viralvideochart.unrulymedia.com

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; .NET CLR 2.0.50727;)

• WebRingChecker

  Example user agent
  WebRingChecker/1.71 LWP/5.79

• cURL

  Example user agent
  curl/7.16.0 (i386-unknown-openbsd4.1) libcurl/7.16.0 OpenSSL/0.9.7j zlib/1.2.3

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-kf600) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0

• Nutch

  Example user agent
  Innovazion Crawler/Nutch-1.7

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-s390g) UCBrowser8.3.1.161/69/405/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; HS-E860 Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• sReleaseBot

  Example user agent
  Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; htc touch viva t2223) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0

• n4p_bot

  Example user agent
  n4p_bot (crawler@n4p.com)

• Spinn3r RSS Aggregator

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19

• Open Vulnerability Assessment System

  Example user agent
  Mozilla/4.75 [en] (X11, U; OpenVAS)

• Google PageSpeed Insights

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36

• Google Bot

  Example user agent
  GoogleEarth/7.1.0002.2011(Android;Android (HTC One X-endeavoru-user-4.2.2);de-DE;kml:2.2;client:Free;type:default)

• Linkdex Bot

  Example user agent
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)

• General Crawlers

  Example user agent
  SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net),gzip(gfe),gzip(gfe)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; u: Android 2.3.4; en-us; ADR6400L Build/GRJ22) Applewebkit/533.1(KHTML, Like Gecko) Version/4.0 Mobile Safari533.1

• Fake Firefox

  Example user agent
  Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1

• Icarus6j

  Example user agent
  Icarus6j - (contact: phil@icarus6.com)

• Anonymizied

  Example user agent
  Mozilla/5.0 (Randomized by FreeSafeIP.com/upgrade-to-remove; compatible; MSIE 9.0; Windows NT 5.0WOW64) Chrome/18.0.1229.79

• Adbeat Bot

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86; nl-NL) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://netarkivet.dk/webcrawler/)

• Fake Android

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.2; fr-fr; Desire_A8181 Build/FRF91) App3leWebKit/53.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

• Fake Chrome

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/33.0.1750.91 Safari/537.36 OPR/20.0.1387.37 (Edition Next-Campaign 21)

• TrendWinHttp

  Example user agent
  TrendWinHttp/

• Wotbox

  Example user agent
  Wotbox/0.7-alpha (bot@wotbox.com; http://www.wotbox.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5380d) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.3.5; ja-jp; F-05D Build/F0001) AppleWebKit/533.1 (KHTML like Gecko) Version/4.0 Mobile Safari/533.1

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; R807 Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; V702NK)

• Mignify Bot

  Example user agent
  Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)

• BaiDu

  Example user agent
  Baiduspider+(+http://www.baidu.com/search/spider.htm)

• Fake Safari

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_1 like MacOS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D201 Safari/9537.53

• RSSOwl

  Example user agent
  RSSOwl/2.2.1.201312301314 (Windows; U; en)

• msnbot-NewsBlogs

  Example user agent
  msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)

• Internet Archive

  Example user agent
  Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler)

• Google Bot

  Example user agent
  Googlebot/2.1 (http://www.googlebot.com/bot.html)

• VoilaBot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)

• General Crawlers

  Example user agent
  Apache-HttpClient/UNAVAILABLE (java 1.4)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; samsung-gt-s7250d) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• SynHttpClient

  Example user agent
  SynHttpClient-Built/5759

• MQbot

  Example user agent
  MQbot http://metaquerier.cs.uiuc.edu/crawler

• MSN

  Example user agent
  msnbot/2.0 (+http://search.msn.com/msnbot.htm)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; T8830 Build HuaweiT8830) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• wget

  Example user agent
  Wget/1.11.4 Red Hat modified

• memoryBot

  Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)

• discoverybot

  Example user agent
  Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)

• WebmasterCoffee

  Example user agent
  Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about)

• JobdiggerSpider

  Example user agent
  Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; HS-EG906 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Web Glance

  Example user agent
  Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1

• BaiDu

  Example user agent
  baiduspider-mobile-gate

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)

• Links4US-Crawler

  Example user agent
  Links4US-Crawler, ( http://links4us.com/)

• LoadTimeBot

  Example user agent
  Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)

• AdsBot Google

  Example user agent
  AdsBot-Google

• Xenus Link Sleuth

  Example user agent
  Xenu Link Sleuth 1.1f

• FeedDemon

  Example user agent
  FeedDemon/4.5 (http://www.feeddemon.com/; Microsoft Windows XP)

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot)

• Envolk

  Example user agent
  envolk/1.7 ( http://www.envolk.com/envolkspider.html)

• Amfibi

  Example user agent
  Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)

• Fake IE

  Example user agent
  "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)"

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s5753e) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0

• Apt HTTP Transport

  Example user agent
  Debian APT-HTTP/1.3 (1.0.1ubuntu2)

• Research Projects

  Example user agent
  CMS crawler ( http://buytaert.net/crawler/)

• Socialradarbot

  Example user agent
  Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com)

• MarketwireBot

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com)

• Sistrix Crawler

  Example user agent
  Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)

• Google Bot

  Example user agent
  Google (+https://developers.google.com/+/web/snippet/)

• XoviBot

  Example user agent
  Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)

• Fake Safari

  Example user agent
  Mozilla/6.0 (iPad; CPU iPad OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25

• Google Structured-Data TestingTool

  Example user agent
  Mozilla/5.0 (compatible; X11; Linux x86_64; Google-StructuredDataTestingTool; +http://www.google.com/webmasters/tools/richsnippets)

• Rogerbot

  Example user agent
  rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)

• ShowyouBot

  Example user agent
  ShowyouBot (http://showyou.com/crawler)

• Feeddler RSS Reader

  Example user agent
  FeeddlerRSS 2.4 (iPad; iPhone OS 7.1; en_GB)

• Esribot

  Example user agent
  Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)

• Google Bot

  Example user agent
  Googlebot/2.1 ( http://www.googlebot.com/bot.html)

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)

• libwww

  Example user agent
  NCSA Mosaic/2.6b1 (X11;UNIX_System_V 4.2 R4000) libwww/2.12 modified

• Powermarks

  Example user agent
  Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT)

• Fake Browser

  Example user agent
  http://www.Syntryx.com/ ANT Chassis 9.27; Mozilla/4.0 compatible crawler

• Google Web Preview

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36

• PDFBot

  Example user agent
  PDFBot (crawler@pdfind.com)

• LinkpadBot

  Example user agent
  Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)

• SurveyBot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; X907 Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; DM003SH)

• VoilaBot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)

• Zookabot

  Example user agent
  Zookabot/2.5;++http://zookabot.com

• QuickFinder

  Example user agent
  QuickFinder Crawler

• FacebookExternalHit

  Example user agent
  facebookexternalhit/1.1

• NewsBlur Favicon Fetcher

  Example user agent
  NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• Google Bot

  Example user agent
  Googlebot/2.1 (+http://www.googlebot.com/bot.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; JY-G2 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Integrity

  Example user agent
  integrity/4

• Nagios

  Example user agent
  check_http/v1.4.16 (nagios-plugins 1.4.16)

• EveryoneSocialBot

  Example user agent
  Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; En-us; Samsung-gt-m8910) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0

• Website Thumbnail Generator

  Example user agent
  Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generat

• BabalooSpider

  Example user agent
  BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)

• grub

  Example user agent
  Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org)

• Shrook

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/434l

• Fake Chrome

  Example user agent
  Mozilla/5.0 (X11; Windows x86_64) AppleWebKit/536.11 (KHTML, like Gecko) chrome/22.0.1190.0 Safari/536.11

• TestCrawler

  Example user agent
  ldspider (BTC 2011 crawl, harth@kit.edu, http://code.google.com/p/ldspider/wiki/Robots)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; aplix jblend) UCBrowser8.3.0.154/69/444/UCWEB Mobile, MOT-A3100/1.0 Release/09.09.2008 Profile/MIDP-2.0 Configuration/CLDC-1.1 Software/WM6.1 UNTRUSTED/1.0

• Windows-Update-Agent

  Example user agent
  Windows-Update-Agent/7.9.9600.18094 Client-Protocol/1.21

• Google Bot

  Example user agent
  GoogleEarth/7.1.0003.1255(Android;Android (LG-E986-geefhd-user-4.1.2);de-DE;kml:2.2;client:Free;type:default)

• Zend_Http_Client

  Example user agent
  Zend_Http_Client

• Willow Internet Crawler

  Example user agent
  Willow Internet Crawler by Twotrees V2.1

• Comodo Spider

  Example user agent
  Comodo Spider 1.1

• YellowPages

  Example user agent
  yp-crawl@attinteractive.com

• YahooSeeker

  Example user agent
  YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)

• NaverBot

  Example user agent
  NANaverNDrive/3.0.0(Android OS 4.2.2;samsung SHV-E300L)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung gt-b5722c) UCBrowser8.3.0.154/69/444/UCWEB Mobile

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung sgh-d980) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile

• FAST Enterprise Crawler

  Example user agent
  RB2B-bot v0.1 (Using Fast Enterprise Crawler 6 search@reedbusiness.com)

• semantic-visions.com crawler

  Example user agent
  Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-gd880) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• ArchiveBot

  Example user agent
  ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36

• General Crawlers

  Example user agent
  UCWEB/2.0 (Java; U; MIDP-2.0; 11; GT-S5230) U2/1.0.0 UCBrowser UCVN.MOBI/9.4.1.377 U2/1.0.0 Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (LG-C510 AppleWebKit/531 MMS/LG-MMS-V1.0/1.2 MediaPlayer/LGPlayer/1.0 Java/ASVM/1.1 Profile/MIDP-2.1),gzip(gfe),gzip(gfe)

• Seznam Screenshot Generator

  Example user agent
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)

• AjaxSnapBot

  Example user agent
  Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html)

• BaiDu

  Example user agent
  Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search

• Slackbot-Link-Expanding

  Example user agent
  Slackbot-LinkExpanding+(+https://api.slack.com/robots)

• akregator

  Example user agent
  Akregator/4.12.3; syndication SUSE

• Fake Browser

  Example user agent
  Mozzila/5.0(Linux; U; Android 2.3.5; ru-ru; Fly IQ239 Build/JZ054K) AppleWebKit/533.1(KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 4.4.2; en-us; D2202 Build/18.4.A.1.25) AppleWebKit/533.1 (KHTML like Gecko) Version/4.0 UCBrowser/9.9.2.467 U3/0.8.0 Mobile Safari/533.1

• DNS Group Crawler

  Example user agent
  DNSGroup/0.1 (DNS Group Crawler; http://www.dnsgroup.com/; crawler@dnsgroup.com)

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

• 5Bot

  Example user agent
  Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s3370) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; Lenovo A690 Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• AcoonBot

  Example user agent
  Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)

• General Crawlers

  Example user agent
  HuaweiU1250/B002 Browser/Access-Browser/Q05A MMS/Obigo-MMS/Q05A SyncML/HW-SyncML/1.0 Java/HWJa/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Player/QTV-Player/5.3

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; samsung-s5610) UCBrowser8.3.0.154/69/444/UCWEB Mobile

• Gigabot

  Example user agent
  Gigabot/3.0 (http://www.gigablast.com/spider.html)

• Fake Firefox

  Example user agent
  Mozilla/5.0 (Mobile; rv:32.0) Gecko/20100101 Firefox/32.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; A900 Build MocorDroid2.3.5) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 4.0 - GetMeLinked Spider www.GetMeLinked.com Web Directory

• Steeler

  Example user agent
  Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

• m65bot

  Example user agent
  DoCoMo/2.0 P904i( m65bot/0.1; c; http://m65.jp/bot.html )

• DuckDuck Favicons Bot

  Example user agent
  Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)

• Ichiro Mobile Bot

  Example user agent
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)

• .NET Framework CLR

  Example user agent
  MS Web Services Client Protocol 1.0.3705.0

• General Crawlers

  Example user agent
  SonyEricssonw760i/R4EA Java/SEMC-Java/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0, sonyericssonw760i

• Java Standard Library

  Example user agent
  Java/1.8.0_25

• MegaIndex Bot

  Example user agent
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)

• Sproose

  Example user agent
  sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com)

• HRCrawler

  Example user agent
  HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko)

• BingPreview

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; SCH-I779 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-c3262) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile, SAMSUNG-GT-C3262/C3262DDLHA Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; GT-I9008L Build FROYO) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Google Web Preview

  Example user agent
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• Y!J-AGENT

  Example user agent
  Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

• HiddenMarket

  Example user agent
  HiddenMarket-1.0-beta (www.hiddenmarket.net/crawler.php)

• NerdyBot

  Example user agent
  NerdyBot

• General Crawlers

  Example user agent
  SonyEricssonJ20i/R7CA Profile/MIDP-2.1 Configuration/CLDC-1.1 UCWEB/2.0 (Java; U; MIDP-2.0; xx; SonyEricssonJ20i) U2/1.0.0 UCBrowser/9.4.1.377 U2/1.0.0 Mobile

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 4.1.1; bg-bg; Xenta TAB09-211 Build/XentaTAB09-211) AppleWebKit/534.30 (KHTML like Gecko) Version/4.0 Safari/534.30

• Generic Research Crawler

  Example user agent
  Research spider - ak1835@albany.edu

• Google Search Appliance

  Example user agent
  gsa-crawler

• WooRank

  Example user agent
  woobot/1.1

• Dolphin smalltalk http client

  Example user agent
  Dolphin http client/10.3.0(280) (Android)

• Google Bot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• Crawlboy

  Example user agent
  CRAWL-E/0.6.4

• MonkeyCrawl

  Example user agent
  MonkeyCrawl/0.05 (MonkeyCrawl; http://www.monkeymethods.org; )

• Ad Muncher

  Example user agent
  Ad Muncher v4.93.33707/4957

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-c3510t) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Symbian/3; Series60/5.2 NokiaN8-00/00.0.000; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebkit/525 (KHTML, like Gecko) Version/3 BrowserNG/7.3.1.8 3gpp-gba,gzip(gfe),gzip(gfe),gzip(gfe)

• Fake Chrome

  Example user agent
  24.0.1284.0.0 (Windows NT 6.1) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/24.0.1284.0.3.742.2 Safari/534.0

• BlogPulse

  Example user agent
  BlogPulse (ISSpider-3.0)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.3; zh-cn; SCH-i809 Build FROYO) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• QuerySeekerSpider

  Example user agent
  QuerySeekerSpider ( http://queryseeker.com/bot.html )

• OpenWebSpider

  Example user agent
  OpenWebSpider v0.1.4 (http://www.openwebspider.org/)

• BingPreview

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0) BingPreview/1.0b

• Google Bot

  Example user agent
  GoogleEarth/7.1.0003.1255(Android;Android (HTC Desire X-protou-user-4.1.1);de-DE;kml:2.2;client:Free;type:default)

• Moreover

  Example user agent
  Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)

• BingBot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)

• Crawlboy

  Example user agent
  Crawlzilla/1.0 (Crawlzilla; http://www.crawlzilla.com; crawler@crawlzilla.com)

• AdsBot Google

  Example user agent
  AdsBot-Google ( http://www.google.com/adsbot.html)

• netEstate NE Crawler

  Example user agent
  netEstate NE Crawler (+http://www.sengine.info/)

• DomainAppender Bot

  Example user agent
  Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; HUAWEI T8300 Build FRF91) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5

• gooblog

  Example user agent
  gooblog/2.0 (http://help.goo.ne.jp/contact/)

• MetaInspector

  Example user agent
  MetaInspector/3.0.0 (+https://github.com/jaimeiniesta/metainspector)

• SemrushBot

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)

• Fake Browser

  Example user agent
  \x22Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.36

• Google Bot

  Example user agent
  Googlebot

• ArchitextSpider

  Example user agent
  ArchitextSpider

• TweetedTimes Bot

  Example user agent
  Mozilla/5.0 (compatible; TweetedTimes Bot/1.0 (resolver); +http://tweetedtimes.com)

• DotBot

  Example user agent
  DotBot/1.0.1 (http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s3653) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Firefox

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7

• Steeler

  Example user agent
  Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

• Internet Archive Special Archiver

  Example user agent
  Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)

• T-Online Browser

  Example user agent
  T-Online Browser

• lbot

  Example user agent
  Mozilla/5.0 (lbot/2.6.4) lbot@mailme.not

• Reddit

  Example user agent
  Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• Uipbot

  Example user agent
  uipbot/1.0 (uipbot@semasio.net)

• Fake Browser

  Example user agent
  Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)

• libwww

  Example user agent
  NCSA_Mosaic/2.7b5 (X11;Linux 2.6.7 i686) libwww/2.12 modified

• AntBot

  Example user agent
  AntBot/1.0 (http://www.ant.com)

• optivo NetHelper

  Example user agent
  optivo(R) NetHelper

• R6 CommentReader

  Example user agent
  R6_CommentReader(www.radian6.com/crawler)

• Another Web Mining Tool

  Example user agent
  Mozilla/5.0 (compatible; Another Web Mining Tool 1.0; +none; awmt)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s8600) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0

• Yahoo Pipes

  Example user agent
  Yahoo Pipes 2.0

• Squid

  Example user agent
  BREW-Applet/0x20068888 (BREW/3.1.5.20; DeviceId: 90197; Lang: zhcn) ucweb-squid

• WeSEE:Search

  Example user agent
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)

• Sogou Orion spider

  Example user agent
  Sogou Orion spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible; MSIE 6.1; Windows NT)

• WDG HTML Validator

  Example user agent
  WDG_Validator/1.6.5

• Fake Safari

  Example user agent
  Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2; zh-cn; LG-P503 Build FRF91) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake IE

  Example user agent
  Mozilla/5.0 (MOT-A3300c/A3300c_03.08; U; Windows Mobile/6.1; Profile/MIDP-2.0 Configuration/CLDC-1.1; 240*400; CTC/2.0) IE/MSIE 6.0 (compatible; MSIE 6.0; Windows CE; IEMobile 8.12; MSIEMobile 6.0)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-gt-b2710) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/84/352/UCWEB Mobile UNTRUSTED/1.0

• Googlebot-News

  Example user agent
  Googlebot-News

• gosquared-thumbnailer

  Example user agent
  gosquared-thumbnailer/1.0

• AdSense Snapshot Bot

  Example user agent
  Adsense-Snapshot-Google

• open-webkit-sharp

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) OpenWebKitSharp

• SocialSpider-Finder

  Example user agent
  SocialSpider-Finder/0.2

• profiller

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) profiller Safari/537.21

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/538.6 HbbTV/1.1.1 (+DL+PVR; inverto; QUANTUM - IDL9000; 1.0; 1.0;) CE-HTML/1.0 hdplusinteraktiv/1.0 (NETRANGEMMH;)

• Bloglovin Bot

  Example user agent
  Bloglovin/1.0 (http://www.bloglovin.com; 1 subscribers)

• MailChimp.com

  Example user agent
  MailChimp.com

• WordPress

  Example user agent
  WordPress 4.0-alpha Feed Client Mozilla/5.0 Compatible

• Google Wireless Transcoder

  Example user agent
  Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14

• General Crawlers

  Example user agent
  Mozilla/5.0 (compatible; Ex-Crawler/v0.1.5ALPHA SVN rev58; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)

• MixBot

  Example user agent
  MixBot (+http://t.co/GSRLLKex24)

• SiteCon

  Example user agent
  Mozilla/5.0 (Linux; U; en-gb) AppleWebKit/418.9.1 (KHTML, like Gecko) SiteCon/8.10.9

• General Crawlers

  Example user agent
  Mozilla/5.0 (compatible; Ex-Crawler/0.1.5a; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)

• NetNewsWire

  Example user agent
  NetNewsWire/4.0.0 (Mac OS X; http://netnewswireapp.com/mac/; gzip-happy)

• Mozdex

  Example user agent
  mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)

• Boitho

  Example user agent
  boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )

• Boitho

  Example user agent
  boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html )

• cars-app-browser

  Example user agent
  cars-app-browser

• Internet Archive

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)

• city-map screenshot service

  Example user agent
  city-map screenshot service

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsungsgh-b7330) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile, SAMSUNG-GT-B7330/1.0 UNTRUSTED/1.0

• worldwebheritage.org Bot

  Example user agent
  Mozilla/5.0 (compatible; worldwebheritage.org/1.0; +crawl@worldwebheritage.org)

• Semantic Discovery

  Example user agent
  semanticdiscovery/0.2(http://www.semanticdiscovery.com/sd/robot.html)

• TestCrawler

  Example user agent
  AppCodes crawler - looking for iOS app mentions. More info: support@appcodes.com. Robots.txt id: AppCodesCrawler

• Fake Firefox

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11;

• Scrubby

  Example user agent
  Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; smart; ZAPPIX HD+; 1.0; 1.0;) CE-HTML/1.0

• Fake IE

  Example user agent
  Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)

• Android Download Manager

  Example user agent
  AndroidDownloadManager/4.1.1 (Linux; U; Android 4.1.1; POV_TAB-P1325(V1.1) Build/JRO03H)

• ca-crawler

  Example user agent
  Mozilla/5.0 (compatible; ca-crawler/1.0)

• Larbin

  Example user agent
  08_800w_web1 (larbin2.6.3@unspecified.mail)

• RamblerMail Bot

  Example user agent
  RamblerMail/6.0 (incompatible; ImageProxy/6.0)

• BingBot

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

• MJ12bot

  Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5260) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0

• MojeekBot

  Example user agent
  Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)

• Owler Bot

  Example user agent
  Mozilla/5.0 (compatible; Owler/0.4; +; )

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.1; zh-cn; 3GC101 Build FRG83) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Nutch

  Example user agent
  ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com)

• NewsBlur Favicon Fetcher

  Example user agent
  NewsBlur Page Fetcher - 7 subscribers - http://www.newsblur.com/site/3966817/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• Ichiro Bot

  Example user agent
  ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Symbian/3; U; E6-00; de-DE; NokiaE6-00) AppleWebKit/534.3 (KHTML, like Gecko) Shazam Mobile Safari/534.3

• Optimizer Bot

  Example user agent
  Mozilla/5.0 (compatible; Optimizer)

• Baidu Image Search

  Example user agent
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; ME722 Build MLS2GC_2.5.0) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; ZTE N880G Build IMM76I) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• ObjectsSearch

  Example user agent
  ObjectsSearch/0.2 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)

• Google Bot

  Example user agent
  GoogleEarth/7.1.0002.2011(Android;Android (HTC One mini-m4-user-4.2.2);de-DE;kml:2.2;client:Free;type:default)

• PageFreezer

  Example user agent
  PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)

• BOTW Spider

  Example user agent
  Mozilla/4.0 (compatible; BOTW Spider; http://botw.org)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.10; Windows NT 6.1; Trident/5.0)

• Securepoint Content Filter

  Example user agent
  securepoint cf

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-a133) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-t310i) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-i900) UCBrowser8.2.0.132/69/452/UCWEB Mobile, SAMSUNG-SGH-i900/1.0 UNTRUSTED/1.0

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; 705NK)

• Google App Engine

  Example user agent
  AppEngine-Google; ( http://code.google.com/appengine)

• ScoutJet

  Example user agent
  Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-id; samsung gt-c3303i) UCBrowser8.2.1.144/69/444/UCWEB Mobile

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0 ; http://help.yahoo.com/help/us/ysearch/slurp)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-t300) UCBrowser8.2.1.144/69/444/UCWEB Mobile

• eZ Publish Link Validator

  Example user agent
  eZ Publish Link Validator

• General Crawlers

  Example user agent
  Epic Empire/1.80 pgid=454af7cb350c2a1b tag=454af7cb350c2a1b cv=180011 hw=ASUS ME371MG; ME371MG; gzip; JavaHttp

• vkShare

  Example user agent
  Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-s3650w) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• CloudFlare AlwaysOnline

  Example user agent
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34

• PagesInventory Bot

  Example user agent
  PagesInventory (robot http://www.pagesinvenotry.com)

• Crawlboy

  Example user agent
  CrawlerProcess (http://www.PowerMapper.com) /5.5.678.0

• Nutch

  Example user agent
  Your Nutch Spider/Nutch-2.2.1

• NaverBot

  Example user agent
  Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)

• WooRank

  Example user agent
  woobot/2.0

• AccServer

  Example user agent
  AccServer[HOGANMIDSCHOOL]/5.0.2.20(24517) 64-bit HTTP-Agent

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; Lenovo A288t Build MocorDroid2.3.5) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• msnbot

  Example user agent
  msnbot/2.0b ( http://search.msn.com/msnbot.htm)

• RssBandit

  Example user agent
  RssBandit/1.9.0.1002 (.NET CLR 2.0.50727.7512; WinNT 6.2.9200.0; http://www.rssbandit.org)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung sgh-zx20) UCBrowser8.4.0.159/70/352/UCWEB Mobile

• Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2,01

• feedly Feed Fetcher

  Example user agent
  Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)

• Locubot

  Example user agent
  Locubot (compatible; Googlebot; msnbot)

• PEAR-PHP

  Example user agent
  PEAR HTTP_Request class

• ADmantX Platform Semantic Analyzer

  Example user agent
  ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; HUAWEI c8810 Build 2.3.6_anzhi_VenXie_V3 Yi23) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Nutch

  Example user agent
  GrowerIdeas Crawler/GrowerIdeas-nutch-1.6 (Crawls URLs for indexing content for our new search startup which aims to provide simple and smart search across curated content. For more info please contact helloworld@growerideas.com. If you think

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.4; zh-cn; SCH-i579 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-kc550 teleca) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/405/UCWEB Mobile UNTRUSTED/1.0

• WebAlta Crawler

  Example user agent
  WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; samsung-sgh-d880) UCBrowser8.3.0.154/69/444/UCWEB Mobile

• Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2.40

• ThumbShotsBot

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86_64; de-DE; rv:1.9.0.19) Gecko/2010091808 ThumbShotsBot (KFSW 3.0.6-3)

• ScreenerBot

  Example user agent
  ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)

• Google Image Proxy

  Example user agent
  Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-a697) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• Windows Phone Search

  Example user agent
  Windows Phone Search (Windows Phone OS 8.0;NOKIA;Nokia 520;8.0;10328)

• Baidu Mobile Search

  Example user agent
  DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0; http://www.baidu.jp/spider/)

• SeznamBot

  Example user agent
  SeznamBot/3.0 (+http://fulltext.sblog.cz/)

• Nutch

  Example user agent
  NutchSpider/Nutch-1.4

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; PT34)

• Patwebbot

  Example user agent
  Patwebbot (http://www.herz-power.de/technik.html)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; samsung-gt-b7722i) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-gb175) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile

• LOOQ

  Example user agent
  LOOQ/0.1 alfa (LOOQ Crawler for european sites; http://looq.eu; root (at) looq dot eu)

• grapeFX

  Example user agent
  Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk

• Fake Chrome

  Example user agent
  =Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/538.8 HbbTV/1.1.1 (+DL+PVR; inverto; IDL-6651N Volksbox Web Edition; 1.0; 1.0;) CE-HTML/1.0

• General Crawlers

  Example user agent
  Squrl Java/1.6.0_22

• webceo Bot

  Example user agent
  Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)

• akregator

  Example user agent
  Akregator/1.2.9; librss/remnants

• Google Bot

  Example user agent
  Googlebot (gocrawl v0.4)

• Forum Poster

  Example user agent
  Forum Poster V3

• Google Bot

  Example user agent
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• Fake Firefox

  Example user agent
  "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko Firefox/11.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s5333) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5620) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; inverto; IDL-6640N Volksbox Essential; 1.0; 1.0;) hdplusinteraktiv/1.0 (NETRANGEMMH;) CE-HTML/1.0

• Ezooms

  Example user agent
  Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-sgh-f400) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0

• Ichiro Mobile Bot

  Example user agent
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)

• Goldfire Server

  Example user agent
  Goldfire Server

• Daumoa

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0

• Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2.22

• libwww

  Example user agent
  NCSA_Mosaic/2.6L10N+ (X11;SunOS 5.6 sun4u) libwww/2.12 modified

• Accoona Bot

  Example user agent
  Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)

• Blog Search

  Example user agent
  Mozilla/4.0 (compatible; Blog Search;)

• Google Image Proxy

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com GoogleImageProxy)

• akregator

  Example user agent
  Mozilla/5.0 (X11;) AppleWebKit/534.34 (KHTML, like Gecko) akregator/4.14.1 Safari/534.34

• Exabot

  Example user agent
  Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)

• YandexBot

  Example user agent
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

• BingPreview

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 (KHTML, like Gecko) BingPreview/1.0b

• Internet Archive

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)

• Yahoo-MMCrawler

  Example user agent
  Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; smart; VX10; 1.0; 1.0;) CE-HTML/1.0

• Java Standard Library

  Example user agent
  Java/1.7.0_71

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-lgc300) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• FAST-WebCrawler

  Example user agent
  FAST-WebCrawler/2.2.11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)

• WIRE

  Example user agent
  WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler)

• Ichiro Bot

  Example user agent
  ichiro/1.0 (ichiro@nttr.co.jp)

• Sogou Pic Spider

  Example user agent
  Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• http_requester

  Example user agent
  http_requester/0.1

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.2.2; en-gb; DesireA8181 Build/FRG83G) AppleWebKit/533.1 (KHTML like Gecko) Version/4.0 Mobile Safari/533.1

• BitlyBot

  Example user agent
  bitlybot

• wget

  Example user agent
  Wget/1.11.4

• HubSpot Webcrawler

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler

• Baiduspider

  Example user agent
  Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

• NetSurf

  Example user agent
  NetSurf/2.0 (RISC OS; armv5l)

• Omea Reader

  Example user agent
  JetBrains Omea Reader 2.2 (http://www.jetbrains.com/omea/reader/)

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; inverto; IDL 6654N Volksbox Web Edition +; 1.0; 1.0;) hdplusinteraktiv/1.0 (NETRANGEMMH;) CE-HTML/1.0

• Rogerbot

  Example user agent
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; Lenovo S850e Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• YandexImageResizer

  Example user agent
  Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-kf510 obigo) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0

• SeznamBot

  Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)

• Fake Browser

  Example user agent
  Mozilla 4.0(compatible; BotSeer/1.0; http://botseer.ist.psu.edu)

• msnbot-media

  Example user agent
  msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

• Larbin

  Example user agent
  binlar_2.6.3 test@mgmt.mic

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; Lenovo P700i Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• DomainCrawler

  Example user agent
  DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/bot

• DomainScan Server Monitoring

  Example user agent
  DomainSCAN

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.4; zh-cn; XT882 Build SWDFS_M7_4.80.0) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• MonoBot

  Example user agent
  Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)

• URLAppendBot

  Example user agent
  Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.6; Windows NT 6.1; Trident/5.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727) 3gpp-gba UNTRUSTED/1.0

• Python-urllib

  Example user agent
  Python-urllib/1.15

• Squid

  Example user agent
  BREW-Applet/0x20068888 (BREW/3.1.5.20; DeviceId: 32056; Lang: zhcn) ucweb-squid

• PluckItCrawler

  Example user agent
  PluckItCrawler/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com;)

• Sogou Head Spider

  Example user agent
  Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; ZTE V889D Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• grub

  Example user agent
  grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/4.0; GTB7.4; InfoPath.3; SV1; .NET CLR 3.1.76908; WOW64; en-US)

• Fake Browser

  Example user agent
  Mozila/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)

• W3C Unicorn

  Example user agent
  W3C_Unicorn/1.0 (http://validator.w3.org/services)

• libwww

  Example user agent
  WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803

• Generic Research Crawler

  Example user agent
  Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru)

• Envolk

  Example user agent
  envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)

• Diffbot

  Example user agent
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15

• Semantic Discovery

  Example user agent
  semanticdiscovery/2.0(http://www.semanticdiscovery.com/robot.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; HTC T328d Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s7233e) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• Slackbot

  Example user agent
  Slackbot (+https://api.slack.com/robots)

• Qwantify

  Example user agent
  Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)

• Pete-Spider

  Example user agent
  Pete-Spider/1.1

• Finderlein Research Crawler

  Example user agent
  Finderlein Research Crawler 1.0

• acapbot

  Example user agent
  Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; ME860 Build OLHKT_U4_0.56.0) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• BaiDu

  Example user agent
  'Mozilla/5.0 (compatible; Baiduspider/2.0; +ht'

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; samsung-gt-s8530) UCBrowser8.2.1.144/70/352/UCWEB Mobile UNTRUSTED/1.0

• Rival IQ Bot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)

• NaverBot

  Example user agent
  NaverWebToon/1.4.4(Android OS 4.0.4;samsung YP-GP1)

• test certificate info

  Example user agent
  test certificate info

• Scrapy

  Example user agent
  Scrapy/0.16.4 (+http://scrapy.org)

• IconSurf

  Example user agent
  IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)

• TurnitinBot

  Example user agent
  TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)

• OWASP_SECRET_BROWSER

  Example user agent
  OWASP_SECRET_BROWSER/3.0

• General Crawlers

  Example user agent
  JNLP/6.0 javaws/1.6.0_14 (b08) Java/1.6.0_14

• General Crawlers

  Example user agent
  SonyEricssonW20a/R7CA Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (Java; U; Id; sonyericssonw20a) UCBrowser8.3.0.154/70/444/UCWEB Mobile

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/4.0; InfoPath.2; SV1; .NET CLR 2.0.50727; WOW64)

• Wada.vn Search Bot

  Example user agent
  Wada.vn Vietnamese Search

• bl.uk_lddc_bot

  Example user agent
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; MOT-XT788 Build IRPMCT_B_02.60.00RPD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Windows-RSS-Platform

  Example user agent
  Windows-RSS-Platform/2.0 (MSIE 9.0; Windows NT 6.0)

• Crawler

  Example user agent
  Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)

• Internet Archive Special Archiver

  Example user agent
  Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)

• Fake Chrome

  Example user agent
  Chrome/15.0.860.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/15.0.860.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; HS-U8 Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-a887) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Internet Archive

  Example user agent
  InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

• Shrook

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/1290y

• SoftListBot

  Example user agent
  Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)

• BingBot

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0 http://www.bing.com/bingbot.htm)

• Fake Chrome

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36

• YioopBot

  Example user agent
  Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)

• GroupHigh Bot

  Example user agent
  Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)

• Website Thumbnail Generator

  Example user agent
  Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)

• Picsearch Bot

  Example user agent
  psbot-page (+http://www.picsearch.com/bot.html)

• Fake Browser

  Example user agent
  Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36

• Nutch

  Example user agent
  testnutch/Nutch-1.8

• Fake Browser

  Example user agent
  Opera/9.80 (Window NT 6.2; Win64; x64) Presto/2.12.388 Version/12.16

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; E920 Build GRJ90) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/538.8 HbbTV/1.1.1 (+DL+PVR; inverto; QUANTUM - IDL9000; 1.0; 1.0;) CE-HTML/1.0 hdplusinteraktiv/1.0 (NETRANGEMMH;)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.11; Windows NT 6.2; Trident/5.0)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; 830CA)

• JobBoerse Bot

  Example user agent
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0

• Univention Corporate Server

  Example user agent
  UCS (ESX) - 4.0-3 errata302 - 28d414cc-2dac-4c0e-a34a-734020b8af66 - 00000000-0000-0000-0000-000000000000 -

• Windows Phone Search

  Example user agent
  Windows Phone Search (Windows Phone OS 7.10;NOKIA;Lumia 900;7.10;8862)

• Y!J-AGENT

  Example user agent
  Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/

• bl.uk_lddc_bot

  Example user agent
  Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxyfile1-hrd)

• Nutch

  Example user agent
  Spider/Nutch-2.3-SNAPSHOT (Webcrawler)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; lg-gm360) UCBrowser8.2.1.144/69/405/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.7; zh-cn; ZTE U960s2 Build GWK74) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-gt365) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0

• Nutch

  Example user agent
  spider/Nutch-1.5.1 (spider; http://www.xxx.com)

• Flatland

  Example user agent
  flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)

• Wotbox

  Example user agent
  Wotbox/2.01 (+http://www.wotbox.com/bot/)

• FAST-WebCrawler

  Example user agent
  FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; Lenovo A698t Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• python-requests

  Example user agent
  python-requests/2.5.1 CPython/2.7.6 Linux/3.13.0-36-generic

• DotBot

  Example user agent
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)

• Squid

  Example user agent
  BREW-Applet/0x20068888 (BREW/3.1.5.20; DeviceId: 90086; Lang: zhcn) ucweb-squid

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-gs170 browser) UCBrowser8.4.0.159/70/352/UCWEB Mobile

• Cyberduck

  Example user agent
  Cyberduck/4.6.4 (16610) (Windows XP/5.1) (x86)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; htc touch hd t8282) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Nutch

  Example user agent
  NutchOSUOSL/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; vivo E3 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.2.2; zh-cn; SCH-I959 Build JDQ39) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Entireweb

  Example user agent
  Speedy Spider (http://www.entireweb.com)

• Second Live Commerce Client

  Example user agent
  Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0 ; .NET CLR 2.0.50215; SL Commerce Client v1.0; Tablet PC 2.0

• FeedBurner

  Example user agent
  FeedBurner/1.0 (http://www.FeedBurner.com)

• WorldLingo

  Example user agent
  WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com)

• bandscraper

  Example user agent
  bandscraper

• BingBot

  Example user agent
  Mozilla/5.0 (compatible;bingbot/2.0;+http://www.bing.com/bingbot.htm)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s3650) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0

• hivaBot

  Example user agent
  hivaBot(http://gorgor.ir and info@gorgor.ir)/hivaBot-0.2

• Yahoo Mobile App

  Example user agent
  YahooMobile/1.0 (Yahoo! Search; 3.1.2) (Apple; iPod touch; iPhone OS/7.1.1); YHOO_Search_App/3.1.2

• BuiBui-Bot

  Example user agent
  BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com)

• Python-urllib

  Example user agent
  Python-urllib/3.4

• hivaBot

  Example user agent
  hivaBot/hivaBot-0.2 (info@gorgor.ir)

• focuseekbot

  Example user agent
  Mozilla/4.0 (compatible; focuseekbot)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-e2220) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile, SAMSUNG-GT-E2220/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0

• TYPO3 Linkvalidator

  Example user agent
  Mozilla/5.0 TYPO3-linkvalidator

• msnbot

  Example user agent
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)

• BigBozz - Financial Search

  Example user agent
  BigBozz/BigBozz-2.2.1

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.4; zh-cn; Lenovo A1-32AB0 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.2; zh-cn; SCH-I829 Build JZO54K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; HI3E)

• FHScan Core

  Example user agent
  Mozilla/5.0 (FHScan Core 1.1)

• Y!J-AGENT

  Example user agent
  Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)

• pdrlabs Bot

  Example user agent
  Cloud mapping experiment. Contact research@pdrlabs.net

• Cookie Reports Bot

  Example user agent
  CookieReports.com/0.1

• fastbot crawler

  Example user agent
  fastbot crawler beta 4.0 (+http://www.fastbot.de)

• SetLinks.ru Crawler

  Example user agent
  Mozilla/5.0 (compatible; SetLinks bot 1.2)

• Yahoo Link Preview

  Example user agent
  Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-lg800g) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• Applebot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

• JobRoboter

  Example user agent
  JobRoboter/1.1

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; ZTE N880E Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• BingPreview

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko

• MS OPD

  Example user agent
  Microsoft Office Protocol Discovery

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; Coolpad W706 Build FRG83G) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  \x22Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b\x22

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; ZTE U795+ Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• 360Spider

  Example user agent
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider

• feedly App

  Example user agent
  FeedlyApp/1.0 (http://www.feedly.com)

• Abonti WebSearch

  Example user agent
  Mozilla/5.0 (compatible; Abonti/0.92 - http://www.abonti.com)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.0; SN3F)

• Crawlboy

  Example user agent
  CrawlFire - you can disable this Robot: http://pastebin.de/25277

• msnbot-media

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b

• UnisterBot

  Example user agent
  Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)

• DBot

  Example user agent
  Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com)

• Nett.io bot

  Example user agent
  Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; .NET4.0C)

• TestCrawler

  Example user agent
  WEBB Crawler - see: http://badcheese.com/robots.html

• A6-Indexer

  Example user agent
  A6-Indexer

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-t500) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsungsgh-i627) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile, SAMSUNG-SGH-i780/1.0 UNTRUSTED/1.0

• NewsGator

  Example user agent
  NewsGator/3.0,gzip(gfe),gzip(gfe)

• WNMbot

  Example user agent
  WNMbot/0.4.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; lg-gm360i) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0

• python-requests

  Example user agent
  python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-24-generic

• Python

  Example user agent
  pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp)

• OpenBSD ftp

  Example user agent
  OpenBSD ftp

• PEAR HTTP_Request2

  Example user agent
  HTTP_Request2/2.1.1 (http://pear.php.net/package/http_request2) PHP/5.3.10-1ubuntu3.6

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; ZTE U795 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5330) UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0

• FAST Enterprise Crawler

  Example user agent
  FAST Enterprise Crawler/5.3.4 (crawler@fast.no)

• Iframely Bot

  Example user agent
  Iframely/0.7.1 (+http://iframe.ly/;)

• Genieo Web Filter

  Example user agent
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; samsung-gt-s5380f) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0

• Superfeedr Bot

  Example user agent
  Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; htc snap s521) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.9; zh-cn; HTC EVO 4G+ Build HTC EVO 4G+) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Google Web Snippet

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b5310l) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-sgh-a777) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozzila/5.0(Linux; Android 4.4.2; ru-ru; Fly IQ4415 Quad Build/KOT49H) AppleWebKit/537.36(KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36;

• SemrushBot

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot/0.97sa; +http://www.semrush.com/bot.html)

• BaiduImageSpider

  Example user agent
  BaiduImagespider ( http://help.baidu.jp/system/05.html)

• Gigabot

  Example user agent
  Gigabot/2.0/gigablast.com/spider.html

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-c6112) UCBrowser8.2.1.144/69/444/UCWEB Mobile

• publiclibraryarchive Bot

  Example user agent
  Mozilla/5.0 (compatible; publiclibraryarchive.org/1.0; +crawl@publiclibraryarchive.org)

• httpclient

  Example user agent
  HTTPClient/1.0 (2.4.0, ruby 1.9.3 (2013-06-27))

• CyberPatrol

  Example user agent
  CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; LG-P993 Build FRG83G) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  SonyEricssonW666i/R2BA Java/SEMC-Java/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0

• Google Bot Mobile

  Example user agent
  DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Y!J-AGENT

  Example user agent
  Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; KC38)

• Android Download Manager

  Example user agent
  AndroidDownloadManager/4.1.1 (Linux; U; Android 4.1.1; MID0714 Build/JRO03H)

• Sosospider

  Example user agent
  Sosospider+(+http://help.soso.com/webspider.htm)

• crawler4j

  Example user agent
  crawler4j (http://code.google.com/p/crawler4j/)

• Fake Browser

  Example user agent
  Mozilla/5.0 ()

• Google Adwords DisplayAds WebRender

  Example user agent
  Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36

• ltx71 Bot

  Example user agent
  ltx71 - (http://ltx71.com/)

• Google Bot

  Example user agent
  GoogleEarth/7.1.0003.1255(Android;Android (HTC Sensation XE with Beats Audio Z715e-pyramid-user-4.0.3);de-DE;kml:2.2;client:Free;type:default)

• Nutch

  Example user agent
  CB/Nutch-1.7

• Microsoft Bing Mobile SocialStreams Bot

  Example user agent
  Microsoft Bing Mobile SocialStreams Bot

• AdsBot Google-Mobile

  Example user agent
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-s7350i) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile UNTRUSTED/1.0

• Google Bot

  Example user agent
  GoogleEarth/7.1.0002.2011(Android;Android (LG-P875-l1e-user-4.1.2);de-DE;kml:2.2;client:Free;type:default)

• Sogou Spider

  Example user agent
  New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98)

• Jigsaw_W3C_CSS_Validator

  Example user agent
  Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See )

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-b5722) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile

• Generic Crawler

  Example user agent
  User-Agent: igmHttp/0.1.501 (Android OS; HTCDesireHDA9191 ; 2.3.5; de_DE; 800x480) gzip

• Werbefreie Deutsche Suchmaschine

  Example user agent
  Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html)

• woriobot

  Example user agent
  Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)

• MSN

  Example user agent
  MSNBot/Nutch-1.5.1

• Google Bot

  Example user agent
  Googlebot v2.1 (+http://www.google.com/bot.html) (http://www.openwebspider.org/)

• Amiga

  Example user agent
  Mozilla/6.0 (Macintosh; U; Amiga-AWeb) Safari 3.1

• Google Feedfetcher

  Example user agent
  Mozilla/5.0 (compatible) Feedfetcher-Google;(+http://www.google.com/feedfetcher.html)

• QuickiWiki Bot

  Example user agent
  QuickiWiki/1.0 (http://www.quickiwiki.com; admin@quickiwiki.com) BasedOnNodeJs/0.10.23

• General Crawlers

  Example user agent
  CTC/HUAWEIC5720 Java/HWJa/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0

• Fake Firefox

  Example user agent
  Mozilla/5.0 (Windows NT 7.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0

• Larbin

  Example user agent
  binlar_2.6.3 (test@mgmt.mic)

• Google App Engine

  Example user agent
  AppEngine-Google; (+http://code.google.com/appengine; appid: xbmacx)

• JoobleBot

  Example user agent
  JoobleBot (compatible; http://jooble.org/jooble-bot)

• MSR-ISRCCrawler

  Example user agent
  MSR-ISRCCrawler

• Nutch

  Example user agent
  SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)

• Googlebot-News

  Example user agent
  Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))

• Twisted PageGetter

  Example user agent
  Twisted PageGetter

• Shim Crawler

  Example user agent
  Shim-Crawler

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; UCBrowser8.2.1.144/70/444/UCWEB Mobile SonyEricssonG700/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1

• YandexMetrika

  Example user agent
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)

• MetaJobBot

  Example user agent
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; samsung-sgh-t528g) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Domnutch Bot

  Example user agent
  Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)

• Fake Browser

  Example user agent
  Http://Atamg.wup.ru/SAMSUNG-GT-S5233T/S5233TXEJF1 SHP/VPP/R5 Jasmine/0.8 Qtv5.3 SMM-MMS/1.2.0 profile/MIDP-2.1 configuration/CLDC-1.1

• Fake Browser

  Example user agent
  (null) [FBAN/FBIOS;FBAV/15.0.0.16.28;FBBV/4463064;FBDV/iPhone6,1;FBMD/iPhone;FBSN/iPhone OS;FBSV/8.0.2;FBSS/2; FBCR/U.S.Cellular;FBID/phone;FBLC/en_US;FBOP/5]

• nuSearch

  Example user agent
  nuSearch Spider www.nusearch.com (compatible; MSIE 4.01; Windows NT)

• Validator.nu/LV

  Example user agent
  Validator.nu/LV http://validator.w3.org/services

• Ichiro Bot

  Example user agent
  ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)

• Amazon CloudFront

  Example user agent
  Amazon CloudFront

• Texis Webscript

  Example user agent
  Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.0)

• General Crawlers

  Example user agent
  HuaweiC6100 Java/HWJa/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UNTRUSTED/1.0

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.0; 940N)

• WebAlta Crawler

  Example user agent
  WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)

• Yahoo-Test

  Example user agent
  Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.2; zh-cn; Lenovo A820 Build JZO54K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• IstellaBot

  Example user agent
  Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)

• wsr-agent

  Example user agent
  wsr-agent/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-a817) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0

• Seznam Screenshot Generator

  Example user agent
  Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)

• memoryBot

  Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.2.1; zh-cn; Lenovo S920 Build JOP40D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Accoona Bot

  Example user agent
  Accoona-Biz-Agent/1.1.1 crawler@accoona.com

• HooWWWer

  Example user agent
  HooWWWer/2.1.0 ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-infohiit.fi)

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; selevision; Selevision EMC1000i; 1.0; 1.0;) CE-HTML/1.0

• Heritrix

  Example user agent
  flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)

• ADmantX Platform Semantic Analyzer

  Example user agent
  ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.com

• ICC-Crawler

  Example user agent
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html; zurukko1552995316;)

• Fake Browser

  Example user agent
  Mozilla 4.0 (compatible:MSIE 6.0; Windows CE; IEMobile 7.6) Sprint MP6900SP

• Wayback Archive Bot

  Example user agent
  Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org)

• lwp

  Example user agent
  LWP::Simple/5.800

• cURL

  Example user agent
  curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; 810T)

• TestCrawler

  Example user agent
  robots

• Google Bot Mobile

  Example user agent
  SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Nutch

  Example user agent
  Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)

• WebMasterAid

  Example user agent
  Mozilla/5.0 (compatible; WebMasterAid/1.0; +http://wmaid.com/explore/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5550) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• Nutch

  Example user agent
  5acd734a-1ed3-4a17-ad9d-0b09ae7bc2d2/Nutch-1.7

• 200PleaseBot

  Example user agent
  Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b7320) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile, SAMSUNG-GT-B7320/1.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/4.0 UNTRUSTED/1.0

• Google Bot

  Example user agent
  GoogleEarth/7.1.0002.2011(Android;Android (HTC Sensation XL with Beats Audio X315e-runnymede-user-4.0.3);de-DE;kml:2.2;client:Free;type:default)

• lbot

  Example user agent
  Mozilla/5.0 (lbot/2.6.4) (lbot@mailme.not)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-u800) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• Webbot.ru

  Example user agent
  Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)

• R6 FeedFetcher

  Example user agent
  R6_FeedFetcher_(www.radian6.com/crawler)

• Orangebot

  Example user agent
  Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)

• FAST MetaWeb Crawler

  Example user agent
  FAST MetaWeb Crawler (helpdesk at fastsearch dot com)

• LinkStats Bot

  Example user agent
  LinkStats Bot

• Northern Light Web Search

  Example user agent
  Mozilla/5.0 (compatible; NLCrawler/2.0.15; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; lg-gs505) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB Mobile

• TinEye Bot

  Example user agent
  TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)

• Rogerbot

  Example user agent
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)

• ayna-crawler

  Example user agent
  Mozilla/5.0 (compatible; ayna-crawler http://www.ayna.com)

• CareerBot

  Example user agent
  Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)

• TweetmemeBot

  Example user agent
  Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0

• CazoodleBot

  Example user agent
  CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com)

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; inverto; IDL-6750N Volksbox II; 1.0; 1.0;) hdplusinteraktiv/1.0 (NETRANGEMMH;) CE-HTML/1.0

• Bot.AraTurka.com

  Example user agent
  Bot.AraTurka.com/0.0.1

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; htc touch diamond p3700) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; xx; samsung-gt-e2202) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; Android 4.4.2; D2212 Build/18.4.B.1.20) AppleWebKit/537.36 (KHTML like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36 Mobile UCBrowser/3.4.1.483

• Fake Browser

  Example user agent
  Mozzila/5.0 (Linux; U; Android 4.1.2; ru-ru; Fly IQ449 Build/JZ054K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

• Yahoo! Slurp

  Example user agent
  Mozilla/3.0 (Slurp.so/1.0; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

• Photon

  Example user agent
  Photon/1.0

• YandexImages

  Example user agent
  Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

• 12345

  Example user agent
  12345

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; htc s740) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/405/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-gu230) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• Riddler

  Example user agent
  Riddler (http://riddler.io/about.html)

• ShopWiki

  Example user agent
  ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

• Google Keyword Suggestion

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s3653w) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0 UNTRUSTED/1.0

• FindLinks

  Example user agent
  findlinks/2.6 ( http://wortschatz.uni-leipzig.de/findlinks/)

• Daumoa

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0

• Ilse

  Example user agent
  INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; selevision; Selevision EMC1000i; 1.0; 1.0;) hdplusinteraktiv/1.0 (NETRANGEMMH;) CE-HTML/1.0

• Windows-Update-Agent

  Example user agent
  Windows-Update-Agent/7.9.9600.17729 Client-Protocol/1.21

• SeznamBot

  Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)

• Y!J-AGENT

  Example user agent
  Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248)

• Java Standard Library

  Example user agent
  Java/1.17

• Android Download Manager

  Example user agent
  AndroidDownloadManager

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.7; zh-cn; ZTE U880E Build GWK74) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; rv:33.0; ; Trident/5.0)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; aplix jblend) UCBrowser8.4.0.159/69/352/UCWEB Mobile, SAMSUNG-SGH-I617/1.0 Mozilla/4.0 UNTRUSTED/1.0

• Java Standard Library

  Example user agent
  Java/1.7

• FeedFetcher

  Example user agent
  livedoor FeedFetcher/0.01 (http://reader.livedoor.com/; 999 subscribers)

• Digg Bot

  Example user agent
  Mozilla/5.0 (Digg/1.0; support@digg.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-a797) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• TestCrawler

  Example user agent
  Yeti-FeedItemCrawler/1.0 (NHN Corp.;+http://help.naver.com/robots/)

• Google HTTP Client Library for Java

  Example user agent
  Google-HTTP-Java-Client/1.17.0-rc (gzip)

• Thumbor

  Example user agent
  Thumbor/3.14.6

• masidani_bot

  Example user agent
  masidani_bot_v0.3 (info@masidani.com)

• Internet Archive

  Example user agent
  ia_archiver/8.7 (Windows NT 5.0; )

• IBM Connections

  Example user agent
  IC OpenGraph Crawler 4.5 (proprietary)

• MSRBOT

  Example user agent
  MSRBOT (http://research.microsoft.com/research/sv/msrbot)

• Microsoft CryptoAPI

  Example user agent
  Microsoft-CryptoAPI/6.1

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-m8910b) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile

• python-requests

  Example user agent
  python-requests/0.14 CPython/2.6 Linux/2.6-43-server

• Daumoa

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0

• WEB.DE MailCheck

  Example user agent
  WEB.DE MailCheck/2.4.0

• cURL

  Example user agent
  curl/7.14.1 (i386-portbld-freebsd4.7) libcurl/7.14.1 OpenSSL/0.9.8 zlib/1.1.3

• Rogerbot

  Example user agent
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)

• Liferea

  Example user agent
  Liferea/1.10.6 (Linux; en_US.UTF8; http://liferea.sf.net/) AppleWebKit (KHTML, like Gecko)

• Adbeat Bot

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86; de-de) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4

• Slackbot-Link-Expanding

  Example user agent
  Slackbot-LinkExpanding (+https://api.slack.com/robots)

• Mail.Ru

  Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)

• BacklinkCrawler

  Example user agent
  BacklinkCrawler (http://www.backlinktest.com/crawler.html)

• AOLbot

  Example user agent
  Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; ZTE U930 Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• YodaoBot

  Example user agent
  Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; Lenovo A780 Build GRJ90) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Google Bot

  Example user agent
  GoogleEarth/7.1.0002.2011(Android;Android (HTC Desire HD A9191-ace-user-2.3.5);de-DE;kml:2.2;client:Free;type:default)

• Fake Browser

  Example user agent
  Mozilla 5.0 (compatible; MSIE 9.0; Win Phone os 7.5; Trident/50; IE Mobile/9.0; HTC;MWP6985

• FacebookExternalHit

  Example user agent
  facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

• yoozBot

  Example user agent
  Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)

• nbot

  Example user agent
  Mozilla/5.0 (compatible; nbot/2.0)

• BabalooSpider

  Example user agent
  BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)

• Fake Browser

  Example user agent
  MozilMozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.4; InfoPath.2; SV1; .NET CLR 3.3.69573; WOW64; en-US)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-t505) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; AMOI N820 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; lg-gu220) UCBrowser8.2.0.132/69/352/UCWEB Mobile UNTRUSTED/1.0

• 007AC9 Crawler

  Example user agent
  Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)

• Coccoc bot

  Example user agent
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)

• Sogou Web Spider

  Example user agent
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

• FreeWebMonitoring SiteChecker

  Example user agent
  FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)

• Save n Keep Screenshot Bot

  Example user agent
  SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com )

• ZumBot

  Example user agent
  ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)

• Amiga

  Example user agent
  Mozilla/5.0 (compatible; Origyn Web Browser; AmigaOS 4; PPC; U) AppleWebKit/532.0+ (KHTML, like Gecko, Safari/532.0+)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.1; zh-cn; HUAWEI G510-0010 Build HuaweiG510-0010) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• oBot

  Example user agent
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters)

• Nutch

  Example user agent
  NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)

• Comodo Spider

  Example user agent
  Comodo Spider 1.2

• OpenWebSpider

  Example user agent
  OpenWebSpider/0.6 (http://www.openwebspider.org)

• SimplePie

  Example user agent
  SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20130514092120

• HuaweiSymantecSpider

  Example user agent
  HuaweiSymantecSpider/1.0 DSE-support@huaweisymantec.com (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)

• Lipperhey SEO Service

  Example user agent
  Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)

• NewsGator FetchLinks

  Example user agent
  NewsGator FetchLinks extension/0.2.0 (http://graemef.com)

• Fake IE

  Example user agent
  Mozilla/5.0 LG-GM750 Browser/IE8.12 MMS/LGMMSv1.0/1.2 Java/LGVM/1.1 Profile/MIDP-2.1 Configuration/CLDC-1.1 (compatible; MSIE 4.01; Windows CE; PPC)/UC Browser7.8.0.95

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; SV1; SE 2.x)

• WordPress

  Example user agent
  WordPress/3.7.1; http://samsung.hdblog.it

• Kraken

  Example user agent
  Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)

• NaverBot

  Example user agent
  nApps (LG-F160S; 16; naversearchapp; 5.0.1)

• W3C I18n Checker

  Example user agent
  W3C_I18n-Checker/1.0 (http://validator.w3.org/services)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Symbian/3; U; E7-00; de-DE; NokiaE7-00) AppleWebKit/534.3 (KHTML, like Gecko) Shazam Mobile Safari/534.3

• SeeBot

  Example user agent
  seebot/2.0 (+http://www.seegnify.com/bot)

• GrapeshotCrawler

  Example user agent
  Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

• BingBot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-lg900g) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  SonyEricssonJ108i/R7EA Profile/MIDP-2.1 Configuration/CLDC-1.1 UNTRUSTED/1.0 UCWEB/2.0 (Java; U; MIDP-2.0; xx; SonyEricssonJ108i) U2/1.0.0 UCBrowser/9.4.0.342 U2/1.0.0 Mobile

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-s5780) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0 UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; smart; CX10; 1.0; 1.0;) CE-HTML/1.0

• Fake Safari

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 1084) AppleWebKit/536.29.13 (KHTML like Gecko) Version/6.0.4 Safari/536.29.13

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-sgh-a597) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0

• WI Job Roboter

  Example user agent
  Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; SCH-i509 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Add Catalog

  Example user agent
  Mozilla/5.0 (compatible; Add Catalog/2.1;)

• URLfilterDB Crawler

  Example user agent
  Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; HUAWEI C8825D Build HuaweiC8825D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• isc header collector handlers

  Example user agent
  isc header collector handlers@isc.sans.edu

• NaverBot

  Example user agent
  NaverCafe/3.1.4 (Android OS 4.1.2; PANTECH IM-A850S; NaverCafe; 3.1.4)

• 80Legs

  Example user agent
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; j2me) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile, SAMSUNG-SGH-I607/I607FG1 Mozilla/4.0/UNTRUSTED/1.0

• Y!J-AGENT

  Example user agent
  Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.run4dom.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.1; zh-cn; Newpad-K97 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Europe Internet Portal

  Example user agent
  EuripBot/0.5 (+http://www.eurip.com) PreCheck

• Mail.Ru

  Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)

• Yahoo Mobile App

  Example user agent
  YahooMobile/1.0 (Yahoo! Search; 3.1.3) (Apple; iPhone; iPhone OS/7.1.2); YHOO_Search_App/3.1.3

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0 SonyEricssonG900/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; SCH-i919 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• oBot

  Example user agent
  Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)

• AddThis.com robot

  Example user agent
  AddThis.com robot tech.support@clearspring.com

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-m5650) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB Mobile UNTRUSTED/1.0

• AdSense Bot

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)

• Scrapy

  Example user agent
  Scrapy/0.24.6 (+http://scrapy.org)

• ht://Dig

  Example user agent
  htdig/3.1.6 (romieu@bastide-medical.fr)

• Superfeedr Bot

  Example user agent
  Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-te365 teleca) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.1-update1; En-us; SH-03C Build/S2140) AppleWebKit/530.17 (KHTML Like Gecko) Version/4.0 Mobile Safari/530.17

• Jupdate

  Example user agent
  jupdate

• Exalead

  Example user agent
  Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)

• CazoodleBot

  Example user agent
  CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)

• ConveraMultiMediaCrawler

  Example user agent
  ConveraMultiMediaCrawler/0.1 ( http://www.authoritativeweb.com/crawl)

• Aboundexbot

  Example user agent
  Aboundex/0.2 (http://www.aboundex.com/crawler/)

• Crazywebcrawler

  Example user agent
  CRAZYWEBCRAWLER 0.9.0, http://www.crazywebcrawler.com

• Cliqzbot

  Example user agent
  Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)

• MagpieRSS

  Example user agent
  MagpieRSS/0.72 (+http://magpierss.sf.net)

• MagpieRSS

  Example user agent
  MagpieRSS/0.61 (+http://magpierss.sf.net)

• CCBot

  Example user agent
  CCBot/1.0 ( http://www.commoncrawl.org/bot.html)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s5380k) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; ZTE U930HD Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.3; zh-cn; C8650 Build HuaweiC8650) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Google Bot

  Example user agent
  Googlebot-Mobile (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.1; en-us; HUAWEI C8813 Build HuaweiC8813) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.2.2; zh-cn; ZTE V818 Build JDQ39) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; SCH-i929 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake IE

  Example user agent
  MOT-A1200i/R532C2_24XPS_1.1P Mozilla/4.0 (compatible; MSIE 6.0; Linux; Motorola A1200i;nnn) Profile/MIDP-2.0 Configuration/CLDC-1.1 Opera 8.00 [en]

• Jakarta Commons HttpClient

  Example user agent
  Jakarta Commons-HttpClient/3.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5250) UCBrowser8.2.0.132/69/352/UCWEB Mobile UNTRUSTED/1.0

• Prlog

  Example user agent
  Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)

• Amiga

  Example user agent
  Mozilla/5.0 (compatible; IBrowse 3.0; AmigaOS4.0)

• Xerka

  Example user agent
  Xerka WebBot v1.0.0 [AIDO_CREA_ab]

• Python-urllib

  Example user agent
  Python-urllib/1.16

• NetNewsWire

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.74.9 (KHTML, like Gecko) NetNewsWire/3.3.2

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung gt-c3520) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile

• Lightspeed Systems Crawler

  Example user agent
  Mozilla/4.0 (compatible; MSIE 7.0;Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727;.NET CLR 3.0.04506.30) Lightspeedsystems

• Yahoo-Test

  Example user agent
  Mozilla/5.0 (Yahoo-Test/4.0 ysm-keystone-crawl-support@yahoo-inc.com)

• YandexDirect-Dyatel

  Example user agent
  Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)

• python-requests

  Example user agent
  python-requests/1.2.3 CPython/2.7.3 Linux/3.5.0-23-generic

• SEOkicks Robot

  Example user agent
  Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)

• Larbin

  Example user agent
  LarbinWebCrawler (internet@bredband.net)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-gx200) UCBrowser8.2.0.132/69/352/UCWEB Mobile UNTRUSTED/1.0

• visionutils

  Example user agent
  visionutils/0.2

• Xenus Link Sleuth

  Example user agent
  Xenu Link Sleuth/1.3.8

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; htc touch2 t3333) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile UNTRUSTED/1.0

• Internet Archive

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.4; de-de; LT18i Build 4.0.2.A.0.69) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• CMS Crawler

  Example user agent
  Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)

• Twingly Recon

  Example user agent
  Mozilla/5.0 (compatible; Twingly Recon; twingly.com)

• Scrapy

  Example user agent
  Scrapy/0.22.2 (+http://scrapy.org)

• Heritrix

  Example user agent
  mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)

• Nutch

  Example user agent
  Mozilla/5.0 (compatible; summers;)/Nutch-1.7

• Internet Archive

  Example user agent
  ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)

• Microsoft-WebDAV

  Example user agent
  DavClnt

• General Crawlers

  Example user agent
  Samsung/910XXIG2/sw_platform=S60;sw_platform_version=5.0;java_build_version=1.3.4 Profile/MIDP-2.1 Configuration/CLDC-1.1 UNTRUSTED/1.0

• Mozilla Crawler

  Example user agent
  Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc)

• restify

  Example user agent
  restify/2.6.3 (x64-darwin; v8/3.14.5.9; OpenSSL/1.0.1i) node/0.10.32

• A6-Indexer

  Example user agent
  A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)

• NaverBot

  Example user agent
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.7; zh-cn; ZTE-T U880 Build GWK74) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Facebook

  Example user agent
  facebookplatform/1.0 (+http://developers.facebook.com)

• msnbot-media

  Example user agent
  msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)

• MerchantCentricBot

  Example user agent
  Mozilla/5.0 (compatible; MerchantCentricBot/1.0)

• HggH Screenshot System with PhantomJS

  Example user agent
  HggH PhantomJS Screenshoter

• ThumbnailAgent

  Example user agent
  ThumbnailAgent

• MJ12bot

  Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;; en-us; Huawei-U8651S Build/U8651SV100R001USAC85B843) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-b7300) UCBrowser8.2.0.132/69/352/UCWEB Mobile, SAMSUNG-GT-B7300/1.0 UNTRUSTED/1.0

• Netvibes

  Example user agent
  Netvibes (http://www.netvibes.com)

• netEstate NE Crawler

  Example user agent
  netEstate NE Crawler (+http://www.website-datenbank.de/)

• WhatWeb Web Scanner

  Example user agent
  WhatWeb/0.4.8-dev

• Sistrix Crawler

  Example user agent
  Sistrix

• 360Spider

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider

• OffByOne

  Example user agent
  Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4

• Spiderlytics

  Example user agent
  Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com)

• IIS Site Analysis Web Crawler

  Example user agent
  iisbot/1.0 (+http://www.iis.net/iisbot.html)

• ThumbSniper

  Example user agent
  ThumbSniper (http://thumbsniper.com)

• Zollard Worm

  Example user agent
  Mozilla/5.0 (compatible; Zollard; Linux)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5380b) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0

• Riddler

  Example user agent
  Riddler (http://riddler.io/about)

• Nutch

  Example user agent
  asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.3.1; fr-fr; AN10BG2DT Build/GINGERBREAD) AppleWebKit/533.1 (KHTML like Gecko) Version/4.0 Mobile Safari/533.1

• General Crawlers

  Example user agent
  Mozilla/4.0 LG-GM750h Browser/IE8.12 MMS/LGMMSv1.0/1.2 Java/LGVM/1.1 Profile/MIDP-2.1 Configuration/CLDC-1.1 (compatible; MSIE 6.0; Windows CE; IEMobile 8.12; MSIEMobile 6.0)

• LinksCrawler

  Example user agent
  LinksCrawler 0.1beta

• Nutch

  Example user agent
  emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)

• The Old Reader

  Example user agent
  Mozilla/5.0 (compatible; theoldreader.com; 1 subscribers; feed-id=51413de5bd9279ea5002d75b)

• TweetedTimes Bot

  Example user agent
  Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)

• General Crawlers

  Example user agent
  LG-GW620 Mozilla/5.0 (Linux; U; Android 1.5) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 Java/Jbed/7.0 Profile/MIDP-2.

• Jayde Crawler

  Example user agent
  Jayde Crawler. http://www.jayde.com

• YaCy Bot

  Example user agent
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html

• Ilse

  Example user agent
  INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.3.5; tr-tr; HTCWildfireSA510e Build/GRJ90) AppleWebKit/533.1 (KHTML like Gecko) Version/4.0 Mobile Safari/533.1

• W3C-mobileOK/DDC

  Example user agent
  W3C-mobileOK/DDC-1.0 (see http://www.w3.org/2006/07/mobileok-ddc)

• BaiDu

  Example user agent
  Baiduspider+(+http://www.baidu.com/search/spider_jp.html)

• Sogou Pic Spider

  Example user agent
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; lg-gb270) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozzila/5.0(Linux; U; Android 4.2.2; ru-ru; Fly IQ447 Build/JZ054K) AppleWebKit/534.30(KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

• WeSEE:Search

  Example user agent
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)

• General Crawlers

  Example user agent
  Apache-HttpClient/4.0-beta2 (java 1.5)

• W3C-checklink

  Example user agent
  W3C-checklink/4.0 [4.4] libwww-perl/5.803

• Socialradarbot

  Example user agent
  Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com)

• Feedfetcher-Google

  Example user agent
  FeedFetcher-Google; ( http://www.google.com/feedfetcher.html)

• GIDBot

  Example user agent
  GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php)

• SMRF URL Expander

  Example user agent
  SMRF URL Expander

• Theme Spider

  Example user agent
  Theme Spider ( http://www.themespider.com/spider.html)

• NewsBlur Favicon Fetcher

  Example user agent
  NewsBlur Feed Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0 SonyEricssonP1i/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-t669mkb) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Google-Sitemaps

  Example user agent
  Google-Sitemaps/1.0

• SiteExplorer

  Example user agent
  Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://SiteExplorer.Info/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-s3100) UCBrowser8.3.0.154/69/444/UCWEB Mobile

• Zao-Crawler

  Example user agent
  Zao-Crawler

• Crawlboy

  Example user agent
  CrawlerBoy Pinpoint.com

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dla-marbach.de)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; htc star trek) UCBrowser8.3.0.154/69/352/UCWEB Mobile

• Google App Engine

  Example user agent
  AppEngine-Google

• libwww

  Example user agent
  libwww-perl/6.05

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; LNV-Lenovo A790e Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; SA3C)

• memoryBot

  Example user agent
  Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)

• ContextAd Bot

  Example user agent
  ContextAd Bot 1.0

• redbot

  Example user agent
  RED/1 (http://redbot.org/)

• BingPreview

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux U; Android 4.0; xx-xx; LG-P714 Build/IML74K) AppleWebKit/534.30 (KHTML like Gecko) Version/4.0 Mobile Safari/534.30 UNTRUSTED/1.0

• DomainSigmaCrawler

  Example user agent
  Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-s3370b) --XIOZ-->8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• W3C_Validator

  Example user agent
  W3C_Validator/1.767

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; N02A)

• Sogou-Test-Spider

  Example user agent
  Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)

• SiteExplorer

  Example user agent
  Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)

• Google Bot

  Example user agent
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ASProxy/5.5b5

• Norbert the Spider

  Example user agent
  Norbert the Spider(Burf.com)

• GarlikCrawler

  Example user agent
  GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)

• libwww

  Example user agent
  NCSA Mosaic/2.6b1 [en-gb] (X11;UNIX_System_V 4.2 R4000) libwww/2.12 modified

• ScollSpider

  Example user agent
  ScollSpider/2.0 ( http://www.webwobot.com/ScollSpider.php)

• DomainScan Server Monitoring

  Example user agent
  Mozilla/5.0 (compatible; DomainSCAN/2.0; +http://domainscan.info/robot/)

• NetLyzer FastProbe

  Example user agent
  NetLyzer FastProbe

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.4; zh-cn; ZTE-C N760 Build GRJ22) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.1-update1; zh-cn; ZTE-C N600 Build ERE27) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• DiscoverEd

  Example user agent
  DiscoverEd/Nutch-1.7 (OER search crawler; http://wiki.creativecommons.org/DiscoverEd; webmaster@creativecommons.org)

• WebThumb

  Example user agent
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0

• Rippers

  Example user agent
  Mozilla/3.0 (compatible; Indy Library)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-s3850) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• AISearchBot

  Example user agent
  AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.)

• Safe-t.net Web Filtering Service

  Example user agent
  FANGCrawl/0.01

• PiplBot

  Example user agent
  Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-c5510) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0

• Superarama.com - BOT

  Example user agent
  Superarama.com - BOT/v.0.1

• itsscan

  Example user agent
  Mozilla/5.0 (compatible; Muenster University of Applied Sciences; +http://fb02itsscan.fh-muenster.de)

• reddit pic scraper

  Example user agent
  Mozilla / reddit pic scraper v0.8 (bklug@tyx.net)

• LinkedInBot

  Example user agent
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)

• Sophora Linkchecker

  Example user agent
  Mozilla/5.0 (compatible; Sophora Linkchecker; http://www.subshell.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; lg-gt350) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-gt-s5350) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/405/UCWEB Mobile

• Screenshot Bot

  Example user agent
  pr-cy.ru Screenshot Bot

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; ZTE U960s3 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• DBot

  Example user agent
  Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws)

• Peeplo Screenshot Bot

  Example user agent
  Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )

• Fake Browser

  Example user agent
  label=ypmobapp;anv=YPmobile4.0.1;os=Android 4.0.4;dev=totemc2;devm=HTC One VX;man=HTC;brw=Android;shwd=540x960;shwa=540x923;cs=yes;is=yes;as=yes;vs=yes;jss=yes;ls=DAM;

• URL Grabber

  Example user agent
  urlgrabber/3.10 yum/3.4.3

• VSE Link Tester

  Example user agent
  VSE/1.0 (vsecrawler@hotmail.com)

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/BD6)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; GN100T Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• pub-crawler

  Example user agent
  Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)

• Fake Browser

  Example user agent
  Mozilla/5.0 compatible; MSIE 9.0; Windows NT 4.1 Chrome/31.0.1229.79

• uMBot

  Example user agent
  Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)

• Safari RSS

  Example user agent
  AppleSyndication/56.1

• Nutch

  Example user agent
  NutchCVS/0.06-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)

• Y!J-AGENT

  Example user agent
  Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )

• YisouSpider

  Example user agent
  YisouSpider

• Scopia Crawler

  Example user agent
  Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; 800P31C Build/JDQ39) AppleWebKit/534.30 (KHTML like Gecko) Version/4.0 Safari/534.30

• EasouSpider

  Example user agent
  Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; MEIZU MX Build GRJ90) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• ZE.bz

  Example user agent
  ZeBot_www.ze.bz (ze.bz@hotmail.com)

• Google Bot

  Example user agent
  Googlebot/2.2

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; samsung-gt-s5380g) UCBrowser8.4.0.159/70/352/UCWEB Mobile UNTRUSTED/1.0

• Web Glance

  Example user agent
  Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1

• WebZIP

  Example user agent
  WebZIP/7.0 (http://www.spidersoft.com)

• CheckSite Verification Agent

  Example user agent
  CheckSite Verification Agent (+http://www.checksite.us)

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.fi/eng/project.html)

• Generic Research Crawler

  Example user agent
  Computer_and_Automation_Research_Institute_Crawler (crawler@info.ilab.sztaki.hu)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; samsung-gt-c3300k-orange) UCBrowser8.4.0.159/69/352/UCWEB Mobile, SAMSUNG-GT-C3300K-ORANGE/C3300KBVJG3 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0

• Google Bot Mobile

  Example user agent
  DoCoMo//2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Fake Browser

  Example user agent
  %%%%Mozilla/5.0 (X11; U; Linux x86_64; en-AU) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.114 Safari/537.36 Puffin/4.4.1IT%%%%

• Swiss Search Engine

  Example user agent
  search.ch/2.8 Android/4.0.3; htc_europe:HTC One V; Mozilla/5.0 (Linux; U; Android-universal)

• ICC-Crawler

  Example user agent
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)

• BaiDu

  Example user agent
  BaiDuSpider

• VSE Link Tester

  Example user agent
  VSE/1.0 (testcrawler@hotmail.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; En-in; Samsung-gt-s7070) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• MJ12bot

  Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)

• Fake Browser

  Example user agent
  \x22Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)\x22

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.3; zh-cn; HTC A510e Build GRI40) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Sogou Pic Spider

  Example user agent
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

• Amiga

  Example user agent
  OWB/3.12 (AmigaOS 4.1; ppc; U)

• BLEXBot

  Example user agent
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)

• WordPress

  Example user agent
  WordPress/3.8.1; http://www.samsung-blog-news.de

• Fake Chrome

  Example user agent
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/35.0.1916.114 Safari/537.36

• ActiveWorlds Bot

  Example user agent
  agent6/Nutch-1.1

• proximic

  Example user agent
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)

• TLSProber

  Example user agent
  TLSProber/0.8

• Nutch

  Example user agent
  your sipder name/Nutch-1.7

• PHP

  Example user agent
  PHP

• Genieo Web Filter

  Example user agent
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html

• ECCP

  Example user agent
  ECCP/1.0 (search@eniro.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; Lenovo A390e Build FRG83G) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Adaxas Spider

  Example user agent
  Adaxas Spider (http://www.adaxas.net/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-gt360 teleca) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; F02A)

• Fake Browser

  Example user agent
  Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; UCBrowser8.2.0.132/69/452/UCWEB Mobile UNTRUSTED/1.0 SonyEricssonW960i/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1

• RankFlex

  Example user agent
  RankFlex.com Webspider

• Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/1.10

• TestCrawler

  Example user agent
  BoardReader/1.0 (http://boardreader.com/info/robots.htm)-Machine1

• Crawlboy

  Example user agent
  CrawlWave/1.2 (crawlwave[at]circular.gr http://www.spiderwave.aueb.gr/

• wget

  Example user agent
  Wget/1.10.2

• BaiDu

  Example user agent
  Baiduspider+(+http://www.baidu.jp/spider/)

• Fake Android

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.2;es-es;GT-I5500 Build/ FROYO) AppleWebKit/533.1 (KHTML, like Geccko) Version/4.0 Mobile Safari/533.1

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-u900) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB Mobile

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung m2710) UCBrowser8.2.1.144/69/352/UCWEB Mobile

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-c3300k) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile, SAMSUNG-GT-C3300K/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0

• Android Download Manager

  Example user agent
  AndroidDownloadManager/4.1.1 (Linux; U; Android 4.1.1; MB886 Build/9.8.0Q-97_MB886_FFW-20)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung c5212) UCBrowser8.3.0.154/69/352/UCWEB Mobile

• Digincore Bot

  Example user agent
  Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.

• Searchteq Bot

  Example user agent
  stq_bot (+http://www.searchteq.de)

• SimplePie

  Example user agent
  SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111118194710

• WordPress

  Example user agent
  WordPress/1.2.1 PHP/4.3.9-1

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; HUAWEI C8860E Build HuaweiC8860E) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• magpie-crawler

  Example user agent
  magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)

• ZumBot

  Example user agent
  Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)

• Fake Browser

  Example user agent
  label=ypmobapp;anv=YPmobile4.3.1;os=Android 4.4.2;dev=hlteatt;devm=SAMSUNG-SM-N900A;man=samsung;brw=Android;shwd=1080x1920;shwa=1080x1845;cs=yes;is=yes;as=yes;vs=yes;jss=yes;ls=DAM;

• YandexNews

  Example user agent
  Mozilla/5.0 (compatible; YandexNews/3.0; +http://yandex.com/bots)

• Y!J-AGENT

  Example user agent
  Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-i637) UCBrowser8.3.0.154/69/444/UCWEB Mobile, SAMSUNG-SGH-I637/1.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/4.0 UNTRUSTED/1.0

• Baidu Image Search

  Example user agent
  Baiduspider-image ( http://www.baidu.com/search/spider.htm)

• Lipperhey SEO Service

  Example user agent
  Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)

• vBulletin SEO Bot

  Example user agent
  Mozilla/4.0 (vBSEO; http://www.vbseo.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; kau; samsung-c6625) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0, UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; samsung-sgh-a687) UCBrowser8.4.0.159/69/444/UCWEB Mobile UNTRUSTED/1.0

• BaiDu

  Example user agent
  Baiduspider ( http://www.baidu.com/search/spider.htm)

• Mnogosearch

  Example user agent
  MnoGoSearch/3.3.15

• BuSecurityProject

  Example user agent
  Mozilla,BuSecurityProject

• Apercite

  Example user agent
  Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Android 4.2.2;) AppleWebKit/1.1 Version/4.0 Mobile Safari/1.1

• RSS2HTML

  Example user agent
  FeedForAll rss2html.php v2

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-c300) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• OpenWebSpider

  Example user agent
  OpenWebSpider (link collector; http://links.port30.se/cia.html) v (http://www.openwebspider.org/)

• Fake Chrome

  Example user agent
  24.0.1284.0.0 (Windows NT 6.1) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/24.0.1284.0.4.742.7 Safari/534.7

• Sogou Develop Spider

  Example user agent
  Sogou develop spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• R6 FeedFetcher

  Example user agent
  R6_FeedFetcher(www.radian6.com/crawler)

• Simpy

  Example user agent
  Simpy/1.1 (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)

• 80Legs

  Example user agent
  Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620

• memoryBot

  Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html)

• NetSurf

  Example user agent
  NetSurf/3.0 (RISC OS)

• Baidu Mobile Search

  Example user agent
  Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html)

• W3C_Validator

  Example user agent
  W3C_Validator/1.3 http://validator.w3.org/services

• SeznamBot

  Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)

• Fake Chrome

  Example user agent
  Chrome LINUX 45.0.2454.85 (4c2743615eaa2806ad014c59bf6acbb652cf3aa8)-devel

• DomainsDB

  Example user agent
  DomainsDB.net MetaCrawler v.0.9.7b (http://domainsdb.net/)

• BLEXBot

  Example user agent
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)

• RMSnapKit

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) RMSnapKit/1.0

• Netcraft Webserver Survey

  Example user agent
  Mozilla/4.0 (compatible; Netcraft Web Server Survey)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s3370l) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• SimplePie

  Example user agent
  SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20121030175911

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; HTC T328t Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Generic Crawler

  Example user agent
  User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.20 Safari/537.36 OPR/15.0.1147.18 (Edition Next)

• PagesInventory Bot

  Example user agent
  PagesInventory (robot http://www.pagesinventory.com)

• Google Bot

  Example user agent
  Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; 823P)

• python-requests

  Example user agent
  python-requests/2.0.0 CPython/2.6.6 Linux/2.6.32-431.el6.x86_64

• Daumoa

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; En-us; Alcatel-ot-806d) UCBrowser8.3.1.161/70/465/UCWEB Mobile UNTRUSTED/1.0

• Twitterbot

  Example user agent
  Twitterbot/1.0

• Crawlboy

  Example user agent
  Crawler 0.1,gzip(gfe),gzip(gfe)

• AboutUs Bot

  Example user agent
  AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)

• Wada.vn Search Bot

  Example user agent
  Mozilla/5.0 (compatible; Wada.vn Vietnamese Image Search)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-a897) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• libwww

  Example user agent
  libwww-perl/5.835

• Phantom.js bot

  Example user agent
  Phantom.js bot

• Embedly

  Example user agent
  Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)

• Fake IE

  Example user agent
  'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)'

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 7.1; Trident/5.0)

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://www.wise-guys.nl/)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.1-[ShunYea-V4.0-??????????????????]; zh-cn; C8500 Build ERE27) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-lg500g) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• TurnitinBot

  Example user agent
  TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)

• Blogshares Spiders

  Example user agent
  Blogshares Spiders (Renewed V1.65)

• Generic Crawler

  Example user agent
  User-Agent: Mozilla/1.0N (X11; SunOS 4.1.1 sun4c)

• MOSBookmarks Link Checker

  Example user agent
  Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker)

• Alexabot

  Example user agent
  Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)

• TestCrawler

  Example user agent
  MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.html

• Paper.li Bot

  Example user agent
  Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-gm210) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/70/352/UCWEB Mobile

• Fake Chrome

  Example user agent
  Mozilla/5.0 (Windows NT 7.1) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30

• EventMachine HttpClient

  Example user agent
  EventMachine HttpClient

• msnbot

  Example user agent
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)._

• Livelap Crawler

  Example user agent
  LivelapBot/0.2 (http://site.livelap.com/crawler)

• Google PageSpeed Insights

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s3572) UCBrowser8.3.0.154/69/352/UCWEB Mobile, SAMSUNG-GT-S3572/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 Untrusted/1.0

• Android Download Manager

  Example user agent
  AndroidDownloadManager/4.3 (Linux; U; Android 4.3; ME302C Build/JSS15Q)

• Fake Browser

  Example user agent
  (){ :; }; :(){ :|:& };:

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.1; zh-cn; SCH-i909 Build FROYO) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• PagePeeker Screenshot Maker

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) domain.com Safari/538.1

• Feedfetcher-Google

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 375 subscribers; feed-id=15381863289700640853)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; htc_s620) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0

• Tiny Tiny RSS

  Example user agent
  Tiny Tiny RSS/1.11.4c63934 (http://tt-rss.org/)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.4; zh-cn; Lenovo P70 Build GRJ22) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  SEC-SGHE900/10 SAMSUNG JAVA CLIENT

• nuSearch

  Example user agent
  NuSearch Spider www.nusearch.com

• pmoz.info ODP link checker

  Example user agent
  Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)

• Fake IE

  Example user agent
  Mozilla/4.0(compatible;MSIE 7.0;Windows NT 6.1;SV1;.NET CLR 1.0.3705;.NET CLR 3.0.30618)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-e2222l) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile

• Nutch

  Example user agent
  Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible; MSIE 8.0; b2w 0.1)

• Fake Safari

  Example user agent
  Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25

• Arachnida Web Crawler

  Example user agent
  Arachnida Web Crawler

• gvfs

  Example user agent
  gvfs/1.24.1

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-gs390) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  ['Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36 GD']

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; Coolpad 5820 Build FRG83G) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Java Standard Library

  Example user agent
  Java/1.5.0_17

• Cliqzbot

  Example user agent
  Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)

• IPv4Scan

  Example user agent
  IPv4Scan (+http://ipv4scan.com)

• Steeler

  Example user agent
  Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; F906i)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; XT928 Build 6.5.1_GC-122-DNRCG-15) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  ProjectWF-java-test-crawler

• TestCrawler

  Example user agent
  Mozilla/5.0 (compatible; TestCrawler)

• python-requests

  Example user agent
  python-requests/2.2.1 CPython/2.7.3 Linux/3.2.0-56-generic

• BingBot

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0; )"

• InternetSeer.com

  Example user agent
  InternetSeer.com

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-p525) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• Scrapy

  Example user agent
  Scrapy/0.20.1 (+http://scrapy.org)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; 001SC)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; ZTE U790 Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Windows-RSS-Platform

  Example user agent
  Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 6.3)

• kulturarw3

  Example user agent
  Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)

• Google Bot

  Example user agent
  Mozilla/5.0 AppleWebKit/537.4 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.4

• BUbiNG Bot

  Example user agent
  BUbiNG (+http://law.di.unimi.it/BUbiNG.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; XT760 Build 02.3C.08RPS) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Google Bot

  Example user agent
  Googlebot/Nutch-1.7

• Zeus

  Example user agent
  Zeusbot/0.8.1 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; htc touch 3g t3232) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0

• Holmes

  Example user agent
  holmes/2.3

• Lipperhey SEO Service

  Example user agent
  Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)

• Yahoo

  Example user agent
  YahooCacheSystem

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-a767) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Windows; U; Windows NT 5.1; lt; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.14 (.NET CLR 3.5.30729)

• PagesInventory Bot

  Example user agent
  PagesInventory (robot +http://www.pagesinventory.com)

• XSpider

  Example user agent
  XSpider 0.01;http://uncool.oicp.net/spider.html

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.4; zh-cn; GT-S5838 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.11; Windows NT 6.1; Trident/5.0)

• Google HTTP Client Library for Java

  Example user agent
  Google-HTTP-Java-Client/1.13.1-beta (gzip)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; UCBrowser8.4.0.159/70/405/UCWEB Mobile SonyEricssonW950i/R100 Profile/MIDP-2.0 Configuration/CLDC-1.1

• Hatena Crawler

  Example user agent
  Hatena::Crawler/0.01

• Orangebot

  Example user agent
  Mozilla/5.0 (compatible; Orangebot/2.0; support.orangebot@orange.com)

• OpenHoseBot

  Example user agent
  Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)

• MixrankBot

  Example user agent
  Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-t310) UCBrowser8.3.0.154/69/405/UCWEB Mobile UNTRUSTED/1.0

• Fake Firefox

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.1.2) Firefox/3.5.2

• meanpathbot

  Example user agent
  Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

• uMBot

  Example user agent
  Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)

• mbot

  Example user agent
  mbot v.1.16

• SMTBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• Lycos

  Example user agent
  Lycos_Spider_(modspider)

• KomodiaBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0

• SMTBot

  Example user agent
  Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• 360Spider

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

• Crawlboy

  Example user agent
  crawl/0.4 langcrawl/0.1

• Adbeat Bot

  Example user agent
  adbeat_bot

• TweetmemeBot

  Example user agent
  Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)

• SimplePie

  Example user agent
  SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111015034325

• Google

  Example user agent
  Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

• Amiga

  Example user agent
  IBrowse/2.4oem (AmigaOS V53; PPC; 68K build)

• CC Metadata Scaper

  Example user agent
  CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper

• Netvibes

  Example user agent
  Netvibes (http://www.netvibes.com/; 8 subscribers; feedID: 2244192)

• Nutch

  Example user agent
  My Nutch Spider/Nutch-1.6

• SafeSearch microdata crawler

  Example user agent
  SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)

• FeedBlitz

  Example user agent
  FeedBlitz/1.0 (1 subscribers)

• Yahoo! Slurp China

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

• SiteCon

  Example user agent
  Mozilla/5.0 (Linux; U; en-gb) AppleWebKit/418.9.1 (KHTML, like Gecko) SiteCon/8.10.22

• NikiBot

  Example user agent
  niki-bot

• Google Bot

  Example user agent
  Google-YouTubeSample/1.0 Google-HTTP-Java-Client/1.7.0-beta (gzip) Google-HTTP-Java-Client/1.7.0-beta (gzip)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-c395) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile

• Lucidworks Bot

  Example user agent
  Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com)

• Begun Advertising Bot

  Example user agent
  Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)

• Picsearch Bot

  Example user agent
  psbot/0.1 (+http://www.picsearch.com/bot.html)

• BingBot

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.1; +http://www.bing.com/bingbot.htm)

• Java Standard Library

  Example user agent
  Java/1.6.0_43

• LinkedInBot

  Example user agent
  LIAuthLibrary:0.0.2 com.linkedin.android:3.4.2 TCL_ALCATEL ONE TOUCH 4037T:android_4.4.2

• Amiga

  Example user agent
  Mozilla/5.0 (compatible; Origyn Web Browser; U; AmigaOS 4.1; xx) AppleWebKit/534.0 (KHTML, like Gecko, Safari/534.0) OWB/3.32

• Linkdex Bot

  Example user agent
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

• Xaldon WebSpider

  Example user agent
  Xaldon WebSpider 2.7.b6

• BingPreview

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/538.14 (KHTML, like Gecko) BingPreview/1.0b

• General Crawlers

  Example user agent
  HotJava/1.1.2 FCS

• Google Bot Mobile

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Mail.Ru

  Example user agent
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

• Research Projects

  Example user agent
  IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)

• XML Sitemaps Generator

  Example user agent
  Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/534.6 HbbTV/1.1.1 (+DL+PVR; inverto; IDL-6651N Volksbox Web Edition; 1.0; 1.0;) hdplusinteraktiv/1.0 (NETRANGEMMH;) CE-HTML/1.0

• BrowseX

  Example user agent
  Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows)

• Picsearch Bot

  Example user agent
  psbot-image (+http://www.picsearch.com/bot.html)

• Download Accelerator

  Example user agent
  masscan/1.0 (https://github.com/robertdavidgraham/masscan)

• phantomas

  Example user agent
  phantomas/1.8.0 (PhantomJS/1.9.8; linux x64)

• IRLbot

  Example user agent
  IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler)

• i1searchbot

  Example user agent
  i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-ks360 teleca) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/444/UCWEB Mobile UNTRUSTED/1.0

• MojeekBot

  Example user agent
  Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; samsung-gt-s5780d) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Ichiro Mobile Bot

  Example user agent
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)

• Fake Firefox

  Example user agent
  Mozilla/5.0 Gecko/2009042316 Firefox/2.0.0.x

• Nagios

  Example user agent
  check_http/v2.0 (nagios-plugins 2.0)

• YRL_ODP_CRAWLER

  Example user agent
  YRL_ODP_CRAWLER

• FAST Enterprise Crawler

  Example user agent
  FAST Enterprise Crawler/6 (www.fastsearch.com)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 107) AppleWebKit/534.48.3 (KHTML like Gecko) Version/5.1 Safari/534.48.3

• Generic Research Crawler

  Example user agent
  LinkStar Research Crawler (http://linkstar.com/),gzip(gfe),gzip(gfe)

• cURL

  Example user agent
  curl/7.29.0

• Fake Browser

  Example user agent
  Mozilla/5.0 (;;;) AppleWebKit/538.8 HbbTV/1.1.1 (+DL+PVR; Inverto; Volksbox; 1.0; 1.0;) CE-HTML/1.0 hdplusinteraktiv/1.0 (NETRANGEMMH)

• Entireweb

  Example user agent
  Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.1; zh-cn; SCH-N719 Build JRO03C) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Moozilla

  Example user agent
  Moozilla

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; lg-s310) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• wget

  Example user agent
  Wget/1.12 (linux-gnu)

• PHP

  Example user agent
  PHP/5.3.77

• Nutch

  Example user agent
  nutch crawler/Nutch-2.2

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b6520) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile, SAMSUNG-GT-B6520/1.0 Mozilla/4.0 UNTRUSTED/1.0

• magpie-crawler

  Example user agent
  magpie-crawler/1.1 (U; Linux amd64; en-GB; http://www.brandwatch.net)

• WWW-Mechanize

  Example user agent
  WWW-Mechanize/1.64

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; GT-I9050 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; 8810 Build FRG83G) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 4.1.1; en-us; Luna TAB07-920N Build/JRO03C) AppleWebKit/534.30 (KHTML like Gecko) Version/4.0 Mobile Safari/534.30

• Nutch

  Example user agent
  Nutch12/Nutch-1.2

• picmole Bot

  Example user agent
  Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)

• lwp

  Example user agent
  LWP LDMusicNews::LDNewsAPI

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung gt-b3410) UCBrowser8.3.0.154/69/444/UCWEB Mobile

• Fake Browser

  Example user agent
  Mozilla (Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html))

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b7320l) UCBrowser8.2.1.144/69/405/UCWEB Mobile, SAMSUNG-GT-B7320L/1.0 Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/4.0 UNTRUSTED/1.0

• AdsBot Google-Mobile

  Example user agent
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; GT-I9300 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Feedfetcher-Google

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=000000000000000000)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-i6712) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/352/UCWEB Mobile UNTRUSTED/1.0

• cURL

  Example user agent
  curl/7.35.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-a230) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; ME811 Build SHDWR_X6_2.200.33) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Generic Crawler

  Example user agent
  User-Agent: Mozilla/1.1N (X11; I; SunOS 4.1.2 sun4c)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.2; zh-cn; Coolpad W706+ Build FRG83G) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• ArchiveBot

  Example user agent
  ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-id; samsung gt-m3310) UCBrowser8.2.1.144/69/444/UCWEB Mobile

• lwp

  Example user agent
  lwp-trivial/1.41

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; lg-c330) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• Holmes

  Example user agent
  holmes/3.12.4 (http://morfeo.centrum.cz/bot)

• Googlebot-Image

  Example user agent
  Googlebot-Image/1.0

• WASALive Bot

  Example user agent
  Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)

• Rippers

  Example user agent
  Mozilla/3.01 (compatible;)

• YandexFavicons

  Example user agent
  Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)

• iZSearch Bot

  Example user agent
  iZSearch.com

• adidxbot

  Example user agent
  adidxbot/2.0 (+http://search.msn.com/msnbot.htm)

• msnbot-media

  Example user agent
  msnbot-media/1.0 ( http://search.msn.com/msnbot.htm)

• Pingdom

  Example user agent
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)

• TweetmemeBot

  Example user agent
  Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)

• Baidu Mobile Search

  Example user agent
  Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spi_der.html)

• BaiDu

  Example user agent
  Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)

• YaCy Bot

  Example user agent
  yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.html

• Internet Archive

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 4.1.2; zh-CN; HTC 9066 Build/JRO03C) AppleWebKit/534.30 (KHTML like Gecko) Version/4.0 Mobile Safari/534.30

• SkypeUriPreview

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5

• Google Bot

  Example user agent
  mozilla/5.0 (iphone; cpu iphone os 8_3 like mac os x) applewebkit/600.1.4 (khtml, like gecko) version/8.0 mobile/12f70 safari/600.1.4 (compatible; googlebot/2.1; +http://www.google.com/bot.html)

• InAGist URL Resolver

  Example user agent
  InAGist URL Resolver (http://inagist.com)

• chlooe Bot

  Example user agent
  bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)

• krakenjs

  Example user agent
  kraken/0.6.0

• DocWeb Link Crawler

  Example user agent
  DocWeb Link Crawler (http://doc.php.net)

• AboutUs Bot Johnny5

  Example user agent
  Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; samsung-gt-c5130) UCBrowser8.3.0.154/69/405/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-a667) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  UNTRUSTED/1.0 Mozilla/5.0 (Java; U; es-la; samsung gt-s3770) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile

• zitebot

  Example user agent
  Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)

• Experibot

  Example user agent
  Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)

• Tiny Tiny RSS

  Example user agent
  Tiny Tiny RSS/1.10 (http://tt-rss.org/)

• SemanticBot

  Example user agent
  Mozilla/5.0 (compatible; SemanticBot/1.1)

• AdvBot

  Example user agent
  Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)

• Fake Chrome

  Example user agent
  Mozilla/5.0 (Chrome 32) WebKit/22.0 Firefox/22.0

• 360Spider

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider

• Netchart Adv Crawler

  Example user agent
  Netchart Adv Crawler 1.0

• NaverBot

  Example user agent
  NANaverNDrive/3.0.0(Android OS 4.1.2;PANTECH IM-A860S)

• Amiga

  Example user agent
  AmigaVoyager/3.2 (AmigaOS/MC680x0)

• CrawlRobot

  Example user agent
  Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php)

• Windows Phone Search

  Example user agent
  Windows Phone Search (Windows Phone OS 8.0;NOKIA;Lumia 1520;8.0;10521)

• DueDil Crawler

  Example user agent
  Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)

• MetaGeneratorCrawler

  Example user agent
  MetaGeneratorCrawler/1.3.8 (www.metagenerator.info)

• wget

  Example user agent
  Wget/1.14 (linux-gnu)

• Werbefreie Deutsche Suchmaschine

  Example user agent
  Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)

• Yahoo-MMCrawler

  Example user agent
  Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; ru; samsung-gt-s5510) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UN

• ImplisenseBot

  Example user agent
  ImplisenseBot 1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.9; zh-cn; HTC EVO 4G++ Build HTC EVO 4G++) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Linkdex Bot

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/)

• NaverBot

  Example user agent
  NaverCafe/3.1.6 (Android OS 4.0.4; samsung SHW-M380S; NaverCafe; 3.1.6)

• Qualidator.com Bot

  Example user agent
  Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)

• Reverse IP Lookup

  Example user agent
  Mozilla/5.0+(compatible; RevIP.info site analyzer v4.00; http://poweredby.revip.info)

• NaverBot

  Example user agent
  NANaverNDrive/3.0.0(Android OS 4.2.2;samsung SHV-E300S)

• Fake Chrome

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/35.0.1916.47 Safari/537.36

• Bloglovin Bot

  Example user agent
  Bloglovin/1.0 (http://www.bloglovin.com; 0 subscribers)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-sgh-d608) UCBrowser8.3.0.154/70/352/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; xx; sonyericssonw150i) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0

• HTTrack

  Example user agent
  Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

• Jeode

  Example user agent
  Jeode/1.x.x

• Pompos

  Example user agent
  Pompos/1.2 http://pompos.iliad.fr

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-lg420g[tf012125009345399008080015075895332] obigo) UCBrowser8.3.0.154/70/352/UCWEB Mobile

• FAST-WebCrawler

  Example user agent
  FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.3; zh-cn; M030 Build IML74K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Google PageSpeed Insights

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1

• Exabot

  Example user agent
  Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/4.0; FDM; MSIECrawler; Media Center PC 5.0)

• Convera

  Example user agent
  ConveraCrawler/0.9c ( http://www.authoritativeweb.com/crawl)

• SemrushBot

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)

• Xenus Link Sleuth

  Example user agent
  Xenu's Link Sleuth 1.0p

• Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2,55

• WBSearchBot

  Example user agent
  Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; N905i)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; samsung-sgh-a927) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/69/352/UCWEB Mobile UNTRUSTED/1.0

• Google Toolbar

  Example user agent
  Mozilla/4.0 (compatible; GoogleToolbar 7.5.5111.1712; Windows XP 5.1; MSIE 8.0.6001.18702)

• Fake Safari

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.3.1; en-gb; AN7G2DTE Build/GINGERBREAD) AppleWebKit/533.1 (KHTML like Gecko) Version/4.0 Mobile Safari/533.1

• Google Bot

  Example user agent
  googlebot

• RedCarpet

  Example user agent
  RedCarpet/2.1 CFNetwork/672.0.2 Darwin/14.0.0

• YandexBot

  Example user agent
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: twitter-mirror),gzip(gfe),gzip(gfe),gzip(gfe)

• Evernote Clip Resolver

  Example user agent
  Evernote Clip Resolver

• Google Bot Mobile

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• MetaHeadersBot

  Example user agent
  MetaHeadersBot (+http://www.metaheaders.com/bot.html)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.1.2; zh-cn; GT-I9228 Build JZO54K) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• DomainsBot

  Example user agent
  domainsbot (+http://www.domainsbot.com)

• Amiga

  Example user agent
  Mozilla/5.0 (compatible; Origyn Web Browser; U; AmigaOS 4.0; xx) AppleWebKit/534.0 (KHTML, like Gecko, Safari/534.0) OWB/3.32

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)

• SophosUpdateManager

  Example user agent
  SophosUpdateManager/1.5.7.50 SDDS/2.0 (u= EO2ANA123G c= 6342da15-f351-4ab7-9656-3f5f2d50885d )

• XING Contenttabreceiver

  Example user agent
  XING-contenttabreceiver/2.0

• Google Bot

  Example user agent
  Googlebot/2.1(+http://www.googlebot.com/bot.html)

• PWBot

  Example user agent
  Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw)

• Elmer

  Example user agent
  Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)

• libcurl

  Example user agent
  libcurl-agent/1.0

• Tutorial Crawler

  Example user agent
  Tutorial Crawler 1.4

• Fake IE

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.1.1.0 http://code.google.com/p/abot)

• bnf.fr_bot

  Example user agent
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)

• Yahoo Slingstone

  Example user agent
  Yahoo:LinkExpander:Slingstone

• Google Image Proxy

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com)

• Updated!

  Example user agent
  updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; zh-cn; SCH-I939 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• bot

  Example user agent
  bot

• Gomez Site Monitor

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1

• Fake Browser

  Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Netscape/8.0.4

• Sogou Web Spider

  Example user agent
  Sogou web spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; SCH-I619 Build GINGERBREAD) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Fake IE

  Example user agent
  Mozilla/4.0 (compatible- MSIE 6.0- Windows NT 5.1- SV1- .NET CLR 1.1.4322

• fr-crawler

  Example user agent
  Mozilla/5.0 (compatible; fr-crawler/1.1)

• PECL HTTP

  Example user agent
  PECL::HTTP/1.7.4 (PHP/5.4.4)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; Lenovo A366t Build GRK39F) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• FeedBlitz

  Example user agent
  FeedBlitz/1.0

• Google-Site-Verification

  Example user agent
  Mozilla/5.0 (compatible; Google-Site-Verification/1.0)

• RssBar

  Example user agent
  RssBar/1.29 (RssBar for unDonut 1.35)

• 80Legs

  Example user agent
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620

• Charlotte

  Example user agent
  Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@beta.spider.com)

• WordPress

  Example user agent
  WordPress/3.2.1; http://www.painlog.jp

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Id; samsung-gt-m3710) UCBrowser8.2.1.144/69/444/UCWEB Mobile UNTRUSTED/1.0

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-in; samsung-gt-b2710d) UCBrowser8.4.0.159/69/352/UCWEB Mobile UNTRUSTED/1.0

• Google Bot

  Example user agent
  GoogleProducer; (+http://goo.gl/7y4SX)

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com)

• Smartsite HTTPClient

  Example user agent
  Smartsite HTTPClient - Mozilla/4.0 (compatible; MSIE 6.0)

• Fake Browser

  Example user agent
  Mozilla Nokia 6300-USER AGENT

• JUST-CRAWLER

  Example user agent
  JUST-CRAWLER( http://www.justsystems.com/jp/tech/crawler/)

• Liferea

  Example user agent
  Liferea/1.6.4 (Linux; en_US.UTF-8; http://liferea.sf.net/)

• Sogou Push Spider

  Example user agent
  Sogou Push Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; vivo E1 Build GRJ90) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• CheckLinks

  Example user agent
  Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks)

• GG PeekBot

  Example user agent
  GG PeekBot 2.0 ( http://gg.pl/ http://info.gadu-gadu.pl/praca )

• Blekkobot

  Example user agent
  Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)

• CCBot

  Example user agent
  CCBot/2.0 (http://commoncrawl.org/faq/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (LG-GD880-Orange/V09d AppleWebKit/531 MMS/LG-MMS-V1.0/1.2 MediaPlayer/LGPlayer/1.0 Java/ASVM/1.1 Profile/MIDP-2.1 Configuration/CLDC-1.1)

• MSN

  Example user agent
  msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)

• General Crawlers

  Example user agent
  Mozilla/4.0 LG-GM750Q Browser/IE8.12 MMS/LGMMSv1.0/1.2 Java/LGVM/1.1 Profile/MIDP-2.1 Configuration/CLDC-1.1 (compatible; MSIE 6.0; Windows CE; IEMobile 8.12; MSIEMobile 6.0)

• Mail.Ru

  Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

• Proxy Gear Pro

  Example user agent
  Proxy Gear Pro/2.1 (Windows; N; Windows NT 6.1; en)

• LinkedInBot

  Example user agent
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 http://www.linkedin.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; ME863 Build SLNCU-60.1.102) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-b7722) UCBrowser8.2.1.144/69/405/UCWEB Mobile UNTRUSTED/1.0

• WinkBot

  Example user agent
  WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; es-la; samsung-sgh-a847) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.6.0.199/84/352/UCWEB Mobile UNTRUSTED/1.0

• YahooSeeker-Mobile

  Example user agent
  LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)

• Butterfly

  Example user agent
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-a130) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0

• FacebookExternalHit

  Example user agent
  facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php)

• Facebookscraper

  Example user agent
  facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php)

• FlipboardProxy

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)

• CrawlConvera

  Example user agent
  CrawlConvera0.1 (CrawlConvera@yahoo.com)

• FeedDemon

  Example user agent
  FeedDemon/4.5 (http://www.feeddemon.com/; Microsoft Windows)

• Feedfetcher-Google

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 19 subscribers; feed-id=13965549748850348809)

• Crawlboy

  Example user agent
  Crawl/0.1 langcrawl/0.1

• PycURL

  Example user agent
  PycURL/7.22.0

• DAWINCI ANTIPLAG SPIDER

  Example user agent
  Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; en-us; lg-lg840g) UCBrowser8.3.0.154/69/444/UCWEB Mobile UNTRUSTED/1.0

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; 706P)

• BingBot

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

• YandexBlogs

  Example user agent
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)

• Wotbox

  Example user agent
  Wotbox/2.01 ( http://www.wotbox.com/bot/)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.7; zh-cn; ZTE-T U960s Build GWK74) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• msnbot

  Example user agent
  msnbot/1.1 (+http://search.msn.com/msnbot.htm)

• Facebot

  Example user agent
  Facebot/1.0

• Aboundexbot

  Example user agent
  Aboundex/0.3 (http://www.aboundex.com/crawler/)

• Fake IE

  Example user agent
  Mozilla/5.0 CK={tiPpw0h4g7kIf1T8snFdwpPeIcraLTUImXjwPsV9F27k+ityupIdD4AzdYENl6wfnuyvIKM9PR/5QIO4s8bR9TktqUNKcfYcpCrtOqEXuc/SNsbkxUc3cw==} (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-kd876) UCBrowser8.2.1.144/69/352/UCWEB Mobile UNTRUSTED/1.0

• NaverBot

  Example user agent
  Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s5750e) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile UNTRUSTED/1.0

• BingPreview

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML; like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 BingPreview/1.0b

• AdSense Bot

  Example user agent
  Mediapartners-Google

• Internet Archive

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.1-x http://pandora.nla.gov.au/crawl.html)

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; F903i)

• Browsershots

  Example user agent
  Browsershots

• sSearch Crawler

  Example user agent
  ssearch_bot (sSearch Crawler; http://www.semantissimo.de)

• wget

  Example user agent
  Wget/1.6

• jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; CA3C)

• Fake Android

  Example user agent
  Mozilla/4.0 (compatible;Android;320x480)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; samsung-gt-s7230b) AppleWebKit/530.13 (KHTML, like Gecko) UCBrowser/8.5.0.185/82/405/UCWEB Mobile UNTRUSTED/1.0

• BaiDu

  Example user agent
  Baiduspider-testbranch

• 360Spider

  Example user agent
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider

• WinHttp

  Example user agent
  WinHTTP

• Google Bot

  Example user agent
  GOOG

• metager2-verification-bot

  Example user agent
  Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; En-us; Samsung-sgh-a877) AppleWebKit/530.13 (KHTML, Like Gecko) UCBrowser/8.5.0.185/83/352/UCWEB Mobile

• WebIndex

  Example user agent
  WebIndex

• Envolk

  Example user agent
  envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)

• NetzCheckBot

  Example user agent
  NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.5; zh-cn; XT319 Build V4.50A) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Applebot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

• General Crawlers

  Example user agent
  Mozilla/5.0 (Java; U; Pt-br; lg-gm750) UCBrowser8.3.0.154/69/352/UCWEB Mobile UNTRUSTED/1.0

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.2.1; zh-cn; SCH-W899 Build FROYO) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• Heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://webarhive.nlc.gov.cn)

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 4.0.4; en-us; SGH-T999 Build IMM76D) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31

• General Crawlers

  Example user agent
  JNLP/1.7.0 javaws/10.17.2.02 () Java/1.7.0_17

• WeSEE:Ads

  Example user agent
  WeSEE:Ads/PageBot (http://www.wesee.com/bot/)

• GLBot

  Example user agent
  Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co)

• Fake Browser

  Example user agent
  Mozilla 5.0 (Linux; U; Android 2.3.6; zh-cn; Motorola MOT-XT681 Build IRMXCT_9_02.19.05RPS) UC AppleWebKit 534.31 (KHTML, like Gecko) Mobile Safari 534.31