Detected as bot - WhichBrowser v2.0.18

• Bing

  Example user agent
  (Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.4.5.3000 Firefox/38

• Example user agent
  Screaming Frog SEO Spider/2.22

• Aboundex

  Example user agent
  Aboundex/0.2 (http://www.aboundex.com/crawler/)

• Example user agent
  Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

• Example user agent
  GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php)

• Example user agent
  RyzeCrawler/1.1.0 (+http://www.ryze.nl/crawler/)

• Perl

  Example user agent
  lwp-trivial/1.41

• Example user agent
  Crawllybot/0.1/0.1 (Crawllybot/0.1; http://www.crawlly.com; crawler@crawlly.com)

• Example user agent
  Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)

• Example user agent
  Logict IPv6 Crawler/1.0 (http://ipv6search.logict.net),gzip(gfe),gzip(gfe)

• Example user agent
  Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)

• Bing

  Example user agent
  msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)

• CiteSeerX

  Example user agent
  mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)

• Example user agent
  Tasapspider/0.9

• Example user agent
  Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)

• Example user agent
  Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)

• Perl

  Example user agent
  libwww-perl/5.64

• Example user agent
  BaiDuSpider

• Example user agent
  Mozilla/5.0 (compatible; LikaholixCrawler/1.0; +http://mylikes.com/about/crawler)

• Example user agent
  Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)

• Blogshares

  Example user agent
  Blogshares Spiders (Renewed V1.65)

• A6 Indexer

  Example user agent
  A6-Indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)

• Example user agent
  camoca/camoca-n.1.2 (super-agent; search crawler; info at does not exist dot com)

• Example user agent
  omgilibot/0.3 +http://www.omgili.com/Crawler.html

• Example user agent
  Shim-Crawler

• Example user agent
  Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)

• W3C Link Checker

  Example user agent
  W3C-checklink/4.81 libwww-perl/5.836

• Example user agent
  449 Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp

• Example user agent
  PluckItCrawler/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com;)

• Baiduspider

  Example user agent
  'Mozilla/5.0 (compatible; Baiduspider/2.0; +ht'

• Example user agent
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup.com/crawler.html)

• NewsGator

  Example user agent
  NewsGator/3.0,gzip(gfe),gzip(gfe)

• Example user agent
  WebTarantula.com Crawler

• Example user agent
  Mozilla/5.0 (compatible; InfuzApp/1.0; +http://www.infuz.com/bot.html)

• Example user agent
  lworldspider

• Example user agent
  Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)

• Googlebot

  Example user agent
  Googlebot/Nutch-1.7

• Googlebot Mobile

  Example user agent
  Googlebot-Mobile (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• MnoGoSearch

  Example user agent
  MnoGoSearch/3.3.15

• Yahoo! Japan

  Example user agent
  Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

• Example user agent
  Sogou develop spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Example user agent
  Content Crawler

• Example user agent
  Screaming Frog SEO Spider/2,01

• Pinterest

  Example user agent
  Pinterest/0.1

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.1-x http://pandora.nla.gov.au/crawl.html)

• Bloglovin

  Example user agent
  Bloglovin/1.0 (http://www.bloglovin.com; 0 subscribers)

• Bing

  Example user agent
  msnbot/1.1 (+http://search.msn.com/msnbot.htm)

• Example user agent
  ScollSpider/2.0 ( http://www.webwobot.com/ScollSpider.php)

• iAsk Spider

  Example user agent
  iaskspider/2.0(+http://iask.com/help/help_index.html)

• Googlebot

  Example user agent
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ASProxy/5.5b5

• FeedZirra

  Example user agent
  feedzirra http://github.com/pauldix/feedzirra/tree/master

• AstraSpider

  Example user agent
  Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)

• Example user agent
  Microsoft Bing Mobile SocialStreams Bot

• iAsk Spider

  Example user agent
  Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)

• iAsk Spider

  Example user agent
  iaskspider2 (iask@staff.sina.com.cn)

• Friendica

  Example user agent
  Friendica

• Example user agent
  Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)

• Apache Nutch

  Example user agent
  Mozilla/5.0 (compatible; summers;)/Nutch-1.7

• Example user agent
  magpie-crawler/1.1 (U; Linux amd64; en-GB; http://www.brandwatch.net)

• Apache Nutch

  Example user agent
  nutch crawler/Nutch-2.2

• Example user agent
  CatchBot/1.0; http://www.catchbot.com

• Google Ads Bot

  Example user agent
  AdsBot-Google (+http://www.google.com/adsbot.html)

• Example user agent
  mixi-news-crawler/1.00 (http://mixi.jp/)

• Example user agent
  Mozilla/5.0 (compatible; WebCrunch/1.2; +http://webcrunch.net/crawler)

• Curl

  Example user agent
  curl/7.19.5 (i586-pc-mingw32msvc) libcurl/7.19.5 zlib/1.2.3

• htdig

  Example user agent
  htdig/3.1.5 (infosys@storm.rmi.org)

• Example user agent
  Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)

• Bing

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0 http://www.bing.com/bingbot.htm)

• Example user agent
  Gigabot/2.0/gigablast.com/spider.html

• Example user agent
  YOURLS v1.7.1 +http://yourls.org/ (running on http://bp7.org)

• Bing

  Example user agent
  msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)

• Googlebot

  Example user agent
  Googlebot/2.1 (http://www.googlebot.com/bot.html)

• Java

  Example user agent
  Java/1.6.0_43

• CommonCrawl

  Example user agent
  CCBot/1.0 ( http://www.commoncrawl.org/bot.html)

• HeartRails Capture

  Example user agent
  Mozilla/5.0 (X11; U; Linux i686; xx; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18

• Example user agent
  CMS crawler ( http://buytaert.net/crawler/)

• Baiduspider

  Example user agent
  BaiduImagespider ( http://help.baidu.jp/system/05.html)

• Curl

  Example user agent
  curl/7.16.0 (i386-unknown-openbsd4.1) libcurl/7.16.0 OpenSSL/0.9.7j zlib/1.2.3

• Baiduspider

  Example user agent
  Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search

• Example user agent
  Mozilla/5.0 (compatible; clumboot; +http://localhost/clumpit/bot.php)

• Example user agent
  VinjaVideoSpider/1.1

• Yandex Anti Virus

  Example user agent
  Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots)

• Bing Preview

  Example user agent
  Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Nokia; Lumia 520) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Mobile Safari/537.36 Edge/12.0 BingPreview/1.0b

• Wget

  Example user agent
  Wget/1.11.4 Red Hat modified

• Example user agent
  Screaming Frog SEO Spider/2,22

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.0-1182; http://social.feder8.ru

• Example user agent
  Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler)

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)

• Example user agent
  DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0; http://www.baidu.jp/spider/)

• FAST Web Crawler

  Example user agent
  Opera/30.0.1835.59 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/29.16 AppleWebKit/599.0+ (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 FAST-WebCrawler/3.8

• Example user agent
  cloudforecastbot

• Example user agent
  NewzCrawler/1.9 (compatible; MSIE 6.00; Newz Crawler 1.9; http://www.newzcrawler.com/ )

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.2-1188; https://friendika.openmindspace.org

• Example user agent
  rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)

• Example user agent
  Gregarius/0.5.2 (+http://devlog.gregarius.net/docs/ua)

• Example user agent
  OmniExplorer_Bot/6.70 ( http://www.omni-explorer.com) WorldIndexer

• Example user agent
  PercolateCrawler/4 (ops@percolate.com)

• Example user agent
  wminer/1.2 (North America Web Crawler; http://www.wminer.com/bot; info at wminer dot com)

• Example user agent
  Mozilla/5.0 (compatible; WangIDSpider/1.0; +http://www.wangid.com/spider.html)

• Pinterest

  Example user agent
  Mozilla/5.0 (Linux; Android 5.0.2; D6503 Build/23.1.A.0.690) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.0.0 Mobile Safari/537.36 [Pinterest/Android]

• Sogou Mobile Spider

  Example user agent
  Sogou Mobile Spider1.0 (http://wap.sogou.com)

• Ezooms Crawler

  Example user agent
  Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)

• Example user agent
  Mozilla/5.0 (compatible; GoGuidesBot; http://www.goguides.org/spider.html)

• Apache Jakarta

  Example user agent
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 3.5.21022; .NET CLR 3.5.30729; MS-RTC LM 8; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; .NET CLR 3.0.30729) Jakarta Commons-HttpClient/3.0-rc3 PHPCrawl GStreamer souphttpsrc libsoup/2.27.4 PycURL/7.19.0 XML-RPC for PHP 2.2.1 GoogleFriendConnect/1.0 HTMLParser/1.6 gPodder/0.15.2 ( http://gpodder.org/) anw webtool LoadControl/1.3 WinHttp urlgrabber/3.1.0

• Example user agent
  Diffbot/0.1

• Browsershots

  Example user agent
  Browsershots

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50215; baiduspider)

• Cliqzbot

  Example user agent
  Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)

• Example user agent
  Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler)

• Example user agent
  dotbot

• Example user agent
  DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)

• Example user agent
  NodejsSpider/1.0

• Example user agent
  GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)

• Soso Web Spider

  Example user agent
  Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)

• Example user agent
  ArchitextSpider

• Example user agent
  Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)

• Example user agent
  AdnormCrawler www.adnorm.com/crawler

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.2-1188; http://mozartweg.dyndns.org/friendica

• W3C Markup Validator

  Example user agent
  W3C_Validator/1.767

• 360spider

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

• Bing

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)

• Example user agent
  Sogou Orion spider/4.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Googlebot

  Example user agent
  Mozilla/5.0 (Windows; Mozilla 4.0(compatible; Googlebot/5.0; http://www.google.com/googlebot); en-US;)

• Example user agent
  EuripBot/0.5 (+http://www.eurip.com) PreCheck

• Example user agent
  Mozilla/4.0 (compatible); MSIE 5.0; Medialab Spider

• Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)

• Bloglines

  Example user agent
  Bloglines/3.0-rho (http://www.bloglines.com; 3 subscribers)

• Example user agent
  ZoomInfo::Beehive Crawler

• Apache Httpd

  Example user agent
  Apache-HttpClient/4.0-beta2 (java 1.5)

• Example user agent
  SMXCrawler/1.0 (www.socialmetrix.com)

• Example user agent
  PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)

• WordPress.com

  Example user agent
  WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 95.215.52.150

• Example user agent
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

• Example user agent
  Punk Spider/PunkSPIDER-v0.1

• Example user agent
  Axtaris Web Crawler/Axtaris-1.0.1 (Hello from Axtaris.org | Web-Crawler; http://axtaris.org; crawler at axtaris dot org)

• Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)

• Example user agent
  ProjectWF-java-test-crawler

• Example user agent
  GoGuidesBot/0.0.1 (GoGuides Indexing Spider; http://www.goguides.org/spider.html)

• Example user agent
  Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.2-1188; https://karl.marx.pm

• Example user agent
  wonderbot/JS 1.0

• Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler

• Example user agent
  LinkAider (http://linkaider.com/crawler/)

• Example user agent
  www.website-analyzer.net Website Analyzer Crawler

• Apache Nutch

  Example user agent
  Spider/Nutch-2.3-SNAPSHOT (Webcrawler)

• Example user agent
  radian6_default_(www.radian6.com/crawler)

• Perl

  Example user agent
  libwww-perl/5.814

• Example user agent
  GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)

• NewsGator FetchLinks

  Example user agent
  NewsGator FetchLinks extension/0.2.0 (http://graemef.com)

• Example user agent
  ldspider (BTC 2011 crawl, harth@kit.edu, http://code.google.com/p/ldspider/wiki/Robots)

• Apache Nutch

  Example user agent
  NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)

• Goo

  Example user agent
  DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo; http://help.goo.ne.jp/door/crawler.html)

• Example user agent
  Mozilla/5.0 (compatible; Linux i686; en-US; URLfilterDB-crawler/1.0) ufdb/1.0

• Example user agent
  OpenWebSpider/0.6 (http://www.openwebspider.org)

• Example user agent
  Mozilla/5.0 (compatible; Scrubby/3.2; +http://seotools.scrubtheweb.com/webpage-analyzer.html)

• Vagabondo

  Example user agent
  Mozilla/5.0 (compatible; Vagabondo/Wapspider; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• Pinterest

  Example user agent
  Pinterest/0.1 +http://pinterest.com/

• NewsGator

  Example user agent
  NewsGatorOnline/2.0 (http://www.newsgator.com)

• iAsk Spider

  Example user agent
  iaskspider/2.0( http://iask.com/help/help_index.html)

• Example user agent
  GSiteCrawler/v1.12 rev. 260 (http://gsitecrawler.com/)

• Example user agent
  Symfony Spider (http://symfony.com/spider)

• Google Feedfetcher

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 19 subscribers; feed-id=13965549748850348809)

• Example user agent
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)

• Example user agent
  MPICrawler/0.1 +http://kermit.news.cs.nyu.edu/crawler.html,gzip(gfe),gzip(gfe)

• Summify

  Example user agent
  Summify (Summify/1.0.1; +http://summify.com)

• Example user agent
  OmniExplorer_Bot/4.32 (+http://www.omni-explorer.com) WorldIndexer

• Example user agent
  hivaBot/hivaBot-0.2 (info@gorgor.ir)

• Example user agent
  kizasi-spider/1.0 (+http://kizasi.jp/)

• Example user agent
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)

• Example user agent
  MultiCrawler, http://sw.deri.org/2006/04/multicrawler/robots.html

• Example user agent
  Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)

• Example user agent
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

• Example user agent
  FlaxCrawler/1.0,gzip(gfe),gzip(gfe)

• Example user agent
  Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)

• BazQux

  Example user agent
  Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 2 subscribers)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows compatible LesnikBot)

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://vaiqui.com)

• Java

  Example user agent
  Java/1.5.0_17

• Picsearch bot

  Example user agent
  psbot-page (+http://www.picsearch.com/bot.html)

• Perl

  Example user agent
  lwp-trivial/1.35

• Example user agent
  RavenCrawler

• Example user agent
  Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php)

• Example user agent
  Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)

• Larbin

  Example user agent
  larbin_2.6.3 larbin2.6.3@unspecified.mail

• Example user agent
  EnaBot/1.2 (http://www.enaball.com/crawler.html)

• Example user agent
  Mozilla/5.0 (compatible; woriobot +http://worio.com)

• Example user agent
  Mozilla/4.0 (compatible; Yet-Another-Spider; )

• Example user agent
  GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)

• Example user agent
  FaviconArchiver/1.0 (+http://moblur.org/workshop/favicon_archive/)

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130207.001528 +http://webarchiv.cz/kontakty/)

• Example user agent
  YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com)

• Example user agent
  Mozilla/5.0 (compatible;WI Job Roboter Spider Version 3;+http://www.webintegration.at)

• Example user agent
  OmniExplorer_Bot/6.63b (+http://www.omni-explorer.com) WorldIndexer

• Example user agent
  Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.2-1188; https://myfriendica.net

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.run4dom.com)

• Example user agent
  nlcrawler/1.0 (+http://northernlight.com/)

• Example user agent
  Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )

• Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us; EasouSpider; +http://www.easou.com/search/spider.html)

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.3-dev-1191; https://friendica.xyz

• Example user agent
  Moreoverbot/5.1 ( http://w.moreover.com; webmaster@moreover.com) Mozilla/5.0

• Googlebot Mobile

  Example user agent
  SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Example user agent
  Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)

• Example user agent
  iisbot/1.0 (+http://www.iis.net/iisbot.html)

• Example user agent
  ExactSeek_Spider

• Livedoor

  Example user agent
  Mozilla/5.0 (compatible; livedoor-Mobile-Gateway/0.02; +http://p.m.livedoor.com/help.html)

• Example user agent
  Grub/2.0 (Grub.org crawler; http://www.grub.org/; bot@grub.org)

• Example user agent
  noobot Spider/Noobot-1.2 (noobot-bot; http://www.noobot.fr)

• Example user agent
  Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)

• Example user agent
  Facebot/1.0

• Example user agent
  Mozilla/5.0 (compatible; SputnikImageBot/2.2)

• Example user agent
  Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

• Example user agent
  Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)

• Livedoor

  Example user agent
  livedoor HttpClient

• Bing Preview

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/538.14 (KHTML, like Gecko) BingPreview/1.0b

• 360spider

  Example user agent
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider

• Example user agent
  Mozilla/5.0 (compatible; OpenCrawler/0.1.6.6; http://code.google.com/p/opencrawler/),gzip(gfe),gzip(gfe)

• Example user agent
  Jambot/0.1.1 (Jambot; http://www.jambot.com/blog; crawler@jambot.com)

• Example user agent
  Mozilla/5.0 (compatible; AbiLogicBot/1.0; +http://www.abilogic.com/bot.html)

• Example user agent
  ReadPath_Spider

• Goo RSS Reader

  Example user agent
  gooRSSreader3.7 - build20090410

• Example user agent
  YOURLS v1.5 +http://yourls.org/ (running on http://shelby.im)

• Example user agent
  Mozilla/5.0 (compatible; archive.bibalex.org_bot; +http://www.test.com)

• Perl

  Example user agent
  libwww-perl/5.76

• Perl

  Example user agent
  libwww-perl/5.805

• Example user agent
  WomlpeFactory/0.1 (+http://www.Womple.com/bot.html)

• AltaVista

  Example user agent
  AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net

• Example user agent
  semantics webbot/1.0

• Example user agent
  Mozilla/5.0 (compatible; NLCrawler/2.0.15; Linux 2.6.3-7; i686; en_US)KHTML/3.4.89 (like Gecko)

• Baiduspider

  Example user agent
  Mozilla/5.0 (compatible; Baiduspider-cpro; +http://www.baidu.com/search/spider.html)

• Gocrawl

  Example user agent
  Googlebot (gocrawl v0.4)

• Radian 6

  Example user agent
  R6_CommentReader(www.radian6.com/crawler)

• Example user agent
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

• Pinterest

  Example user agent
  Mozilla/5.0 (iPad; CPU OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Mobile/12F69 [Pinterest/iOS]

• Example user agent
  Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)

• Example user agent
  Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/)

• Google App Engine

  Example user agent
  Mozilla/5.0 (Android; Mobile; rv:29.0) Gecko/29.0 Firefox/29.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~oldecreator)

• Example user agent
  Mozilla/5.0 (compatible; Lipperhey Spider; http://www.lipperhey.com/)

• Example user agent
  YOURLS v1.5 +http://yourls.org/ (running on http://oerk.at)

• Baiduspider

  Example user agent
  Baiduspider+(+http://www.baidu.com/search/spider_jp.html)

• Sogou Web Spider

  Example user agent
  sogou spider

• Example user agent
  Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)

• Google Web Preview

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36

• Bing

  Example user agent
  msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

• Example user agent
  YOURLS v1.5.1 +http://yourls.org/ (running on http://u.i4x3.tk)

• Example user agent
  fastbot crawler beta 4.0 (+http://www.fastbot.de)

• Example user agent
  Jyxobot/1

• Google App Engine

  Example user agent
  AppEngine-Google; ( http://code.google.com/appengine)

• Example user agent
  Denodo IECrawler/4.5,gzip(gfe),gzip(gfe)

• Googlebot

  Example user agent
  Googlebot/1.3

• Example user agent
  Mozilla/5.0 ( compatible ; Veooz/1.0 ; +http://www.veooz.com/veoozbot.html )

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/4.0/EU; http://www.wise-guys.nl/)

• Example user agent
  Simpy/1.1 (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)

• Friendica

  Example user agent
  Friendica 'Ginger' 3.3.1-1174; http://ccc.serpens.uberspace.de

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)

• Example user agent
  IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)

• Bing

  Example user agent
  Mozilla/5.0 (compatible;bingbot/2.0;+http://www.bing.com/bingbot.htm)

• Example user agent
  IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)

• Site24x7

  Example user agent
  Site24x7

• Google Mediapartners

  Example user agent
  Mediapartners-Google

• Google Feedfetcher

  Example user agent
  Mozilla/5.0 (compatible) Feedfetcher-Google;(+http://www.google.com/feedfetcher.html)

• Example user agent
  Mozilla/5.0 (compatible; PAD-bot/9.0; +http://www.descargarprogramagratis.com/)

• Bing Preview

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)

• Example user agent
  Moreoverbot/5.00 (+http://www.moreover.com)

• Google App Engine

  Example user agent
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1626.1 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~hex-plus)

• Example user agent
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)

• Example user agent
  OmniExplorer_Bot/6.65a (+http://www.omni-explorer.com) WorldIndexer

• Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)

• Perl

  Example user agent
  libwww-perl/5.36

• Perl

  Example user agent
  Mozilla/5.0 (Windows; Windows NT 6.1; WOW64; rv:2.0b8pre; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8; OfficeLiveConnector.1.4; OfficeLivePatch.1.3; SLCC1; SLCC2; Media Center PC 6.0; GTB6.4; InfoPath.2; en-US; FunWebProducts; Zango 10.1.181.0; SV1; PRTG Network Monitor (www.paessler.com)) Gecko/20101114 Firefox/4.0b8pre QuickTime/7.6.2 Songbird/1.1.2 Web-sniffer/1.0.36 lftp/3.7.4 libwww-perl/5.820 GSiteCrawler/v1.12 rev. 260 Snoopy v1.2

• Example user agent
  IssueCrawler

• Inktomi Slurp

  Example user agent
  Mozilla/3.0 (Slurp.so/1.0; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://2u4.us)

• Example user agent
  Automattic Analytics Crawler/0.2;+http://wordpress.com/crawler/

• Google Page Speed

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/4.0.5 Mobile/8A306 Safari/6531.22.7

• Example user agent
  SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com)

• Example user agent
  BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)

• Perl

  Example user agent
  libwww-perl/5.75

• Yahoo! Pipes

  Example user agent
  Yahoo Pipes 2.0

• Example user agent
  OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )

• Example user agent
  PeerFactor Crawler

• Server Density

  Example user agent
  Server Density Service Monitoring v2

• Example user agent
  WSDLSpider/1.0 (http://www.wsdlworld.com)

• Example user agent
  AcquiloSpider/1.0,gzip(gfe),gzip(gfe)

• Example user agent
  Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)

• Apache Nutch

  Example user agent
  Your Nutch Spider/Nutch-2.2.1

• Example user agent
  SearQuBot/SearQuBot v1.0

• Example user agent
  WebAlta Crawler/1.3.18 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)

• Example user agent
  axadine/ (Axadine Crawler; http://www.axada.de/; )

• CloudFare Always Online

  Example user agent
  Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34

• Perl

  Example user agent
  libwww-perl/5.816

• Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)

• Example user agent
  Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html)

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://clickand.co)

• PowerMapper

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.10.736.0

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://pagn.info)

• FAST Enterprise Crawler

  Example user agent
  FAST Enterprise Crawler/6 (www.fastsearch.com)

• Scrapy

  Example user agent
  Scrapy/1.0.3.post6+g2d688cd (+http://scrapy.org)

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

• Bing Preview

  Example user agent
  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.11067 BingPreview/1.0b

• Google Web Preview

  Example user agent
  Mozilla/5.0 (iPad; CPU OS 5_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/5.1 Mobile/9A334 Safari/7534.48.3

• Example user agent
  Mozilla/5.0 (compatible; ca-crawler/1.0)

• Fastladder

  Example user agent
  Fastladder FeedFetcher/0.01 (http://fastladder.com/; 27 subscribers)

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• Example user agent
  Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)

• Example user agent
  Mozilla/5.0 (lbot/2.6.4) lbot@mailme.not

• Example user agent
  EDI/1.6.0 (Edacious & Intelligent Web Crawler)

• Example user agent
  Xaldon_WebSpider/2.0.b1

• Baiduspider

  Example user agent
  Baiduspider

• Example user agent
  Mozilla/5.0 (compatible; phpservermon/3.1.1; +http://www.phpservermonitor.org)

• Example user agent
  Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0

• Example user agent
  BeijingCrawler

• Example user agent
  Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/)

• Soso Web Spider

  Example user agent
  Sosospider+(+help.soso.com/webspider.htm)

• Example user agent
  uipbot/1.0 (uipbot@semasio.net)

• Example user agent
  TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)

• Example user agent
  ERACrawler/1.0

• Example user agent
  Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; LYCOSA;+http://lycosa.se)

• Example user agent
  Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• Example user agent
  PageBitesHyperBot/600 (http://www.pagebites.com/)

• Soso Image Spider

  Example user agent
  Sosoimagespider ( http://help.soso.com/soso-image-spider.htm)

• Comodo

  Example user agent
  Comodo Spider 1.1

• Ask Jeeves

  Example user agent
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma)

• Example user agent
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)

• Example user agent
  Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)

• Example user agent
  SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)

• 80legs

  Example user agent
  Mozilla/5.0 (compatible; 008/0.85; http://www.80legs.com/webcrawler.html) Gecko/2008032620

• Example user agent
  Mozilla/4.0 (compatible; BOTW Spider; http://botw.org)

• NewsGator

  Example user agent
  NewsGator/2.0 Bot (http://www.newsgator.com)

• Example user agent
  Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8webcrawler) http://skateboarddirectory.com

• Example user agent
  Lycos_Spider_(modspider)

• PagesInventory

  Example user agent
  PagesInventory (robot +http://www.pagesinventory.com)

• W3C Validator.nu

  Example user agent
  Validator.nu/LV http://validator.w3.org/services

• Example user agent
  CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)

• Example user agent
  silk/1.0 (+http://www.slider.com/silk.htm)/3.7

• Baiduspider

  Example user agent
  Baiduspider+(+http://www.baidu.jp/spider/)

• Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)

• htdig

  Example user agent
  htdig/3.1.5 (webmaster@online-medien.de)

• Example user agent
  Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)

• Apache Nutch

  Example user agent
  spider/Nutch-1.5.1 (spider; http://www.xxx.com)

• Example user agent
  Mozilla/5.0 (compatible; Dazoobot/0.1; +http://dazoo.fr)

• Facebook

  Example user agent
  facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)

• Example user agent
  Mozilla/5.0 (lbot/2.6.4) (lbot@mailme.not)

• Example user agent
  SmartAndSimpleWebCrawler/1.3 (https://crawler.dev.java.net),gzip(gfe),gzip(gfe)

• Yahoo! Ad Monitoring

  Example user agent
  Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)

• Example user agent
  Screaming Frog SEO Spider/2.40

• Example user agent
  Slinky v1.0 +http://dentedreality.com.au/projects/slinky/

• Google News

  Example user agent
  Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))

• NewsBlur

  Example user agent
  NewsBlur Feed Finder (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• Google App Engine

  Example user agent
  AppEngine-Google; (+http://code.google.com/appengine; appid: xbmacx)

• Example user agent
  XYZ Spider

• Perl

  Example user agent
  WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803

• Example user agent
  Mozilla/5.0 (compatible; TweetedTimes Bot/1.0 (resolver); +http://tweetedtimes.com)

• Example user agent
  Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)

• Curl

  Example user agent
  curl/7.8 (i386-redhat-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6b) (ipv6 enabled)

• Example user agent
  Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)

• Facebook

  Example user agent
  facebookplatform/1.0 (+http://developers.facebook.com)

• Example user agent
  Mozilla/5.0 (compatible; CloudServerMarketSpider/1.0; +http://www.cloudservermarket.com/spider.html)

• Example user agent
  OmniExplorer_Bot/6.70 (+http://www.omni-explorer.com) WorldIndexer

• Example user agent
  http://www.almaden.ibm.com/cs/crawler

• Example user agent
  Mozilla/5.0 (compatible; TestCrawler)

• Python

  Example user agent
  Python-urllib/2.4

• Example user agent
  OmniExplorer_Bot/5.25 (+http://www.omni-explorer.com) WorldIndexer

• Yahoo! Japan

  Example user agent
  Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/)

• Google Play Newsstand

  Example user agent
  GoogleProducer; (+http://goo.gl/7y4SX)

• Example user agent
  OmniExplorer_Bot/5.28 (+http://www.omni-explorer.com) WorldIndexer

• 80legs

  Example user agent
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620

• Sogou Web Spider

  Example user agent
  Sogou web spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Googlebot

  Example user agent
  Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)

• Example user agent
  Sangfor Spider

• Flamingo Search

  Example user agent
  Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)

• Perl

  Example user agent
  libwww-perl/5.53

• Example user agent
  updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)

• Example user agent
  Zeusbot/0.8.1 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)

• Example user agent
  SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)

• Bing

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0; )"

• FAST MetaWeb Crawler

  Example user agent
  FAST MetaWeb Crawler (helpdesk at fastsearch dot com)

• Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

• Example user agent
  Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

• Ichiro

  Example user agent
  ichiro/1.0 (ichiro@nttr.co.jp)

• Example user agent
  Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)

• Example user agent
  T3census-Crawler/1.0

• Example user agent
  netEstate Impressumscrawler (+http://www.netestate.de/De/Loesungen/Impressumscrawler)

• Example user agent
  Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/)

• Example user agent
  Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender)

• Yahoo! Link Expander

  Example user agent
  Yahoo:LinkExpander:Slingstone

• Example user agent
  IRLbot/2.0 (+http://irl.cs.tamu.edu/crawler)

• Voilabot

  Example user agent
  Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)

• WordPress.com

  Example user agent
  WordPress.com; https://ramaransonvsthecult.wordpress.com

• Example user agent
  tivraSpider/1.0 (crawler@tivra.com)

• Example user agent
  BabalooSpider/1.2 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)

• Example user agent
  Arikus_Spider

• Example user agent
  Comodo HTTP(S) Crawler - http://www.instantssl.com/crawler

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.2-1188; https://f.tschlotfeldt.de

• Yahoo! Japan

  Example user agent
  Y!J-BRP/YFSBJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerBetaJp/2.0; users 0; views 80)

• Yahoo! Image

  Example user agent
  Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)

• Example user agent
  Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)

• Example user agent
  EverbeeCrawler

• Example user agent
  BejiBot Crawler (BNL Services; http://www.bejjan.net/crawler/)

• Bing

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/534+ (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 MsnBot-Media /1.0b

• Alexa

  Example user agent
  ia_archiver/8.7 (Windows NT 5.0; )

• Example user agent
  zspider/0.9-dev http://feedback.redkolibri.com/

• Example user agent
  Mozilla/5.0 (compatible; HiveCrawler/1.2 http://www.businessinsider.com)

• Google Wireless Transcoder

  Example user agent
  Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14

• Example user agent
  adbeat_bot

• Example user agent
  TouTrix crawler/1.0

• Example user agent
  Mozilla/5.0 (compatible; BountiiBot/1.1; +http://bountii.com/contact.php)

• Example user agent
  Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)

• Example user agent
  Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)

• Livedoor

  Example user agent
  Girls/2.0 (livedoor Co.,Ltd.; Peachy 2.1; iPhone; RSS Version 2.0; +http://girls.livedoor.com/)

• Yahoo! Cache System

  Example user agent
  YahooCacheSystem

• Example user agent
  Mozilla/5.0 (compatible; PornSpider/1.0; +http://www.pornspider.net)

• Yandex Favicons

  Example user agent
  Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)

• Perl

  Example user agent
  libwww-perl/5.835

• Example user agent
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)

• Example user agent
  Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)

• Hatena Antenna

  Example user agent
  Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)

• FAST Enterprise Crawler

  Example user agent
  FAST Enterprise Crawler/5.3.4 (crawler@fast.no)

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

• Example user agent
  Jaxified Bot (+http://www.jaxified.com/crawler/)

• Vagabondo

  Example user agent
  Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• Feedly

  Example user agent
  MetaFeedly/1.0 (http://www.feedly.com)

• Python

  Example user agent
  Python-urllib/2.6

• Example user agent
  Mozilla/5.0 (compatible; Orangebot/2.0; support.orangebot@orange.com)

• Example user agent
  Mozilla/5.0 PhantomJS (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/)

• Amazon CloudFront

  Example user agent
  Amazon CloudFront

• Bing

  Example user agent
  msnbot/1.1 ( http://search.msn.com/msnbot.htm)

• Example user agent
  seebot/2.0 (+http://www.seegnify.com/bot)

• Example user agent
  sproose/1.0beta (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com)

• FAST Web Crawler

  Example user agent
  FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

• Xerka WebBot

  Example user agent
  Xerka WebBot v1.0.0 [AIDO_CREA_ab]

• Example user agent
  Mozilla/5.0 (compatible; WbSrch/1.1 +http://wbsrch.com)

• Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

• Curl

  Example user agent
  curl/7.14.1 (i386-portbld-freebsd4.7) libcurl/7.14.1 OpenSSL/0.9.8 zlib/1.1.3

• Apache Nutch

  Example user agent
  MySpider/Nutch-2.2

• Goo

  Example user agent
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)

• Example user agent
  MSR-ISRCCrawler

• Example user agent
  blackspider

• Example user agent
  Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/)

• Example user agent
  Mozilla/4.0 (compatible; Iplexx Spider/1.1)

• OpenWebSpider

  Example user agent
  OpenWebSpider v0.1.4 (http://www.openwebspider.org/)

• Example user agent
  LEIA/3.01pr (LEIAcrawler; leia@gseek.com; http://www.gseek.com)

• Blogtrottr

  Example user agent
  Blogtrottr/2.0

• Example user agent
  Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net)

• Bing

  Example user agent
  "msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)

• Amorank Spider

  Example user agent
  AmorankSpider/0.1; +http://amorank.com/webcrawler.html

• Example user agent
  Teragram/SAS Crawler

• Example user agent
  blog_crawler/1.0

• Example user agent
  Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)

• Example user agent
  SimpleCrawler/0.1

• Example user agent
  Mozilla/5.0 (compatible; Finderbots finder bot; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)

• Example user agent
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15

• Example user agent
  Mozilla/5.0 (compatible; mxbot/1.0; http://www.chainn.com/mxbot.html)

• PHP

  Example user agent
  GuzzleHttp/6.1.0 curl/7.38.0 PHP/7.0.0RC7

• NewsGator

  Example user agent
  NewsGatorOnline/2.0 (http://www.newsgator.com; 2 subscribers)

• Example user agent
  Cityreview Robot (+http://www.cityreview.org/crawler/)

• Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)

• Example user agent
  AppCodes crawler - looking for iOS app mentions. More info: support@appcodes.com. Robots.txt id: AppCodesCrawler

• Example user agent
  Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)

• Example user agent
  Sangfor Spider/0.7

• Example user agent
  Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html

• FAST Web Crawler

  Example user agent
  FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

• YaCy Bot

  Example user agent
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html

• ArchiveBot

  Example user agent
  ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36

• Example user agent
  SheenBot/SheenBot-1.0.4 (Sheen web crawler)

• Google Page Speed

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25

• Go

  Example user agent
  Go 1.1 package http

• Example user agent
  Mozilla/5.0 (compatible; BeslistBot; nl; BeslistBot 1.0; http://www.beslist.nl/

• Example user agent
  HappyFunBot/1.1 ( http://www.happyfunsearch.com/bot.html)

• Example user agent
  Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)

• Example user agent
  Mozilla/4.0 (compatible; Inspyder-Crawler; http://www.inspyder.com)

• Vodafone mCrawler

  Example user agent
  Vodafone mCrawler (bergum@fast.no)

• Example user agent
  SearchSpider/1.2.10

• Example user agent
  Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)

• Quora Link Preview

  Example user agent
  Quora Link Preview/1.0 (http://www.quora.com)

• Example user agent
  Mozilla/5.0 (compatible; Prlog/1.0; +http://prlog.ru/)

• HubPages

  Example user agent
  HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy)

• Google Web Preview

  Example user agent
  Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13

• Example user agent
  Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)

• PHP

  Example user agent
  PHP/5.2.13

• Python

  Example user agent
  Python-urllib/2.5

• Bing

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b

• Example user agent
  Crowsnest/0.5 (+http://www.crowsnest.tv/)

• Yahoo! Seeker

  Example user agent
  YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)

• Googlebot Mobile

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Example user agent
  8bo Crawler Bot

• Magpie RSS

  Example user agent
  MagpieRSS/0.7 ( http://magpierss.sf.net)

• Digg

  Example user agent
  Digg Deeper/v1 (http://digg.com/about)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://shortr.at)

• TelegramBot

  Example user agent
  TelegramBot (like TwitterBot)

• Example user agent
  OmniExplorer_Bot/5.01 (+http://www.omni-explorer.com) WorldIndexer

• Example user agent
  Moreoverbot/5.00 (+http://www.moreover.com; webmaster@moreover.com)

• NewsBlur

  Example user agent
  NewsBlur Feed Fetcher - 0 subscribers - http://www.newsblur.com/site/1719739/nouvelles-cinema (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• CommaFeed

  Example user agent
  CommaFeed/1.0 (http://www.commafeed.com)

• Example user agent
  http://www.Syntryx.com/ ANT Chassis 9.27; Mozilla/4.0 compatible crawler

• Yandex Bot

  Example user agent
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

• Google Mediapartners

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)

• Kouio

  Example user agent
  kouio.com RSS reader

• Bing

  Example user agent
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)._

• Example user agent
  Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)

• Example user agent
  Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)

• NewsBlur

  Example user agent
  NewsBlur Feed Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• Example user agent
  Mozilla/5.0 (compatible; Hailoobot/1.2; http://www.hailoo.com/spider.html)

• Example user agent
  psbot-image (+http://www.picsearch.com/bot.html)

• W3C MobileOK Checker

  Example user agent
  W3C-mobileOK/DDC-1.0 (see http://www.w3.org/2006/07/mobileok-ddc)

• SilverReader

  Example user agent
  SilverReader/1.0; http://silverreader.com

• Example user agent
  Googlebot v2.1 (+http://www.google.com/bot.html) (http://www.openwebspider.org/)

• Alexa

  Example user agent
  ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)

• 360spider

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider

• Example user agent
  Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)

• Example user agent
  WebCompanyCrawler

• Example user agent
  Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

• 360spider

  Example user agent
  360spider-image

• Ruby

  Example user agent
  Ruby, Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6

• SimplePie

  Example user agent
  SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111118194710

• Example user agent
  NMG Spider/0.3 (szukanko.com)

• Example user agent
  Mozilla/5.0 (compatible; StreamScraper/1.0; +http://code.google.com/p/streamscraper/)

• Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)

• Sophora Linkchecker

  Example user agent
  Mozilla/5.0 (compatible; Sophora Linkchecker; http://www.subshell.com)

• Example user agent
  Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)

• Pinterest

  Example user agent
  Mozilla/5.0 (Linux; Android 4.4.2; A3-A11 Build/KOT49H) AppleWebKit/537.36 (KHTML like Gecko) Version/4.0 Chrome/30.0.0.0 Safari/537.36 [Pinterest/Android]

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://sgfx.co)

• Perl

  Example user agent
  libwww-perl/5.808

• Example user agent
  Mozilla/5.0 (compatible; Ex-Crawler/v0.1.5ALPHA SVN rev58; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)

• Example user agent
  Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)

• Googlebot

  Example user agent
  Googlebot/2.1 ( http://www.googlebot.com/bot.html)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://mrk.im)

• Baiduspider

  Example user agent
  Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)

• Example user agent
  mixi-crawler/2.00 (http://mixi.jp/)

• Yandex Newslinks

  Example user agent
  Mozilla/5.0 (compatible; YandexNewslinks; +http://yandex.com/bots)

• Example user agent
  PortalBSpider/2.0 (spider@portalb.com)

• Example user agent
  FyberSpider (+http://www.fybersearch.com/fyberspider.php)

• Bing

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

• PHP

  Example user agent
  Guzzle/4.2.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.11

• Scrapy

  Example user agent
  Scrapy/0.24.6 (+http://scrapy.org)

• Example user agent
  Mozilla/5.0 (Linux; U; Android 4.2.2; it-it; S222 Build/CUBOT) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

• PHP

  Example user agent
  PHP/5.3.77

• Example user agent
  Altresium/4.6 (+http://www.altresium.com/bot.html)

• Apache Nutch

  Example user agent
  your sipder name/Nutch-1.7

• Example user agent
  BlogRangerCrawler/1.0

• Example user agent
  NationalDirectory-WebSpider/1.3

• Example user agent
  DRKSpider - Website link validator - http://www.drk.com.ar/spider/,gzip(gfe),gzip(gfe),gzip(gfe)

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://www.easylink.mobi)

• Googlebot Image

  Example user agent
  Googlebot-Image/1.0

• Yahoo! Seeker

  Example user agent
  LG-C1500 UP.Browser/6.2.3 (GUI) MMP/1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)

• Googlebot

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1

• Example user agent
  Mozilla/5.0 (compatible; Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/)

• Feedly

  Example user agent
  FeedlyApp/1.0 (http://www.feedly.com)

• Example user agent
  Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch

• Example user agent
  ByWebSite Search Engine Spider

• Data Hotel Watchdog

  Example user agent
  Data-Hotel-Watchdog/1.1

• ADmantX

  Example user agent
  ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com

• Example user agent
  YOURLS v1.5.1 +http://yourls.org/ (running on http://fhort.com)

• Example user agent
  jrCrawler/1.0b

• Apache Nutch

  Example user agent
  MSNBot/Nutch-1.5.1

• Bing Preview

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML; like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 BingPreview/1.0b

• Example user agent
  Liquida.it-Crawler/1.0 ( crawler@liquida.it +http://www.liquida.it )

• SimplePie

  Example user agent
  SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111015034325

• Example user agent
  Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Crayon Crawler)

• Example user agent
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)

• Facebook

  Example user agent
  facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php)

• Googlebot

  Example user agent
  Googlebot/2.1(+http://www.googlebot.com/bot.html)

• Example user agent
  Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://z-url.us)

• Example user agent
  Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)

• BlogBridge

  Example user agent
  BlogBridge 6.7 (http://www.blogbridge.com/) 1.6.0_18

• Example user agent
  Mozilla/5.0 GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)

• Example user agent
  Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)

• FeedValidator

  Example user agent
  FeedValidator/1.3

• StatusCake Error Detector

  Example user agent
  StatusCake Error Detector Crawler V0.01

• QuerySeekerSpider

  Example user agent
  QuerySeekerSpider ( http://queryseeker.com/bot.html )

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dla-marbach.de)

• Example user agent
  LinkStats Bot

• Google Feedfetcher

  Example user agent
  FeedFetcher-Google; ( http://www.google.com/feedfetcher.html)

• Typhoeus

  Example user agent
  Typhoeus - https://github.com/typhoeus/typhoeus

• Example user agent
  CheckSite Verification Agent (+http://www.checksite.us)

• Example user agent
  fastlwspider/1.0

• Pinterest

  Example user agent
  Pinterest/0.2 (+http://www.pinterest.com/)

• Example user agent
  niki-bot

• Example user agent
  Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)

• Baiduspider

  Example user agent
  Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

• FAST Web Crawler

  Example user agent
  FAST-WebCrawler/2.2.11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)

• Baiduspider

  Example user agent
  Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html)

• PHP

  Example user agent
  CakePHP

• Pinterest

  Example user agent
  Mozilla/5.0 (Linux; Android 5.0; SAMSUNG-SM-N900A Build/LRX21V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/42.0.2311.129 Mobile Safari/537.36 [Pinterest/Android]

• Example user agent
  New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98)

• Example user agent
  Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.

• Example user agent
  Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)

• Google App Engine

  Example user agent
  AppEngine-Google

• Example user agent
  cis455crawler

• Pinterest

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Mobile/10A525 [Pinterest/iOS]

• Butterfly

  Example user agent
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8

• WordPress

  Example user agent
  WordPress/3.7.1; http://wordpress.com

• Example user agent
  Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)

• Example user agent
  wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)

• Naver Yeti

  Example user agent
  Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)

• Ask Jeeves

  Example user agent
  Inne: Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html)

• Example user agent
  Mozilla/5.0 (compatible; zyklop; +http://www.seoratio.de/zyklop-crawler/)

• Example user agent
  MSRBOT (http://research.microsoft.com/research/sv/msrbot)

• Baiduspider

  Example user agent
  Baiduspider+(+http://www.baidu.com/search/spider.htm)

• Example user agent
  Aleksika Spider/1.0 ( http://www.aleksika.com/)

• Google App Engine

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~chengzhoukun)

• Ichiro

  Example user agent
  ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)

• Google Ads Bot

  Example user agent
  AdsBot-Google ( http://www.google.com/adsbot.html)

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.0-1182; http://frndc.saschaschroeder.eu

• Example user agent
  Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)

• Apache Nutch

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7

• Example user agent
  agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com)

• Borland Delphi

  Example user agent
  Mozilla/3.0 (compatible; Indy Library)

• Example user agent
  WeSEE:Ads/PageBot (http://www.wesee.com/bot/)

• Perl

  Example user agent
  libwww-perl/5.821

• Yahoo! Japan

  Example user agent
  Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)

• Example user agent
  VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)

• Example user agent
  Mozilla/5.0 (compatible; Qwantify/2.0; +https://www.qwant.com/)

• Example user agent
  Screaming Frog SEO Spider/2,55

• Example user agent
  IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)

• RssBar

  Example user agent
  RssBar/1.29 (RssBar for unDonut 1.35)

• Example user agent
  Mozilla/5.0 (compatible; VodpodCrawler/1.0; +http://vodpod.com/site/help)

• Example user agent
  IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler)

• Example user agent
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)

• Example user agent
  Mozilla/5.0 (compatible; PiplBot; http://www.pipl.com/bot/)

• Example user agent
  MetaInspector/3.0.0 (+https://github.com/jaimeiniesta/metainspector)

• Bing

  Example user agent
  msnbot/2.0 (+http://search.msn.com/msnbot.htm)

• Googlebot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• Soso Web Spider

  Example user agent
  Sosospider+(+http://help.soso.com/webspider.htm)

• Example user agent
  Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

• Example user agent
  Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)

• ArchiveBot

  Example user agent
  ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36

• Example user agent
  Mozilla/5.0 (compatible; ZemlyaCrawl/1.0; +http://zemlyaozer.com/bot)

• Friendica

  Example user agent
  Friendica 'Ginger' 3.3.3-RC-1178; https://anthron.octans.uberspace.de/friendica

• Example user agent
  Hatena::Crawler/0.01

• Ask Jeeves

  Example user agent
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)

• Example user agent
  Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html)

• Internet Archive

  Example user agent
  InternetArchive/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)

• Example user agent
  Mozilla/5.0 (compatible; Scrubby/2.2; +http://www.scrubtheweb.com/)

• Example user agent
  DotSpotsBot/0.2 (crawler; support at dotspots.com)

• Example user agent
  WSDL Crawler

• Example user agent
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)

• Bing Preview

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)

• Example user agent
  CowCrawler/CowCrawler-dev (+http://beta.cow.com),gzip(gfe),gzip(gfe)

• Example user agent
  Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler

• Example user agent
  Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)

• Apache Nutch

  Example user agent
  NutchSpider/Nutch-1.4

• Example user agent
  Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)

• Example user agent
  SensikaBot/x.33 (+http://sensika.com)

• Example user agent
  Wotbox/2.01 (+http://www.wotbox.com/bot/)

• Bing

  Example user agent
  (Windows NT 10.0; Googlebot Googlebot-Image/1.0 msnbot-media/1.1 facebookexternalhit/1.1 Trident/7.0) Maxthon/4.0

• Example user agent
  Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)

• Example user agent
  TwitSpider

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://www.h-link.de)

• Example user agent
  woobot/2.0

• ADmantX

  Example user agent
  ADmantX Platform Semantic Analyzer US Async - ADmantX Inc. - www.admantx.com - support@admantx.com

• Blogram

  Example user agent
  BlogramCrawler/1.0.1(+http://blogram.jp/)

• WebbCrawler

  Example user agent
  WebbCrawler 1.0 ( http://badcheese.com/crawler.html )

• Goo Blog Search

  Example user agent
  gooblog/2.0 (http://help.goo.ne.jp/contact/)

• SimplePie

  Example user agent
  SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20121030175911

• Example user agent
  Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )

• Example user agent
  Mozilla/5.0 (compatible; fr-crawler/1.1)

• Example user agent
  igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)

• XMLSitemapGenerator

  Example user agent
  XmlSitemapGenerator - http://xmlsitemapgenerator.org

• Baiduspider

  Example user agent
  Baiduspider ( http://www.baidu.com/search/spider.htm)

• Example user agent
  deepcrawler/3.1 (http://www.queusearch.com/whatis_deepcrawler.php),gzip(gfe),gzip(gfe)

• Example user agent
  YOURLS v1.0-rc +http://yourls.org/ (running on http://epi.my)

• Tiny Tiny RSS

  Example user agent
  Tiny Tiny RSS/1.11.4c63934 (http://tt-rss.org/)

• Example user agent
  WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)

• Example user agent
  haupia-crawler

• Example user agent
  YOURLS v1.5 +http://yourls.org/ (running on http://jmp.su)

• GomezAgent

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• Example user agent
  ProloCrawler/1.0 (http://www.prolo.com)

• WordPress

  Example user agent
  Pachacutec/0.5 (qpImageBuilders.com); WordPress/3.1; http://contacto-latino.com/news

• Yahoo! Japan

  Example user agent
  Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)

• Example user agent
  WNMbot/0.4.0

• Example user agent
  ConveraMultiMediaCrawler/0.1 ( http://www.authoritativeweb.com/crawl)

• Example user agent
  Mozilla/5.0 (compatible; kgbody/2.0; +http://kids.goo.ne.jp/tool/kgbody.php)

• Example user agent
  Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)

• Example user agent
  TwengaBot

• Example user agent
  DomainCrawler/2.0 (info@domaincrawler.com; http://www.domaincrawler.com/bot

• Example user agent
  Lijit Crawler (+http://www.lijit.com/robot/crawler)

• Curl

  Example user agent
  curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

• Apache Nutch

  Example user agent
  Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)

• Example user agent
  Feedfetcher-feedraider; (+http://feedraider.com/help/feedfetcher.html; X subscribers; Allow like Gecko)

• Example user agent
  YOURLS v1.7.1 +http://yourls.org/ (running on https://ivyti.es)

• Example user agent
  Mozilla/5.0 (compatible; socialbm_bot/1.0; +http://spider.socialbm.net)

• Example user agent
  TAMU_CS_IRL_CRAWLER/1.0

• Example user agent
  Superarama.com - BOT/v.0.1

• Example user agent
  Facebot

• PHP

  Example user agent
  Zend_Http_Client

• Example user agent
  Mozilla/5.0 (compatible; StoneSunSpider/1.1)

• Example user agent
  RankFlex.com Webspider

• Example user agent
  ScSpider/0.2

• Apache Nutch

  Example user agent
  CB/Nutch-1.7

• PostRank

  Example user agent
  PostRank/2.0 (postrank.com; 1 subscribers)

• WWW Mechanize

  Example user agent
  WWW-Mechanize/1.64

• Yahoo! Japan

  Example user agent
  Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)

• ShopWiki

  Example user agent
  ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)

• Google News

  Example user agent
  Googlebot-News

• Example user agent
  Apexoo Spider (http://www.apexoo.com/spider/)

• Example user agent
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html; zurukko1552995316;)

• Perl

  Example user agent
  libwww-perl/5.69

• Example user agent
  Adknowledge Engage Crawler

• Example user agent
  tCrawler/0.1,gzip(gfe),gzip(gfe),gzip(gfe)

• Example user agent
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)

• Example user agent
  YOURLS v1.7.1 +http://yourls.org/ (running on http://sresc.io)

• Wayback Archive

  Example user agent
  Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org)

• Example user agent
  Automattic Analytics Crawler/0.1; http://wordpress.com/crawler/

• Apache Httpd

  Example user agent
  Apache-HttpClient/UNAVAILABLE (java 1.4)

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5

• Wget

  Example user agent
  Wget/1.9.1

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.fi/eng/project.html)

• Example user agent
  Mozilla/5.0 (compatible; Squzer/0.0.1; +http://www.declum.com/)

• Example user agent
  RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)

• PowerMapper

  Example user agent
  CrawlerProcess (http://www.PowerMapper.com) /5.5.678.0

• Bing

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; msnbot/2.1) Jakarta Commons-HttpClient/3.0-rc3 PHPCrawl GStreamer souphttpsrc libsoup/2.27.4 PycURL/7.19.0 XML-RPC for PHP 2.2.1 GoogleFriendConnect/1.0 HTMLParser/1.6 gPodder/0.15.2 ( http://gpodder.org/) anw webtool LoadControl/1.3 WinHttp urlgrabber/3.1.0

• Java

  Example user agent
  Java/1.17

• Example user agent
  www2.apserver.net ASP-Ranker Feed Crawler

• FAST Enterprise Crawler

  Example user agent
  FAST EnterpriseCrawler 6

• Example user agent
  Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)

• Example user agent
  EasouSpider

• Example user agent
  Mozilla/5.0 (compatible; Thumbshots.ru; +http://thumbshots.ru/bot) Firefox/3

• Microsoft Social Streams

  Example user agent
  Microsoft MSN SocialStreams Bot,gzip(gfe),gzip(gfe)

• Bing Preview

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 (KHTML, like Gecko) BingPreview/1.0b

• Example user agent
  boitho crawler

• Example user agent
  Mozilla/5.0 (Android 3.0; YRSpider; +http://www.yunrang.com/yrspider.html)

• Perl

  Example user agent
  lwp-trivial/1.38

• Example user agent
  NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com)

• Daumoa

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0

• Example user agent
  Mozilla/5.0 (compatible; PaxleFramework/0.1.1; http://www.paxle.net/en/bot)

• Mixi

  Example user agent
  mixi-check/1.0 (http://mixi.jp/)

• Example user agent
  SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com )

• Yahoo! Image

  Example user agent
  Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)

• Example user agent
  Mozilla/5.0 (compatible; Exploratodo/1.0; +http://www.exploratodo.com

• Bing Preview

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://urlmicro.me)

• Example user agent
  LexxeBot/1.0 (lexxebot@lexxe.com)

• Example user agent
  Autonomy Spider

• Google Feedfetcher

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 2 subscribers; feed-id=17860707833818568603)

• Friendica

  Example user agent
  Mozilla/5.0 (compatible; Friendica)

• Example user agent
  Mozilla/5.0 (compatible; BegunAdvertising/3.0; +http://begun.ru/begun/technology/indexer/)

• Yahoo! Cache System

  Example user agent
  YahooCacheSystem; YahooWebServiceClient

• Example user agent
  DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)

• Example user agent
  Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

• AhrefsBot

  Example user agent
  AhrefsBot.Feeds v0.1; http://ahrefs.com/

• Mapian News Bot

  Example user agent
  "mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)"

• Curl

  Example user agent
  curl/7.19.6 (i686-pc-cygwin) libcurl/7.19.6 OpenSSL/0.9.8n zlib/1.2.3 libidn/1.18 libssh2/1.2

• Example user agent
  SeznamBot/3.0 (+http://fulltext.sblog.cz/)

• PHP

  Example user agent
  PHP

• Example user agent
  Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)

• Example user agent
  Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/)

• Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

• Example user agent
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)

• Linkedin Bot

  Example user agent
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 http://www.linkedin.com)

• Example user agent
  OmniExplorer_Bot/6.60 (+http://www.omni-explorer.com) WorldIndexer

• W3C Unified Validator

  Example user agent
  W3C_Unicorn/1.0 (http://validator.w3.org/services)

• Example user agent
  Mozilla/5.0 (compatible; X11; Linux x86_64; Google-StructuredDataTestingTool; +http://www.google.com/webmasters/tools/richsnippets)

• Example user agent
  Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com)

• Example user agent
  NerdyBot

• BlogPulse

  Example user agent
  BlogPulse (ISSpider-3.0)

• Example user agent
  Mozilla/5.1 (compatible; DBot/7.5.6; +http://a14download.com)

• Example user agent
  Mozilla/5.0 (compatible; SputnikBot/2.2)

• Baiduspider

  Example user agent
  baiduspider-mobile-gate

• Example user agent
  Mozilla/5.0 (compatible; SRCCN!Spider/1.1; +http://site.srccn.com/spider.html)

• Example user agent
  netEstate NE Crawler (+http://www.sengine.info/)

• Example user agent
  Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)

• GNIP

  Example user agent
  UnwindFetchor/1.0 (+http://www.gnip.com/)

• Bing

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.1; +http://www.bing.com/bingbot.htm)

• Ad Muncher

  Example user agent
  Ad Muncher v4.93.33707/4957

• WordPress.com

  Example user agent
  WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 23.19.176.193

• Example user agent
  WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)

• Feedspot

  Example user agent
  Feedspot http://www.feedspot.com

• Facebook

  Example user agent
  FacebookSecurity/2.1 ([www.FacebookSecurity.com])

• Bing Preview

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0) BingPreview/1.0b

• Example user agent
  Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org)

• Example user agent
  IPv4Scan (+http://ipv4scan.com)

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; special_archiver/3.3.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)

• Example user agent
  Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3

• Example user agent
  Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws)

• Example user agent
  Mozilla/5.0 (compatible; FunkyCrawler; )

• Googlebot Mobile

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Example user agent
  Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)

• Example user agent
  Locubot (compatible; Googlebot; msnbot)

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.3-dev-1191; http://toktan.org

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)

• Example user agent
  Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• Example user agent
  WinWebBot/1.0; (Balaena Ltd, UK); http://www.balaena.com/winwebbot.html; winwebbot@balaena.com;)

• Sogou Web Spider

  Example user agent
  Sogou inst spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

• Example user agent
  stq_bot (+http://www.searchteq.de)

• Genieo Web Filter

  Example user agent
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)

• TLSProber

  Example user agent
  TLSProber/0.8

• Example user agent
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 4.0; MSIECrawler; Windows 95)

• Google YouTube Sample

  Example user agent
  Google-YouTubeSample/1.0 Google-HTTP-Java-Client/1.7.0-beta (gzip) Google-HTTP-Java-Client/1.7.0-beta (gzip)

• Example user agent
  Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)

• Baiduspider

  Example user agent
  Baiduspider-image+(+http://www.baidu.com/search/spider.htm)

• Friendica

  Example user agent
  Friendica 'Ginger' 3.3.1-1174; http://friendica.offerman.com

• Example user agent
  truwoGPS/1.0 (GNU/Linux; U; i686; en-US; +http://www.lan4lano.net/browser.html )

• Perl

  Example user agent
  libwww-perl/5.820

• Example user agent
  Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)

• Example user agent
  Google (+https://developers.google.com/+/web/snippet/)

• Domain Re-Animator

  Example user agent
  Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com

• Example user agent
  pr-cy.ru Screenshot Bot

• Perl

  Example user agent
  LWP::Simple/5.800

• Example user agent
  Screaming Frog SEO Spider/1.10

• CommonCrawl

  Example user agent
  CCBot/2.0 (http://commoncrawl.org/faq/)

• Python

  Example user agent
  Python-urllib/3.1

• Grammarly

  Example user agent
  Grammarly/1.0 (http://www.grammarly.com)

• Grub

  Example user agent
  grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)

• Example user agent
  NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)

• Bing

  Example user agent
  msnbot/0.9 (+http://search.msn.com/msnbot.htm)

• WordPress.com

  Example user agent
  WordPress.com; http://fleurdelyz.com

• Vagabondo

  Example user agent
  Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)

• Example user agent
  BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com)

• Example user agent
  Mozilla/4.0 (compatible: FDSE robot)

• AddThis.com

  Example user agent
  AddThis.com robot tech.support@clearspring.com

• Bing

  Example user agent
  msnbot/2.1

• Yahoo! Link Preview

  Example user agent
  Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)

• Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)

• Example user agent
  Bimbot/1.0

• Example user agent
  YoonoCrawler/1.0 (crawler@yoono.com)

• Example user agent
  AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)

• Example user agent
  Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)

• Example user agent
  Mozilla/5.0 (compatible; AbiLogicBot/1.0; +http://www.abilogic.com)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/4.0; FDM; MSIECrawler; Media Center PC 5.0)

• Bing

  Example user agent
  msnbot/2.0b ( http://search.msn.com/msnbot.htm)

• Twisted PageGetter

  Example user agent
  Twisted PageGetter

• Example user agent
  Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)

• htdig

  Example user agent
  htdig/3.1.5 (root@localhost)

• Example user agent
  AnomaliesBot/0.06-dev (The Anomalies Network Search Spider; http://www.anomalies.net; info@anomalies.net)

• WordPress.com

  Example user agent
  WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 93.61.114.253

• Example user agent
  RSSIncludeBot/1.0 (http://www.rssinclude.com/spider)

• Example user agent
  Mozilla/5.0 (Linux; U; Android 2.3.3; ja-jp; sdk Build/GRI34; Y!J-BRZ/YATSHA crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

• LTX71

  Example user agent
  ltx71 - (http://ltx71.com/)

• Example user agent
  Zookabot/2.4; +http://zookabot.com

• Example user agent
  Wotbox/2.01 ( http://www.wotbox.com/bot/)

• Mapian News Bot

  Example user agent
  mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)

• Example user agent
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)

• Example user agent
  CosmixCrawler/0.1

• Radian 6

  Example user agent
  R6_FeedFetcher(www.radian6.com/crawler)

• Python

  Example user agent
  Python-urllib/1.16

• WordPress

  Example user agent
  WordPress/3.8.1; http://www.samsung-blog-news.de

• Example user agent
  GematchCrawler/2.1 (http://www.gematch.com/crawler.html)

• Googlebot

  Example user agent
  Mozilla (Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html))

• Example user agent
  WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)

• Example user agent
  PostPost/1.0 (+http://postpo.st/crawlers)

• Apache Nutch

  Example user agent
  NutchCVS/0.06-dev (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)

• Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• Example user agent
  MonitorBacklinksWP (+http://monitorbacklinks.com/blog/incoming-links/)

• Example user agent
  iaskspider

• Example user agent
  Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)

• Goo

  Example user agent
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)

• Example user agent
  Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)

• Example user agent
  Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)

• Example user agent
  Slackbot (+https://api.slack.com/robots)

• Bing

  Example user agent
  msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)

• Example user agent
  compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com

• Example user agent
  Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com)

• Example user agent
  everyfeed-spider/2.0 (http://www.everyfeed.com)

• Example user agent
  Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://ezurl.me)

• Ruby

  Example user agent
  Ruby, link_thumbnailer

• htdig

  Example user agent
  htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)

• Netvibes

  Example user agent
  Netvibes (http://www.netvibes.com)

• XML Sitemaps Generator

  Example user agent
  Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0

• Apache Nutch

  Example user agent
  asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)

• Example user agent
  NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com)

• Example user agent
  Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)

• Goo

  Example user agent
  Mozilla/5.0 (Linux; U; Android 1.6; ja-jp; SonyEricssonSO-01B Build/R1EA018) AppleWebKit/528.5 (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 (compatible; ichiro/mobile goo; http://help.goo.ne.jp/help/article/1142)

• Bing

  Example user agent
  msnbot-media/1.0 ( http://search.msn.com/msnbot.htm)

• Example user agent
  Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spi_der.html)

• Example user agent
  Orbiter (+http://www.dailyorbit.com/bot.htm)

• Example user agent
  Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)

• Example user agent
  Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) METASpider

• Example user agent
  FFC Trap Door Spider

• Spiderlytics

  Example user agent
  Mozilla/5.0 (compatible; Spiderlytics/1.0; +spider@spiderlytics.com)

• Python

  Example user agent
  Python-urllib/2.0a1

• Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com)

• Java

  Example user agent
  Java/1.7

• Example user agent
  Mozilla/5.0 (compatible; TwitterCrawler)

• Analytics SEO

  Example user agent
  Curious George - www.analyticsseo.com/crawler

• Example user agent
  Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)

• Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)

• HTTPMon

  Example user agent
  HTTPMon/1.0b (http://www.httpmon.com)

• Example user agent
  IRLbot/2.0 ( http://irl.cs.tamu.edu/crawler)

• Vagabondo

  Example user agent
  Mozilla/4.0 (compatible; Vagabondo/2.3; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• Pinterest

  Example user agent
  Pinterest/4.1.3 CFNetwork/672.1.14 Darwin/14.0.0

• Example user agent
  Mozilla/5.0 (compatible; +http://tweetedtimes.com)

• Ruby

  Example user agent
  Ruby

• W3C Internationalization Checker

  Example user agent
  W3C_I18n-Checker/1.0 (http://validator.w3.org/services)

• Example user agent
  IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org)

• Apache Nutch

  Example user agent
  DiscoverEd/Nutch-1.7 (OER search crawler; http://wiki.creativecommons.org/DiscoverEd; webmaster@creativecommons.org)

• Example user agent
  googlebot

• Example user agent
  UniversalFeedParser/3.3 +http://feedparser.org/

• Yandex Images

  Example user agent
  Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

• Example user agent
  Zao-Crawler

• Example user agent
  Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)

• Perl

  Example user agent
  libwww-perl/5.48

• Example user agent
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)

• Perl

  Example user agent
  libwww-perl/5.63

• Friendica

  Example user agent
  Friendica 'Lily of the valley' 3.4.2-1188; https://friends.tausys.de

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/

• Example user agent
  dCrawlBot/1.0.1120

• Sogou Web Spider

  Example user agent
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)

• Example user agent
  Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about)

• Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)

• Yandex Metrika

  Example user agent
  Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters)

• Example user agent
  Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

• Example user agent
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)

• OrangeBot

  Example user agent
  Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)

• Example user agent
  Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620

• Example user agent
  abot/0.1 (abot; http://www.abot.com; abot@abot.com)

• Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)

• Example user agent
  MSRBOT (http://research.microsoft.com/research/sv/msrbot/)

• Example user agent
  Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) )

• Example user agent
  Websquash.com (Add url robot)

• ExactSeek

  Example user agent
  ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com)

• Example user agent
  Mozilla/5.0 (compatible; DomainSCAN/2.0; +http://domainscan.info/robot/)

• Example user agent
  Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )

• FAST mSEARCH Crawler

  Example user agent
  FAST-mSEARCH Crawler 0.1 (bergum@fast.no)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://www.vrbas.be/url)

• Example user agent
  CamontSpider/1.0 http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html

• Example user agent
  QuickFinder Crawler

• Example user agent
  Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

• Example user agent
  Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)

• Example user agent
  MSNBOT/0.1 (http://search.msn.com/msnbot.htm)

• Googlebot

  Example user agent
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• PagesInventory

  Example user agent
  PagesInventory (robot http://www.pagesinvenotry.com)

• Example user agent
  Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/)

• Example user agent
  Mozilla/5.0 (compatible; WebCorp/5.0; +http://www.webcorp.org.uk)

• XMLSitemapGenerator

  Example user agent
  XmlSitemapGenerator - http://XmlSitemapGenerator.org

• Example user agent
  HPI FeedCrawler/0.1 (+http://www.hpi.uni-potsdam.de/meinel/bross/feedcrawler)

• Example user agent
  SynooBot/0.7.1 (SynooBot; http://www.synoo.de/bot.html; webmaster@synoo.com)

• Example user agent
  Pete-Spider/1.1

• Wget

  Example user agent
  Wget/1.6

• Example user agent
  Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)

• Ichiro

  Example user agent
  ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)

• Adaxas

  Example user agent
  Adaxas Spider (http://www.adaxas.net/)

• Example user agent
  Mozilla/5.0 (compatible; loc-crawl/1.10.1 +http://www.google.com)

• Example user agent
  Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org)

• SPDYCheck

  Example user agent
  SPDYCheck SPDY Protocol Tester, see http://spdycheck.org/about.html

• SafeSearch Microdata

  Example user agent
  SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)

• Facebook

  Example user agent
  facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

• Hatena Pagetitle

  Example user agent
  Hatena Pagetitle Agent/1.0

• Magpie RSS

  Example user agent
  MagpieRSS/0.72 \(+http://magpierss.sf.net\)

• Example user agent
  Jaxified Crawler 1.0a (+http://www.jaxified.com/),gzip(gfe),gzip(gfe)

• Example user agent
  Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)

• GomezAgent

  Example user agent
  Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)

• Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040707 Lightningspider/0.9.2

• Example user agent
  Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)

• Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html)

• Radian 6

  Example user agent
  R6_FeedFetcher_(www.radian6.com/crawler)

• Butterfly

  Example user agent
  Mozilla/5.0 (Macintosh; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8

• Example user agent
  WIRE/1.0 (Linux;i686;Bot,Robot,Spider,Crawler)

• Example user agent
  HttpSpider/0.91

• Example user agent
  Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

• Google Page Speed

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36

• Example user agent
  JoobleBot (compatible; http://jooble.org/jooble-bot)

• Example user agent
  Mozilla/5.0 (compatible; SiteBot/0.1; http://www.sitebot.org/robot/)

• Example user agent
  Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)

• Example user agent
  Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html)

• Example user agent
  OmniExplorer_Bot/5.20 (+http://www.omni-explorer.com) WorldIndexer

• Naver Yeti

  Example user agent
  Yeti/1.0 (+http://help.naver.com/robots/)

• Pingdom

  Example user agent
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com)

• Example user agent
  INGRID/2.0 (http://spsearch.ilse.nl/; Startpagina dochter links spider)

• Bing

  Example user agent
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

• Wget

  Example user agent
  Wget/1.10.2

• Example user agent
  Slackbot-LinkExpanding (+https://api.slack.com/robots)

• Google Sitemaps

  Example user agent
  Google-Sitemaps/1.0

• Spinn3r

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19

• Python

  Example user agent
  Python-urllib/1.17

• Example user agent
  Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)

• Example user agent
  MetaGeneratorCrawler/1.3.8 (www.metagenerator.info)

• Example user agent
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://u.vsil.org)

• Yahoo! Japan

  Example user agent
  Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0; users 0; views 248)

• Example user agent
  mozilla/5.0 (iphone; cpu iphone os 8_3 like mac os x) applewebkit/600.1.4 (khtml, like gecko) version/8.0 mobile/12f70 safari/600.1.4 (compatible; googlebot/2.1; +http://www.google.com/bot.html)

• Google Ads Bot

  Example user agent
  AdsBot-Google

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://xurl.co)

• Example user agent
  Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)

• Example user agent
  CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com)

• Example user agent
  TECOMAC-Crawler/0.3

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)

• Example user agent
  Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)

• Sogou Web Spider

  Example user agent
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

• Pinterest

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_4 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Mobile/10B350 [Pinterest/iOS]

• Apache Synapse

  Example user agent
  Mozilla/4.0 (compatible; Synapse)

• Googlebot

  Example user agent
  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36

• ExactSeek

  Example user agent
  ExactSeek Crawler/0.1

• YaCy Bot

  Example user agent
  yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.html

• Example user agent
  Opera/8.01 (J2ME/MIDP; MXit WebBot/1.8.3.119) Opera Mini/3.1

• Google Ads Bot

  Example user agent
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36

• Example user agent
  1NET/Crawler (compatible; 1NET; robot; http://www.1NET.gr)

• Example user agent
  envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)

• Example user agent
  FAST Enteprise Crawler/6 (www dot fastsearch dot com)

• Google App Engine

  Example user agent
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: twitter-mirror),gzip(gfe),gzip(gfe),gzip(gfe)

• 360spider

  Example user agent
  360Spider

• Example user agent
  Yeti-FeedItemCrawler/1.0 (NHN Corp.;+http://help.naver.com/robots/)

• Example user agent
  MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)

• Googlebot Mobile

  Example user agent
  DoCoMo//2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 5.5; SuperSpider/139; Windows 98; Win 9x 4.90)

• Feedbin

  Example user agent
  Feedbin - 9 subscribers

• Example user agent
  OmniExplorer_Bot/4.80 (+http://www.omni-explorer.com) WorldIndexer

• Example user agent
  Mozilla/5.0 (compatible; Webbot/0.1; http://www.webbot.ru/bot.html)

• Wget

  Example user agent
  Wget/1.15 (linux-gnu)

• Apache Jakarta

  Example user agent
  Jakarta Commons-HttpClient/3.0

• Example user agent
  CatchBot/1.0; +http://www.catchbot.com

• Bing

  Example user agent
  \x22Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b\x22

• Example user agent
  Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)

• Perl

  Example user agent
  libwww-perl/6.05

• CiteSeerX

  Example user agent
  flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)

• Example user agent
  Sogou Push Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Example user agent
  LargeSmall Crawler

• Example user agent
  bitlybot

• Googlebot

  Example user agent
  Googlebot/2.1 (+http://www.googlebot.com/bot.html)

• Bing

  Example user agent
  msnbot/0.11 ( http://search.msn.com/msnbot.htm)

• Scrapy

  Example user agent
  Scrapy/0.20.1 (+http://scrapy.org)

• Google FeedBurner

  Example user agent
  FeedBurner/1.0 (http://www.FeedBurner.com)

• Example user agent
  Comodo-Certificates-Spider

• Example user agent
  Mozilla/5.0 (compatible; Host-Spy Crawler/1.2; +http://www.host-spy.com/)

• Example user agent
  Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw)

• Example user agent
  kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)

• Apache Nutch

  Example user agent
  testnutch/Nutch-1.8

• SimplePie

  Example user agent
  freshrss/0.8-dev (Linux; http://freshrss.org) SimplePie/1.4-dev-FreshRSS

• Example user agent
  lb-spider

• Example user agent
  Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)

• Example user agent
  MQbot http://metaquerier.cs.uiuc.edu/crawler

• FAST Crawler

  Example user agent
  Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no)

• Googlebot

  Example user agent
  Googlebot/2.2

• Perl

  Example user agent
  libwww-perl/5.50

• Example user agent
  Mozilla/5.0 (compatible; MerchantCentricBot/1.0)

• Googlebot Mobile

  Example user agent
  KDDI-CA34 UP.Browser/6.2.0.10.2.2(GUI)MMP/2.0 (compatible; KDDI-Googlebot-Mobile/2.1; http://www.google.com/bot.html)

• Example user agent
  PArchiveCrawler/1.0

• Example user agent
  pipeLiner/0.10 (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)

• Example user agent
  YOURLS v1.5.1 +http://yourls.org/ (running on http://rlink.eu)

• Googlebot

  Example user agent
  Googlebot/2.1 (+http://www.google.com/bot.html)

• Example user agent
  BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)

• Example user agent
  Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co)

• WordPress.com

  Example user agent
  WordPress.com; https://positivecups.wordpress.com

• Netvibes

  Example user agent
  Netvibes (http://www.netvibes.com/; 8 subscribers; feedID: 2244192)

• Example user agent
  CSimpleSpider Robot

• W3C Link Checker

  Example user agent
  W3C-checklink/4.0 [4.4] libwww-perl/5.803

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

• Example user agent
  Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0

• Example user agent
  OmniExplorer_Bot/6.62 (+http://www.omni-explorer.com) WorldIndexer

• PHP

  Example user agent
  PEAR HTTP_Request class

• Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)

• Example user agent
  Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)

• Example user agent
  BebopBot/2.5.1 (compatible; media crawler V1; http://www.apassion4jazz.net/bebopbot.html;)

• Example user agent
  woobot/1.1

• Example user agent
  SeekOn Spider 1.9(+http://www.seekon.com/spider.html)

• Example user agent
  Mozilla/5.0 (compatible; Linux i686; Yandex.Gazeta Bot/1.0; +http://gazeta.yandex.ru)

• Example user agent
  envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)

• Example user agent
  Gnam Gnam Spider

• Example user agent
  Yasaklibot/v1.2 (http://www.Yasakli.com/bot.php)

• Example user agent
  LB-Crawler/1.0

• Example user agent
  Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)

• Google Ads Bot

  Example user agent
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari

• Example user agent
  LSSRocketCrawler/1.0 LightspeedSystems

• Example user agent
  Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot-crawler.net)

• Wget

  Example user agent
  Wget/1.14 (linux-gnu)

• PagesInventory

  Example user agent
  PagesInventory (robot http://www.pagesinventory.com)

• Voilabot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)

• Netcraft Survey Bot

  Example user agent
  Mozilla/4.0 (compatible; Netcraft Web Server Survey)

• FAST Crawler

  Example user agent
  Mozilla/5.0 (compatible; FAST Crawler 6.3)

• Example user agent
  Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)

• Example user agent
  Mozilla/5.0 (compatible; ptd-crawler; http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://webarhive.nlc.gov.cn)

• Example user agent
  Mozilla/5.0 (compatible; Sitemap Generator/1.3; +http://www.check-domains.com/sitemap/index.php)

• Scrapy

  Example user agent
  Scrapy/0.16.4 (+http://scrapy.org)

• Ruby

  Example user agent
  Ruby, Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11

• Example user agent
  adidxbot/2.0 (+http://search.msn.com/msnbot.htm)

• Googlebot Mobile

  Example user agent
  DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

• Wget

  Example user agent
  Wget/1.11.4

• Baiduspider

  Example user agent
  Baiduspider-image ( http://www.baidu.com/search/spider.htm)

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://3uves.com)

• Example user agent
  Mozilla/5.0 (compatible; Infohelfer/1.2.0; +http://www.infohelfer.de/crawler.php)

• Jayde

  Example user agent
  Jayde Crawler. http://www.jayde.com

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.2.0 +http://suki.ling.helsinki.f

• 360spider

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider

• Example user agent
  AntBot/1.0 (http://www.ant.com)

• Example user agent
  Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)

• Example user agent
  Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)

• Example user agent
  EricssonR320/R1A (Fast Wireless Crawler)

• Example user agent
  Iframely/0.7.1 (+http://iframe.ly/;)

• Facebook

  Example user agent
  facebookexternalhit/1.1

• Example user agent
  Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )

• WordPress

  Example user agent
  SAE/fetchurl-22wy0njxnm WordPress/3.2.1; http://1.webfront.sinaapp.com

• Pinterest

  Example user agent
  Pinterest/3.3.3 CFNetwork/609.1.4 Darwin/13.0.0

• Example user agent
  Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)

• Example user agent
  noxtrumbot/1.0 (crawler@noxtrum.com)

• Example user agent
  node-fetch/1.0 (+https://github.com/bitinn/node-fetch)

• Example user agent
  Brekiri crawler/1.0

• Example user agent
  it2media-domain-crawler/1.0 on crawler-prod.it2media.de

• Example user agent
  Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler)

• NewsBlur

  Example user agent
  NewsBlur Page Fetcher - 7 subscribers - http://www.newsblur.com/site/3966817/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• Bing Preview

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko

• Google Feedfetcher

  Example user agent
  Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)

• Example user agent
  YOURLS v1.6 +http://yourls.org/ (running on http://revmkt.in)

• Example user agent
  hl_ftien_spider

• Example user agent
  JikeSpider; +http://shoulu.jike.com/spider.html

• Example user agent
  ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)

• Example user agent
  IXE Crawler

• Example user agent
  Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

• Example user agent
  Mozilla/5.0 (compatible; BlinkaCrawler/1.0; http://www.blinka.jp/crawler/)

• Example user agent
  ExactSeekCrawler/1.0

• Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.1.1.0 http://code.google.com/p/abot)

• Example user agent
  JUST-CRAWLER( http://www.justsystems.com/jp/tech/crawler/)

• Example user agent
  Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)

• Example user agent
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)

• Example user agent
  Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)

• Fever

  Example user agent
  Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)

• Example user agent
  OOZBOT/0.17 (--; http://www.setooz.com/oozbot.html; pvvpr at iiit dot ac dot in)

• Example user agent
  ExB Language Crawler 2.1.5 (+http://www.exb.de/crawler)

• Example user agent
  Mozilla/5.0 (compatible; CompSpyBot/1.0; +http://www.compspy.com/spider.html)

• Example user agent
  BuildCMS crawler (http://www.buildcms.com/crawler)

• Example user agent
  Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 http://www.archive.org/details/archive_crawler)

• Tiny Tiny RSS

  Example user agent
  Tiny Tiny RSS/1.10 (http://tt-rss.org/)

• Monitor.us

  Example user agent
  Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)

• Example user agent
  GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)

• Python

  Example user agent
  IMVU Client/498.0 Python-urllib/2.7

• Goo Blog Search

  Example user agent
  gooblogsearch/2.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)

• Example user agent
  TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)

• SimplePie

  Example user agent
  SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20130514092120

• OrangeBot

  Example user agent
  Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com)

• Python

  Example user agent
  Python-urllib/3.4

• Example user agent
  ssearch_bot (sSearch Crawler; http://www.semantissimo.de)

• Example user agent
  ZillaCrawler

• Example user agent
  Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)

• Facebook

  Example user agent
  facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php)

• Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)

• Python

  Example user agent
  Python-urllib/1.15

• WordPress

  Example user agent
  WordPress/3.7.1; http://samsung.hdblog.it

• Example user agent
  FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)

• Example user agent
  magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)

• Example user agent
  WocBot/Mozilla/5.0 (Wocodi Web Crawler 1.0; http://www.wocodi.com/crawler; crawler@wocodi.com)

• Apache Nutch

  Example user agent
  agent6/Nutch-1.1

• Example user agent
  gsa-crawler

• Example user agent
  Mozilla/4.0 (compatible: Shoula robot)

• FTRF: Friendly Robot

  Example user agent
  FTRF: Friendly robot/1.3

• NewsBlur

  Example user agent
  NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• Example user agent
  i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)

• Example user agent
  bingbot

• Example user agent
  BillyBobBot/1.0 (+http://www.billybobbot.com/crawler/)

• Example user agent
  HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko)

• Example user agent
  Checklinks/1.3 (pywikipedia robot; http://toolserver.org/~dispenser/view/Checklinks)

• Feedly

  Example user agent
  FeedlyBot/1.0 (http://feedly.com)

• Example user agent
  Yahoo-Newscrawler/3.9 (news-search-crawler at yahoo-inc dot com)

• Hatena Diary

  Example user agent
  Hatena Diary RSS Module (http://d.hatena.ne.jp/help#rssmodule)

• Yahoo! Japan

  Example user agent
  Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

• Apache Nutch

  Example user agent
  5acd734a-1ed3-4a17-ad9d-0b09ae7bc2d2/Nutch-1.7

• Apache Nutch

  Example user agent
  My Nutch Spider/Nutch-1.6

• Example user agent
  Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://jim-s.com)

• Example user agent
  Mozilla/5.0 (compatible; Muenster University of Applied Sciences; +http://fb02itsscan.fh-muenster.de)

• Baiduspider

  Example user agent
  Baiduspider-testbranch

• Example user agent
  Gigabot/3.0 (http://www.gigablast.com/spider.html)

• Googlebot Video

  Example user agent
  Googlebot-Video/1.0

• Naver Yeti

  Example user agent
  Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/)

• WordPress Jetpack

  Example user agent
  Jetpack by WordPress.com

• Example user agent
  netEstate NE Crawler (+http://www.website-datenbank.de/)

• Perl

  Example user agent
  libwww-perl/5.800

• Example user agent
  MixBot (+http://t.co/GSRLLKex24)

• Apache Nutch

  Example user agent
  Innovazion Crawler/Nutch-1.7

• Python

  Example user agent
  Mozilla/5.0 (Python-urllib2)

• Example user agent
  Mozilla/5.0 (compatible; WebThumbnail/2.2; Website Thumbnail Generator; +http://webthumbnail.org)

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot)

• Example user agent
  Mozilla/5.0 (compatible; ProCogBot/1.0; +http://www.procog.com/spider.html)

• Wget

  Example user agent
  Wget/1.16.1 (darwin14.0.0)

• Example user agent
  Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Bing

  Example user agent
  msnbot/2.0b v1394720947.6703

• Example user agent
  Mozilla/5.0 (compatible; VSMCrawler; http://www.visualsitemapper.com/crawler/)

• Example user agent
  ProoXiBot

• FeedWrangler

  Example user agent
  Feed Wrangler/1.0 (3 subscribers; feed-id=248559; http://feedwrangler.net; Allow like Gecko)

• Example user agent
  YOURLS v1.5 +http://yourls.org/ (running on http://mv2.me)

• Example user agent
  Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)

• Perl

  Example user agent
  libwww-perl/5.803

• Example user agent
  Mozilla/5.0 (compatible; YRSpider; +http://www.yunyun.com/SiteInfo.php?r=about)

• Example user agent
  nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com),gzip(gfe),gzip(gfe)

• Example user agent
  Mozilla/5.0 (compatible; EtaoSpider/1.0; http://open.etao.com/dev/EtaoSpider)

• Example user agent
  Mozilla/5.0 (compatible; TridentSpider/3.1)

• Example user agent
  GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)

• Java

  Example user agent
  Java/1.8.0_25

• WordPress.com

  Example user agent
  WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 23.19.127.211

• Linkdex Bot

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/)

• Example user agent
  PicSpider/1.1 (spider@217-20-118-26.internetserviceteam.com; http://www.bildkiste.de)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://ecra.se)

• Example user agent
  persomm-spider/v1.0

• Example user agent
  Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)

• Example user agent
  KIT webcrawler/0.2.4

• Example user agent
  Phantom.js bot

• Ezooms Crawler

  Example user agent
  Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)

• Example user agent
  Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)

• Googlebot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1

• Example user agent
  Mozilla/5.0 (compatible; Scrubby/2.1; +http://www.scrubtheweb.com/abs/meta-check.html)

• Example user agent
  Mozilla/5.0 (compatible; CrawlBot/1.0.0)

• Example user agent
  Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc)

• Example user agent
  TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, +http://tweetedtimes.com)

• PHP

  Example user agent
  GuzzleHttp/6.1.0 curl/7.35.0 PHP/5.5.9-1ubuntu4.14

• Googlebot

  Example user agent
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• Example user agent
  Sosoblogspider+(+http://help.soso.com/soso-blog-spider.htm)

• Example user agent
  GrowerIdeas Crawler/GrowerIdeas-nutch-1.6 (Crawls URLs for indexing content for our new search startup which aims to provide simple and smart search across curated content. For more info please contact helloworld@growerideas.com. If you think

• Example user agent
  psbot/0.1 (+http://www.picsearch.com/bot.html)

• Example user agent
  Jomjaibot/1.0 Crawl (+http://www.jomjaibot.com/)

• PHP

  Example user agent
  Mozilla/5.0 (compatible; PEAR HTTP_Request class;)

• htdig

  Example user agent
  htdig/3.1.5

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp

• Example user agent
  bot

• Example user agent
  MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)

• Kouio

  Example user agent
  kouio.com RSS reader - 6 subscribers

• Example user agent
  YisouSpider

• Example user agent
  Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://loc.gov),gzip(gfe),gzip(gfe),gzip(gfe)

• Example user agent
  LivelapBot/0.2 (http://site.livelap.com/crawler)

• Apache Nutch

  Example user agent
  Nutch12/Nutch-1.2

• Curl

  Example user agent
  curl/7.29.0

• Example user agent
  TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)

• Example user agent
  Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster)

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com)

• Ask Jeeves

  Example user agent
  Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)

• Example user agent
  YRL_ODP_CRAWLER

• Example user agent
  Mozilla/5.0 (compatible; SemanticBot/1.1)

• Example user agent
  thumbshots-de-bot (+http://www.thumbshots.de/)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://linkcut.eu)

• Example user agent
  Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)

• Java

  Example user agent
  Java/1.6.0_22

• CiteSeerX

  Example user agent
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://netarkivet.dk/webcrawler/)

• Example user agent
  Thesis_php_crawler

• Example user agent
  Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)

• Magpie RSS

  Example user agent
  MagpieRSS/0.61 (+http://magpierss.sf.net)

• Yandex Blogs

  Example user agent
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)

• Google App Engine

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko AppEngine-Google; (+http://code.google.com/appengine; appid: s~oowsxs)

• Example user agent
  Mozilla/5.0 (compatible; SemrushBot/0.97sa; +http://www.semrush.com/bot.html)

• Soso Image Spider

  Example user agent
  Mozilla/5.0 (compatible; Sosoimagespider/2.0; +http://help.soso.com/soso-image-spider.htm)

• Aboundex

  Example user agent
  Aboundex/0.3 (http://www.aboundex.com/crawler/)

• Perl

  Example user agent
  libwww-perl/5.800 (+http://passoire.afraid.org/mylittlewebsurvey/index.html)

• Example user agent
  WoFindeIch Robot 1.0(+http://www.search.wofindeich.com/robot.php)

• 360spider

  Example user agent
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider

• ScreenerBot

  Example user agent
  ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)

• Example user agent
  omgilibot/0.4 +http://omgili.com

• Example user agent
  Mozilla/5.0 (compatible; Ex-Crawler/0.1.5a; powered by ex-crawler; +http://www.ex-crawler.de/) Java/1.6.0_20,gzip(gfe),gzip(gfe)

• Example user agent
  ownCloud Server Crawler

• Example user agent
  Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)

• Cliqzbot

  Example user agent
  Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)

• Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

• 80legs

  Example user agent
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620

• Example user agent
  EsperanzaBot(+http://www.esperanza.to/bot/)

• Example user agent
  OmniExplorer_Bot/6.47 (+http://www.omni-explorer.com) WorldIndexer

• Example user agent
  EmeraldShield.com WebBot

• Example user agent
  DoCoMo/2.0 P904i( m65bot/0.1; c; http://m65.jp/bot.html )

• Example user agent
  Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)

• Example user agent
  Arachnida Web Crawler

• Voilabot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)

• Pingdom

  Example user agent
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)

• Python

  Example user agent
  Python-urllib/3.0

• Example user agent
  Mozilla/5.0 (Linux; U; Android 4.2.2; it-it; S308 Build/CUBOT) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

• Example user agent
  Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)

• Genieo Web Filter

  Example user agent
  Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html

• Example user agent
  domainsbot (+http://www.domainsbot.com)

• Bing

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

• Example user agent
  Spider/5.0

• Example user agent
  ConveraCrawler/0.9c ( http://www.authoritativeweb.com/crawl)

• Python

  Example user agent
  Python-urllib/2.7

• Example user agent
  Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)

• Google Rich Snippets

  Example user agent
  Googlebot-richsnippets

• Livedoor

  Example user agent
  livedoor FeedFetcher/0.01 (http://reader.livedoor.com/; 999 subscribers)

• Example user agent
  GarlikCrawler/1.2 (http://garlik.com/

• Example user agent
  Mozilla/5.0 (compatible; nbot/2.0)

• Example user agent
  YOURLS v1.5 +http://yourls.org/ (running on http://onj.me)

• Ruby

  Example user agent
  Ruby/Curb

• Example user agent
  CatchBot/2.0; +http://www.catchbot.com

• Example user agent
  Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )

• Yandex News

  Example user agent
  Mozilla/5.0 (compatible; YandexNews/3.0; +http://yandex.com/bots)

• Example user agent
  Top10Ranking Spider/3.1 ( http://www.top10Ranking.nl/, Top10ranking.nl heeft op een aantal woorden uw posities in Google gecheckt)

• Perl

  Example user agent
  lwp-trivial/1.36

• Example user agent
  BOT/0.1 (BOT for JCE)

• Example user agent
  Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)

• Example user agent
  OpenLink Virtuoso RDF crawler

• A6 Indexer

  Example user agent
  A6-Indexer

• Example user agent
  Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html)

• Example user agent
  Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)

• Naver Yeti

  Example user agent
  Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/robots/)

• Example user agent
  HuaweiSymantecSpider/1.0 DSE-support@huaweisymantec.com (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)

• Google Adsense Snapshot

  Example user agent
  Adsense-Snapshot-Google

• Yandex Image Resizer

  Example user agent
  Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)

• Example user agent
  Fast PartnerSite Crawler

• Google Mediapartners

  Example user agent
  DoCoMo/2.0 SH905i(c100;TB;W24H16) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)

• Example user agent
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

• Example user agent
  YOURLS v1.5.1 +http://yourls.org/ (running on http://tty.nu)

• WordPress

  Example user agent
  WordPress/3.2.1; http://www.painlog.jp

• Comodo

  Example user agent
  Comodo Spider 1.2

• Example user agent
  DoubleVerify Crawler

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://rutlink.info)

• Example user agent
  Mozilla/5.0 (compatible; YoudaoBot-rts/1.0; http://www.youdao.com/help/webmaster/spider/; )

• Example user agent
  FyberSpider

• Example user agent
  Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)

• PHP

  Example user agent
  HTTP_Request2/2.1.1 (http://pear.php.net/package/http_request2) PHP/5.3.10-1ubuntu3.6

• Curl

  Example user agent
  curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3

• Example user agent
  Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)

• Example user agent
  Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net)

• BoardReader

  Example user agent
  BoardReader/1.0 (http://boardreader.com/info/robots.htm)-Machine1

• Example user agent
  Googlebot

• Google Feedfetcher

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 375 subscribers; feed-id=15381863289700640853)

• Example user agent
  Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)

• Example user agent
  Mozilla/5.0 (compatible; ADJUSTbot/2.0; +http://www.ad-just.jp/)

• Yandex Direct

  Example user agent
  Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)

• Pinterest

  Example user agent
  Pinterest/3356 CFNetwork/711.0.6 Darwin/14.0.0

• PHP

  Example user agent
  GuzzleHttp/6.1.0 curl/7.45.0 PHP/5.6.14

• Example user agent
  OmniExplorer_Bot/5.91c (+http://www.omni-explorer.com) WorldIndexer

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp/3.0 ; http://help.yahoo.com/help/us/ysearch/slurp)

• Python

  Example user agent
  Python-urllib/2.1

• Example user agent
  rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)

• Example user agent
  wscheck.com/1.0.0 (+http://wscheck.com/)

• Example user agent
  holmes/3.12.4 (http://morfeo.centrum.cz/bot)

• Example user agent
  visaduhoc.info Crawler

• Example user agent
  Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!

• Inktomi Slurp

  Example user agent
  Slurp/2.0

• Example user agent
  omgilibot/0.3 http://www.omgili.com/Crawler.html

• Example user agent
  yrspider (Mozilla/5.0 (compatible; YRSpider; http://www.yunrang.com/yrspider.html))

• BacklinkCrawler

  Example user agent
  BacklinkCrawler (http://www.backlinktest.com/crawler.html)

• Example user agent
  1470.net crawler

• Java

  Example user agent
  Java/1.7.0_71

• Example user agent
  dtSearchSpider

• Scrapy

  Example user agent
  Scrapy/0.22.2 (+http://scrapy.org)

• Example user agent
  X-Crawler

• Example user agent
  Mozilla/5.0 (compatible; .None-Crawler/0.1 +http://domains.ericbinek.None/)

• Example user agent
  PsSpider/0.7 (spider@bildkiste.de)

• Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Instapaper/4.0 (+http://www.instapaper.com/)

• Perl

  Example user agent
  libwww-perl/5.65

• Example user agent
  UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/

• WordPress.com

  Example user agent
  WordPress.com; https://egointernationaltruffa.wordpress.com; verifying pingback from 23.19.127.108

• Example user agent
  Zookabot/2.5;++http://zookabot.com

• Example user agent
  Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)

• W3C Markup Validator

  Example user agent
  W3C_Validator/1.3 http://validator.w3.org/services

• Example user agent
  audioCrawlerBot/1.0 (http://www.audiocrawler.com/)

• AhrefsBot

  Example user agent
  Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxyfile1-hrd)

• Example user agent
  Webspider/1.0 (web spider; )

• Example user agent
  MetaHeadersBot (+http://www.metaheaders.com/bot.html)

• W3C CSS Validator

  Example user agent
  Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See <http://validator.w3.org/services>)

• BUbiNG

  Example user agent
  BUbiNG (+http://law.di.unimi.it/BUbiNG.html)

• Example user agent
  Mozilla/4.0 (compatible; BlitzBot)

• Example user agent
  Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)

• Example user agent
  Mozilla/5.0 (compatible; wmtips.com/1.0; +http://www.wmtips.com/tools/)

• Example user agent
  Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Infocrawler; Alexa Toolbar)

• Example user agent
  Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)

• GomezAgent

  Example user agent
  Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)

• Example user agent
  Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)

• Example user agent
  Mozilla/5.0 (Votay bot/4.0; http://www.votay.com/arts/comics/)

• Google Site Verification

  Example user agent
  Mozilla/5.0 (compatible; Google-Site-Verification/1.0)

• Example user agent
  Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)

• Example user agent
  SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)

• Example user agent
  flatlandbot/allspark (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)

• Exabot Thumbnails

  Example user agent
  Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)

• Google Web Preview

  Example user agent
  Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.4 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.4

• Apache Nutch

  Example user agent
  emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)

• Example user agent
  CazoodleBot/0.1 (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com)

• Example user agent
  Mozilla/5.0 (compatible; AnyApexBot/1.0; +http://www.anyapex.com/bot.html)

• VocusBot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; aggregator VocusBot 0.4; +http://www.vocus.com/vnhs.html)

• Pinterest

  Example user agent
  Pinterest/3.2 CFNetwork/672.0.8 Darwin/14.0.0

• Bloglines

  Example user agent
  Bloglines/3.1 (http://www.bloglines.com)

• Google Feedfetcher

  Example user agent
  Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; feed-id=000000000000000000)

• WordPress.com

  Example user agent
  WordPress.com; https://longboardrules.wordpress.com

• Wget

  Example user agent
  Wget/1.12 (linux-gnu)

• Example user agent
  Zspider (+http://www.zhanzhangsou.com/index.htm)

• Example user agent
  DotBot/1.0.1 (http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)

• Example user agent
  YOURLS v1.7 +http://yourls.org/ (running on http://www.bv-xa.com)

• Googlebot

  Example user agent
  Mozilla/5.0 AppleWebKit/537.4 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.4

• Bing

  Example user agent
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)

• Example user agent
  Mozilla/4.0, VM-Crawler/cs version info ofni noisrev sc

• Example user agent
  TheSuBot/0.2 (www.thesubot.de)

• JamesBOT

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html

• Example user agent
  ShowyouBot (http://showyou.com/crawler)

• ROI Hunter

  Example user agent
  ROI Hunter; https://api-dev.roihunter.com

• Example user agent
  Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/om/projekt/Svenska-webbsidor---Kulturarw3/)

• Example user agent
  Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)

• AddThis.com

  Example user agent
  AddThis.com (http://support.addthis.com/)

• Example user agent
  Mozilla/4.0 (compatible; bmdspider; windows 5.1)

• Goo

  Example user agent
  DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)

• WebIndexer

  Example user agent
  WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com)

• Yahoo! Japan

  Example user agent
  Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )

• Example user agent
  bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)

• Example user agent
  Mozilla/5.0 (compatible; WebMasterAid/1.0; +http://wmaid.com/explore/)

• XML Sitemaps Generator

  Example user agent
  Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0

• Digg

  Example user agent
  Mozilla/5.0 (Digg/1.0; support@digg.com)

• Example user agent
  PKU Student Spider

• Bloglovin

  Example user agent
  Bloglovin/1.0 (http://www.bloglovin.com; 1 subscribers)

• Example user agent
  Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com)

• Yahoo! Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

• Twitterbot

  Example user agent
  Twitterbot/1.0

• Example user agent
  FreeWebMonitoring SiteChecker/0.1 (+http://www.freewebmonitoring.com)

• Linkdex Bot

  Example user agent
  linkdex.com/v2.0 and linkdex.com/v2.1

• Magpie RSS

  Example user agent
  MagpieRSS/0.72 (+http://magpierss.sf.net)

• Example user agent
  Liquida Spider 1.0 +http://liquida.com/

• Curl

  Example user agent
  curl/7.35.0

• htdig

  Example user agent
  htdig/3.1.6 (romieu@bastide-medical.fr)

• Inktomi Slurp

  Example user agent
  Mozilla/5.0 (compatible; Yahoo!; U; Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 ()

• Example user agent
  TailsweepBlogCrawler/Tailsweep-2.9-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)

• Example user agent
  rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+pr1-crawler-14@moz.com)

• Archive.org

  Example user agent
  Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)

• Example user agent
  Mozilla/4.0 (compatible; focuseekbot)

• Example user agent
  Slackbot-LinkExpanding+(+https://api.slack.com/robots)

• htdig

  Example user agent
  htdig/3.1.6 (mathieu.peltier@inrialpes.fr)

• Linkedin Bot

  Example user agent
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)

• Example user agent
  RSpider/1.0

• Example user agent
  Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)

• Backlink-Check

  Example user agent
  Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)

• Example user agent
  TheSuBot/0.1 (www.thesubot.de)

• Example user agent
  Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://SiteExplorer.Info/)

• WordPress

  Example user agent
  WordPress/1.2.1 PHP/4.3.9-1

• Example user agent
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)

• Example user agent
  DNSGroup/0.1 (DNS Group Crawler; http://www.dnsgroup.com/; crawler@dnsgroup.com)