No bot name found - Woothee v1.2.0

• 3x detected (2x unique)
  SetLinks bot,SetLinks.ru Crawler

  Example user agent
  Mozilla/5.0 (compatible; SetLinks bot 1.2)

• 5x detected (3x unique)
  SemrushBot,Semrush Bot,SemrushBot/0.97

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)

• 3x detected (1x unique)
  proximic

  Example user agent
  Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)

• 3x detected (1x unique)
  sReleaseBot

  Example user agent
  Mozilla/5.0 (compatible; sReleaseBot/6.1; +http://www.srelease.com/)

• 4x detected (2x unique)
  NE Crawler,netEstate NE Crawler

  Example user agent
  netEstate NE Crawler (+http://www.website-datenbank.de/)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.53

• 3x detected (1x unique)
  SMTBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.1) (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• 5x detected (1x unique)
  MojeekBot

  Example user agent
  Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)

• 2x detected (1x unique)
  Zollard Worm

  Example user agent
  Mozilla/5.0 (compatible; Zollard; Linux)

• 3x detected (1x unique)
  diffbot

  Example user agent
  Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.15 (KHTML, like Gecko; +http://www.diffbot.com) Chrome/24.0.1295.0 Safari/537.15

• 2x detected (1x unique)
  Smartsite HTTPClient

  Example user agent
  Smartsite HTTPClient - Mozilla/4.0 (compatible; MSIE 6.0)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.820

• 6x detected (3x unique)
  SeznamBot,Seznam Bot,SeznamBot/3.1-test

  Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/)

• 3x detected (2x unique)
  Socialradarbot,Socialradar Bot

  Example user agent
  Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; bot@infegy.com)

• 4x detected (1x unique)
  Wget

  Example user agent
  Wget/1.10.2

• 4x detected (3x unique)
  discoverybot,Discovery Bot,discoverybot/2.0

  Example user agent
  Mozilla/5.0 (compatible; discoverybot/2.0; +http://discoveryengine.com/discoverybot.html)

• 3x detected (2x unique)
  uk_lddc_bot,bl.uk_lddc_bot

  Example user agent
  Mozilla/5.0 (bl.uk_lddc_bot; Linux x86_64) PhantomJS/1.9.7 (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

• 6x detected (4x unique)
  Apache Nutch,Nutch,Domnutch Bot,Nutch-based Bot

  Example user agent
  Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)

• 2x detected (1x unique)
  Adbeat Bot

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) adbeat.com/policy AppleWebKit/537.21 (KHTML, like Gecko) Firefox/34.0 Safari/537.21

• 4x detected (1x unique)
  Zookabot

  Example user agent
  Zookabot/2.4; +http://zookabot.com

• 2x detected (1x unique)
  Windows Phone Search

  Example user agent
  Windows Phone Search (Windows Phone OS 7.10;NOKIA;Lumia 900;7.10;8862)

• 2x detected (1x unique)
  Werbefreie Deutsche Suchmaschine

  Example user agent
  Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)

• 5x detected (4x unique)
  Mail.RU_Bot,Mail.Ru,Mail.Ru Bot,Mail.RU_Bot/2.0

  Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

• 4x detected (3x unique)
  spbot,SEOprofiler,OpenLinkProfiler

  Example user agent
  Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.808

• 4x detected (2x unique)
  NE Crawler,netEstate NE Crawler

  Example user agent
  netEstate NE Crawler (+http://www.sengine.info/)

• 6x detected (3x unique)
  Aboundex,Aboundexbot,Aboundexbot/0.3

  Example user agent
  Aboundex/0.3 (http://www.aboundex.com/crawler/)

• 4x detected (2x unique)
  BLEXBot,BLEXBot Crawler

  Example user agent
  Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)

• 3x detected (1x unique)
  SemanticBot

  Example user agent
  Mozilla/5.0 (compatible; SemanticBot/1.1)

• 4x detected (1x unique)
  Applebot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)

• 4x detected (1x unique)
  woriobot

  Example user agent
  Mozilla/5.0 (compatible; woriobot +http://worio.com)

• 2x detected (1x unique)
  007AC9 Crawler

  Example user agent
  Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)

• 3x detected (1x unique)
  MarketwireBot

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT; MarketwireBot +http://www.marketwire.com)

• 3x detected (2x unique)
  ANTIPLAG SPIDER,DAWINCI ANTIPLAG SPIDER

  Example user agent
  Mozilla/4.0 (compatible; DAWINCI ANTIPLAG SPIDER)

• 5x detected (2x unique)
  CareerBot,CareerBot/1.1

  Example user agent
  Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)

• 3x detected (1x unique)
  lbot

  Example user agent
  Mozilla/5.0 (lbot/2.6.4) (lbot@mailme.not)

• 4x detected (3x unique)
  Mail.RU_Bot,Mail.Ru,Mail.Ru Bot

  Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots; +http://go.mail.ru/help/robots)

• 4x detected (2x unique)
  uMBot,uMBot-FC/1.0

  Example user agent
  Mozilla/5.0 (compatible; uMBot-FC/1.0; mailto: crawling@ubermetrics-technologies.com)

• 2x detected (1x unique)
  Digg Feed Fetcher

  Example user agent
  Digg Feed Fetcher 1.0 (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.48

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.50

• 9x detected (2x unique)
  Bing,Bingbot

  Example user agent
  Mozilla/5.0 (compatible;bingbot/2.0;+http://www.bing.com/bingbot.htm)

• 7x detected (2x unique)
  Picsearch bot,psbot

  Example user agent
  psbot-page (+http://www.picsearch.com/bot.html)

• 6x detected (4x unique)
  Apache Nutch,Nutch,DiscoverEd,Nutch-based Bot

  Example user agent
  DiscoverEd/Nutch-1.7 (OER search crawler; http://wiki.creativecommons.org/DiscoverEd; webmaster@creativecommons.org)

• 4x detected (1x unique)
  UnisterBot

  Example user agent
  Mozilla/5.0 (compatible; UnisterBot; crawler@unister.de)

• 3x detected (2x unique)
  woobot,WooRank

  Example user agent
  woobot/2.0

• 4x detected (2x unique)
  FDSE,FDSE robot

  Example user agent
  Mozilla/4.0 (compatible: FDSE robot)

• 5x detected (2x unique)
  TweetmemeBot,Tweetmeme Bot

  Example user agent
  Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0

• 3x detected (2x unique)
  grapeFX,grapeFX/0.9

  Example user agent
  Mozilla/5.0 (compatible; grapeFX/0.9; crawler@grapeshot.co.uk

• 4x detected (3x unique)
  Perl,lwp-trivial,lwp

  Example user agent
  lwp-trivial/1.38

• 2x detected (1x unique)
  RMSnapKit

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) RMSnapKit/1.0

• 3x detected (1x unique)
  GLBot

  Example user agent
  Mozilla/5.1 (compatible; GLBot/1.0; +http://galaxydownloads.co)

• 4x detected (3x unique)
  com Bot,Qualidator.com Bot,Qualidator.com Bot 1.0

  Example user agent
  Mozilla/5.0 (compatible; Qualidator.com Bot 1.0;)

• 3x detected (1x unique)
  KomodiaBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; Win64; x64) KomodiaBot/1.0

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.803

• 3x detected (1x unique)
  LinkStats Bot

  Example user agent
  LinkStats Bot

• 2x detected (1x unique)
  Web Glance

  Example user agent
  Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/2.5

• 2x detected (1x unique)
  Superarama.com - BOT

  Example user agent
  Superarama.com - BOT/v.0.1

• 4x detected (2x unique)
  ZyBorg,WiseNut search engine crawler

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)

• 8x detected (4x unique)
  Sogou Web Spider,sogou spider,Sogou web spider/4.0,Sogou

  Example user agent
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

• 2x detected (1x unique)
  BrokenLinkCheck

  Example user agent
  Mozilla/5.0 (Windows NT 5.1) BrokenLinkCheck.com/1.1

• 4x detected (2x unique)
  PHP,CakePHP

  Example user agent
  CakePHP

• 5x detected (3x unique)
  Daumoa,Daum,Daumoa/3.0

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0

• 2x detected (1x unique)
  Rival IQ Bot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9) AppleWebKit/537.71 (KHTML, like Gecko) Version/7.0 Safari/537.71 (Rival IQ, rivaliq.com)

• 6x detected (1x unique)
  360spider

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider

• 5x detected (2x unique)
  BecomeBot,BecomeBot/3.0

  Example user agent
  Mozilla/5.0 (compatible; BecomeBot/3.0; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)

• 6x detected (3x unique)
  AddThis.com,com robot,AddThis.com robot

  Example user agent
  AddThis.com robot tech.support@clearspring.com

• 4x detected (3x unique)
  iisbot,IIS Site Analysis Web Crawler,IIS Site Analysis

  Example user agent
  iisbot/1.0 (+http://www.iis.net/iisbot.html)

• 3x detected (1x unique)
  ContextAd Bot

  Example user agent
  ContextAd Bot 1.0

• 3x detected (2x unique)
  AboutUsBot,AboutUs Bot

  Example user agent
  AboutUsBot/Harpy (Website Analysis; http://www.aboutus.org/Aboutus:Bot; help@aboutus.org)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/1.15

• 3x detected (1x unique)
  MixBot

  Example user agent
  MixBot (+http://t.co/GSRLLKex24)

• 2x detected (1x unique)
  Powermarks

  Example user agent
  Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT)

• 5x detected (2x unique)
  IRLbot,IRLbot/3.0

  Example user agent
  IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)

• 3x detected (2x unique)
  com Crawler,WebTarantula

  Example user agent
  WebTarantula.com Crawler

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.75

• 3x detected (1x unique)
  GG PeekBot

  Example user agent
  GG PeekBot 2.0 ( http://gg.pl/ http://info.gadu-gadu.pl/praca )

• 3x detected (2x unique)
  stq_bot,Searchteq Bot

  Example user agent
  stq_bot (+http://www.searchteq.de)

• 3x detected (1x unique)
  memorybot

  Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.41 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)

• 5x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Speedy Spider (http://www.entireweb.com)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/1.17

• 3x detected (2x unique)
  URLfilterDB-crawler,URLfilterDB Crawler

  Example user agent
  Mozilla/5.0 (URLfilterDB-crawler/1.1) ufdb/1.0

• 6x detected (2x unique)
  Sogou Web Spider,sogou spider

  Example user agent
  sogou spider

• 4x detected (3x unique)
  WASALive-Bot,WASALive Bot, WASALive-Bot

  Example user agent
  Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)

• 5x detected (3x unique)
  voyager,Voyager Bot,voyager/2.0

  Example user agent
  voyager/2.0 (http://www.kosmix.com/crawler.html)

• 7x detected (3x unique)
  Googlebot,Google PageSpeed Insights,Google

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1

• 3x detected (1x unique)
  LinkpadBot

  Example user agent
  Mozilla/5.0 (compatible; LinkpadBot/1.06; +http://www.linkpad.ru)

• 5x detected (2x unique)
  Accoona-AI-Agent,Accoona-AI-Agent/1.1.2

  Example user agent
  Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)

• 4x detected (1x unique)
  Google Wireless Transcoder

  Example user agent
  Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14

• 3x detected (1x unique)
  seebot

  Example user agent
  seebot/2.0 (+http://www.seegnify.com/bot)

• 3x detected (2x unique)
  SNK Screenshot Bot,Save n Keep Screenshot Bot

  Example user agent
  SNK Screenshot Bot/0.20 ( abuse at savenkeep dot_com )

• 2x detected (1x unique)
  Google Image Proxy

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com)

• 3x detected (1x unique)
  BuiBui-Bot

  Example user agent
  BuiBui-Bot/1.0 (email: buibui[at]dadapro[dot]com)

• 4x detected (3x unique)
  DuckDuckGo-Favicons-Bot,DuckDuck Favicons Bot,DuckDuckGo

  Example user agent
  Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)

• 3x detected (2x unique)
  io bot,Nett.io bot

  Example user agent
  Mozilla/5.0 (compatible; Nett.io bot/1.0; +http://nett.io/)

• 4x detected (2x unique)
  Yahoo Mobile App,Yahoo

  Example user agent
  YahooMobile/1.0 (Yahoo! Search; 3.1.3) (Apple; iPhone; iPhone OS/7.1.2); YHOO_Search_App/3.1.3

• 3x detected (1x unique)
  zitebot

  Example user agent
  Mozilla/5.0 (compatible; zitebot support [at] zite [dot] com +http://zite.com)

• 3x detected (1x unique)
  Google Keyword Suggestion

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko, Google Keyword Suggestion) Chrome/10.0.648.127 Safari/534.16

• 2x detected (1x unique)
  Second Live Client

  Example user agent
  Mozilla/5.0 (Windows; U; ; en-US) AppleWebKit/533.3 (KHTML, like Gecko) SecondLife/3.7.23.297296 (Second Life Release; default skin) Safari/533.3

• 3x detected (1x unique)
  metager2-verification-bot

  Example user agent
  Mozilla/5.0 (compatible; metager2-verification-bot; +http://metager2.de/technology.php)

• 4x detected (1x unique)
  Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2.40

• 7x detected (5x unique)
  Google Ads Bot,AdsBot-Google,AdsBot Google-Mobile,Googlebot,AdsBot Google

  Example user agent
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari

• 2x detected (1x unique)
  Cyberduck

  Example user agent
  Cyberduck/4.6.4 (16610) (Windows XP/5.1) (x86)

• 4x detected (2x unique)
  Peeplo Screenshot Bot,Peeplo Screenshot Bot/0.20

  Example user agent
  Peeplo Screenshot Bot/0.20 ( abuse at peeplo dot_com )

• 3x detected (1x unique)
  LinksCrawler

  Example user agent
  LinksCrawler 0.1beta

• 4x detected (2x unique)
  OOZBOT,yoozBot

  Example user agent
  Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)

• 2x detected (1x unique)
  python-requests

  Example user agent
  python-requests/2.0.0 CPython/2.6.6 Linux/2.6.32-431.el6.x86_64

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/6.05

• 3x detected (2x unique)
  5Bot,Plus5Bot

  Example user agent
  Mozilla/5.1 (compatible; +5Bot/1.0; +http://plus5files.net)

• 3x detected (1x unique)
  LoadTimeBot

  Example user agent
  Mozilla/5.0 (compatible; LoadTimeBot/0.9; +http://www.loadtime.net/bot.html)

• 6x detected (3x unique)
  MJ12bot,MJ12 Bot,MJ12bot/v1.4.3

  Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)

• 2x detected (1x unique)
  EventMachine HttpClient

  Example user agent
  EventMachine HttpClient

• 6x detected (3x unique)
  Exabot,Alexabot,Alexa Crawler

  Example user agent
  Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)

• 3x detected (1x unique)
  redbot

  Example user agent
  RED/1 (http://redbot.org/)

• 2x detected (1x unique)
  python-requests

  Example user agent
  python-requests/2.3.0 CPython/2.7.6 Linux/3.13.0-24-generic

• 4x detected (3x unique)
  Bing Preview,BingPreview,BingBot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML; like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25 BingPreview/1.0b

• 6x detected (2x unique)
  Baiduspider,Baidu Spider

  Example user agent
  BaiDuSpider

• 6x detected (3x unique)
  Googlebot,Google Bot,Google

  Example user agent
  Googlebot

• 4x detected (2x unique)
  New-Sogou-Spider,Sogou Spider

  Example user agent
  New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98)

• 4x detected (2x unique)
  PagesInventory ,PagesInventory Bot

  Example user agent
  PagesInventory (robot +http://www.pagesinventory.com)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/3.1

• 4x detected (2x unique)
  genieBot,GenieKnows

  Example user agent
  genieBot (http://64.5.245.11/faq/faq.html)

• 5x detected (3x unique)
  AcoonBot,Acoon,AcoonBot/4.11.1

  Example user agent
  Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/1.16

• 5x detected (2x unique)
  ZumBot,ZumBot/1.0

  Example user agent
  ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)

• 5x detected (2x unique)
  Accoona-AI-Agent,Accoona-AI-Agent/1.1.1

  Example user agent
  Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)

• 3x detected (2x unique)
  GetintentCrawler,Getintent Crawler

  Example user agent
  GetintentCrawler getintent.com

• 4x detected (1x unique)
  GurujiBot

  Example user agent
  Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)

• 3x detected (1x unique)
  CrawlRobot

  Example user agent
  Mozilla/5.0 (compatible; CrawlRobot; +http://localhost/bot.php)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/2.4

• 3x detected (1x unique)
  WNMbot

  Example user agent
  WNMbot/0.4.0

• 4x detected (1x unique)
  URLAppendBot

  Example user agent
  Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)

• 6x detected (3x unique)
  googlebot,Google Bot,Google

  Example user agent
  mozilla/5.0 (iphone; cpu iphone os 8_3 like mac os x) applewebkit/600.1.4 (khtml, like gecko) version/8.0 mobile/12f70 safari/600.1.4 (compatible; googlebot/2.1; +http://www.google.com/bot.html)

• 5x detected (2x unique)
  Girafabot,Girafabot c

  Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; Girafabot [girafa.com])

• 4x detected (1x unique)
  wonderbot

  Example user agent
  wonderbot/JS 1.0

• 2x detected (1x unique)
  jig browser web

  Example user agent
  Mozilla/4.0 (jig browser web; 1.0.4; N02A)

• 5x detected (2x unique)
  pub-crawler,Yahoo

  Example user agent
  Mozilla/5.0 (compatible; pub-crawler; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.63

• 5x detected (2x unique)
  GurujiBot,GurujiBot/1.0

  Example user agent
  Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)

• 4x detected (3x unique)
  TinEye,TinEye Bot,TinEye/1.1

  Example user agent
  TinEye/1.1 (http://tineye.com/crawler.html)

• 6x detected (3x unique)
  MSRBOT,BingBot,MSRBOT b

  Example user agent
  MSRBOT (http://research.microsoft.com/research/sv/msrbot/)

• 3x detected (1x unique)
  Digincore bot

  Example user agent
  Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.

• 3x detected (2x unique)
  com_bot,LinksManager

  Example user agent
  Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html)

• 6x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)

• 3x detected (1x unique)
  HRCrawler

  Example user agent
  HRCrawler/2.0 (XF86; MacOS x86_64) AppleWebKit/537.31 (KHTML, like Gecko)

• 4x detected (3x unique)
  SafeSearch Microdata,microdata crawler,SafeSearch microdata crawler

  Example user agent
  SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)

• 4x detected (3x unique)
  com_bot,LinksManager,linksmanager_bot

  Example user agent
  Mozilla/5.0 (compatible; LinksManager.com_bot +http://linksmanager.com/linkchecker.html)

• 4x detected (2x unique)
  oBot,oBot/2.3.1

  Example user agent
  Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)

• 4x detected (1x unique)
  SimplePie

  Example user agent
  SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111015034325

• 4x detected (1x unique)
  Facebot

  Example user agent
  Facebot/1.0

• 3x detected (2x unique)
  uk_lddc_bot,bl.uk_lddc_bot

  Example user agent
  bl.uk_lddc_bot/3.3.0-SNAPSHOT-2014-10-07T09:33:31Z (+http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)

• 3x detected (2x unique)
  our-crawler,DueDil Crawler

  Example user agent
  Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)

• 2x detected (1x unique)
  Google Adwords DisplayAds WebRender

  Example user agent
  Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/27.0.1453Safari/537.36

• 6x detected (4x unique)
  Archive.org,archiver/3.1.1 +http://www.archive.org/details/archive.org_bot,Internet Archive Special Archiver,archive.org bot

  Example user agent
  Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)

• 4x detected (2x unique)
  Steeler,Steeler/3.5

  Example user agent
  Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)

• 3x detected (2x unique)
  Experibot_v1,Experibot

  Example user agent
  Experibot_v1 (https://dl.dropboxusercontent.com/u/8024465/site/Info.html)

• 3x detected (2x unique)
  js bot,Phantom.js bot

  Example user agent
  Phantom.js bot

• 4x detected (1x unique)
  Superfeedr bot

  Example user agent
  Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch

• 5x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Speedy Spider (Beta/1.0; www.entireweb.com)

• 6x detected (3x unique)
  Googlebot,acapbot,Google

  Example user agent
  Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)

• 3x detected (2x unique)
  Webspider,RankFlex

  Example user agent
  RankFlex.com Webspider

• 5x detected (2x unique)
  CatchBot,CatchBot/2.0

  Example user agent
  CatchBot/2.0; +http://www.catchbot.com

• 5x detected (3x unique)
  ZyBorg,WiseNut search engine crawler,ZyBorg/1.0 - b

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0 (wn-16.zyborg@looksmart.net; http://www.WISEnutbot.com)

• 5x detected (1x unique)
  TweetedTimes Bot

  Example user agent
  Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)

• 2x detected (1x unique)
  PagePeeker Screenshot Maker

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) domain.com Safari/538.1

• 4x detected (1x unique)
  OpenHoseBot

  Example user agent
  Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)

• 2x detected (1x unique)
  BOT for JCE

  Example user agent
  BOT/0.1 (BOT for JCE)

• 5x detected (1x unique)
  Vagabondo

  Example user agent
  Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)

• 2x detected (1x unique)
  Daumoa

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;)

• 3x detected (1x unique)
  ImplisenseBot

  Example user agent
  ImplisenseBot 1.0

• 3x detected (2x unique)
  LSSRocketCrawler,Lightspeed Systems RocketCrawler

  Example user agent
  LSSRocketCrawler/1.0 LightspeedSystems

• 2x detected (1x unique)
  Shrook

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/434l

• 4x detected (2x unique)
  WeSEE:Ads,WeSEE:Search

  Example user agent
  WeSEE:Ads/PageBot (http://www.wesee.com/bot/)

• 3x detected (1x unique)
  AOLbot

  Example user agent
  Mozilla/5.0 (compatible; AOLbot/4.0; +http://www.aol-soft.com/)

• 5x detected (2x unique)
  SeznamBot,Seznam Bot

  Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/)

• 5x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)

• 4x detected (2x unique)
  ZyBorg,WiseNut search engine crawler

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)

• 7x detected (3x unique)
  Soso Web Spider,Sosospider,Soso Spider

  Example user agent
  Sosospider+(+http://help.soso.com/webspider.htm)

• 2x detected (1x unique)
  Feeddler RSS Reader

  Example user agent
  FeeddlerRSS 2.4 (iPad; iPhone OS 7.1; en_GB)

• 4x detected (2x unique)
  Ruby,link_thumbnailer

  Example user agent
  Ruby, link_thumbnailer

• 2x detected (1x unique)
  python-requests

  Example user agent
  python-requests/2.5.1 CPython/2.7.6 Linux/3.13.0-36-generic

• 5x detected (2x unique)
  SeznamBot,Seznam Bot

  Example user agent
  Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/)

• 3x detected (2x unique)
  IC OpenGraph Crawler,IBM Connections

  Example user agent
  IC OpenGraph Crawler 4.5 (proprietary)

• 4x detected (2x unique)
  ArchiveBot,ArchiveTeam ArchiveBot

  Example user agent
  ArchiveTeam ArchiveBot/20141009.02 (wpull 0.1002a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36

• 5x detected (2x unique)
  Daumoa,Daum

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa 4.0

• 3x detected (2x unique)
  niki-bot,NikiBot

  Example user agent
  niki-bot

• 3x detected (1x unique)
  PWBot

  Example user agent
  Mozilla/5.1 (compatible; PWBot/1.0; +http://opendownload.pw)

• 3x detected (2x unique)
  woobot,WooRank

  Example user agent
  woobot/1.1

• 5x detected (2x unique)
  BecomeBot,BecomeBot/2.3

  Example user agent
  Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)

• 3x detected (2x unique)
  Google Image Proxy,Gmail Image Proxy

  Example user agent
  Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)

• 3x detected (1x unique)
  ca-crawler

  Example user agent
  Mozilla/5.0 (compatible; ca-crawler/1.0)

• 3x detected (1x unique)
  ShowyouBot

  Example user agent
  ShowyouBot (http://showyou.com/crawler)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.805

• 3x detected (2x unique)
  seekbot,focuseekbot

  Example user agent
  Mozilla/4.0 (compatible; focuseekbot)

• 9x detected (3x unique)
  Googlebot,Google Bot,Google

  Example user agent
  Googlebot/2.1 (+http://www.googlebot.com/bot.html)

• 5x detected (4x unique)
  GooglePlusBot,Google Web Snippet,Googlebot,Googlebot snippet

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)

• 6x detected (3x unique)
  Voilabot,Orange Bot,VoilaBot BETA 1.2

  Example user agent
  Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/3.0

• 5x detected (2x unique)
  ICC-Crawler,ICC-Crawler/2.0

  Example user agent
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)

• 3x detected (1x unique)
  DomainSigmaCrawler

  Example user agent
  Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)

• 3x detected (1x unique)
  PiplBot

  Example user agent
  Mozilla/5.0 (compatible; PiplBot; http://www.pipl.com/bot/)

• 3x detected (1x unique)
  FyberSpider

  Example user agent
  FyberSpider (+http://www.fybersearch.com/fyberspider.php)

• 5x detected (2x unique)
  psbot,Picsearch Bot

  Example user agent
  psbot-image (+http://www.picsearch.com/bot.html)

• 2x detected (1x unique)
  Windows-Update-Agent

  Example user agent
  Windows-Update-Agent/7.9.9600.17729 Client-Protocol/1.21

• 4x detected (1x unique)
  bitlybot

  Example user agent
  bitlybot

• 4x detected (1x unique)
  YioopBot

  Example user agent
  Mozilla/5.0 (compatible; YioopBot; +http://173.13.143.74/bot.php)

• 4x detected (2x unique)
  GrapeshotCrawler,GrapeshotCrawler/2.0

  Example user agent
  Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

• 3x detected (1x unique)
  memorybot

  Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.71 +http://archivethe.net/en/index.php/about/internet_memory1 on behalf of DNB)

• 4x detected (1x unique)
  NetSeer Crawler

  Example user agent
  Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)

• 3x detected (2x unique)
  Slackbot,Slackbot-Link-Expanding

  Example user agent
  Slackbot-LinkExpanding+(+https://api.slack.com/robots)

• 3x detected (1x unique)
  hivaBot

  Example user agent
  hivaBot(http://gorgor.ir and info@gorgor.ir)/hivaBot-0.2

• 4x detected (3x unique)
  fr_bot,bnf.fr Bot,bnf.fr_bot

  Example user agent
  Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)

• 5x detected (2x unique)
  Magpie RSS,MagpieRSS

  Example user agent
  MagpieRSS/0.72 \(+http://magpierss.sf.net\)

• 3x detected (1x unique)
  MonoBot

  Example user agent
  Mozilla/5.1 (compatible; MonoBot/1.0; +http://mono.name)

• 5x detected (1x unique)
  TweetedTimes Bot

  Example user agent
  Mozilla/5.0 (compatible; TweetedTimes Bot/1.0 (resolver); +http://tweetedtimes.com)

• 4x detected (2x unique)
  ssearch_bot,sSearch Crawler

  Example user agent
  ssearch_bot (sSearch Crawler; http://www.semantissimo.de)

• 3x detected (1x unique)
  lbot

  Example user agent
  Mozilla/5.0 (lbot/2.6.4) lbot@mailme.not

• 3x detected (2x unique)
  Webcrawler,HubSpot Webcrawler

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler

• 3x detected (1x unique)
  domainsbot

  Example user agent
  domainsbot (+http://www.domainsbot.com)

• 4x detected (2x unique)
  SemrushBot,Semrush Bot

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)

• 4x detected (2x unique)
  Google Page Speed,Google PageSpeed Insights

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25

• 3x detected (1x unique)
  YisouSpider

  Example user agent
  YisouSpider

• 4x detected (3x unique)
  Bing Preview,BingPreview,BingBot

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)

• 6x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)

• 4x detected (1x unique)
  GurujiBot

  Example user agent
  Mozilla/5.0 GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)

• 4x detected (3x unique)
  Bing Preview,BingPreview,BingBot

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0) BingPreview/1.0b

• 2x detected (1x unique)
  Second Live Commerce Client

  Example user agent
  Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0 ; .NET CLR 2.0.50215; SL Commerce Client v1.0; Tablet PC 2.0

• 4x detected (1x unique)
  WeSEE:Search

  Example user agent
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)

• 3x detected (2x unique)
  Google Image Proxy,Gmail Image Proxy

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (via ggpht.com GoogleImageProxy)

• 3x detected (1x unique)
  Scopia Crawler

  Example user agent
  Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)

• 2x detected (1x unique)
  Windows-Update-Agent

  Example user agent
  Windows-Update-Agent/7.9.9600.18094 Client-Protocol/1.21

• 3x detected (1x unique)
  Typhoeus

  Example user agent
  Typhoeus - https://github.com/typhoeus/typhoeus

• 7x detected (2x unique)
  Exabot,Exabot/3.0

  Example user agent
  Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)

• 5x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)

• 3x detected (1x unique)
  memoryBot

  Example user agent
  Mozilla/5.0 (compatible; memoryBot/1.20.235 +http://internetmemory.org/en/)

• 4x detected (2x unique)
  PagesInventory ,PagesInventory Bot

  Example user agent
  PagesInventory (robot http://www.pagesinvenotry.com)

• 3x detected (1x unique)
  uipbot

  Example user agent
  uipbot/1.0 (uipbot@semasio.net)

• 3x detected (2x unique)
  cosmos,Xyleme

  Example user agent
  cosmos/0.9_(robot@xyleme.com)

• 2x detected (1x unique)
  Windows-RSS-Platform

  Example user agent
  Windows-RSS-Platform/2.0 (IE 11.0; Windows NT 6.3)

• 3x detected (1x unique)
  Slackbot

  Example user agent
  Slackbot (+https://api.slack.com/robots)

• 2x detected (1x unique)
  Ruby

  Example user agent
  Ruby

• 5x detected (1x unique)
  360spider

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

• 3x detected (2x unique)
  ru Screenshot Bot,Screenshot Bot

  Example user agent
  pr-cy.ru Screenshot Bot

• 6x detected (2x unique)
  ScreenerBot,ScreenerBot Crawler Beta 2.0

  Example user agent
  ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)

• 6x detected (2x unique)
  Feedly,FeedlyBot

  Example user agent
  FeedlyBot/1.0 (http://feedly.com)

• 5x detected (2x unique)
  WBSearchBot,WBSearchBot/1.1

  Example user agent
  Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.69

• 3x detected (2x unique)
  chlooe Bot,bot-pge.chlooe.com/1.0.0

  Example user agent
  bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)

• 2x detected (1x unique)
  Google Structured-Data TestingTool

  Example user agent
  Mozilla/5.0 (compatible; X11; Linux x86_64; Google-StructuredDataTestingTool; +http://www.google.com/webmasters/tools/richsnippets)

• 4x detected (2x unique)
  SemrushBot,Semrush Bot

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.76

• 3x detected (2x unique)
  adbeat_bot,Adbeat Bot

  Example user agent
  adbeat_bot

• 6x detected (3x unique)
  Yahoo! Japan,Y!J-AGENT,Yahoo

  Example user agent
  Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)

• 3x detected (2x unique)
  com/bot,GroupHigh Bot

  Example user agent
  Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/bot.html)

• 4x detected (1x unique)
  Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2,22

• 3x detected (1x unique)
  Apercite

  Example user agent
  Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)

• 5x detected (3x unique)
  ZyBorg,WiseNut search engine crawler,ZyBorg/1.0

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)

• 3x detected (2x unique)
  Thinglink ImageBot,Elmer

  Example user agent
  Elmer, the Thinglink ImageBot (http://www.thinglink.com/help/ThinglinkImageBot)

• 5x detected (4x unique)
  TinEye,TinEye-bot,TinEye Bot,TinEye Crawler

  Example user agent
  TinEye-bot/0.02 (see http://www.tineye.com/crawler.html)

• 3x detected (2x unique)
  Slackbot,Slackbot-Link-Expanding

  Example user agent
  Slackbot-LinkExpanding (+https://api.slack.com/robots)

• 3x detected (1x unique)
  CMS Crawler

  Example user agent
  Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com)

• 2x detected (1x unique)
  Lucidworks Bot

  Example user agent
  Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com)

• 4x detected (2x unique)
  MixrankBot,Mixrank Bot

  Example user agent
  Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)

• 3x detected (1x unique)
  UniversalFeedParser

  Example user agent
  UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/

• 3x detected (2x unique)
  com/bot,SEOdiver Bot

  Example user agent
  Mozilla/5.0 (compatible; SEOdiver/1.0; +http://www.seodiver.com/bot)

• 4x detected (3x unique)
  Perl,lwp-trivial,lwp

  Example user agent
  lwp-trivial/1.36

• 4x detected (2x unique)
  GomezAgent,Gomez Site Monitor

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0; GomezAgent 3.0) Gecko/20100101 Firefox/13.0.1

• 3x detected (1x unique)
  AskPeterBot

  Example user agent
  Mozilla/5.0 (compatible; AskPeterBot; +http://www.askpeter.de/bot.html)

• 4x detected (1x unique)
  CatchBot

  Example user agent
  CatchBot/1.0; http://www.catchbot.com

• 2x detected (1x unique)
  open-webkit-sharp

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) OpenWebKitSharp

• 2x detected (1x unique)
  SiteCon

  Example user agent
  Mozilla/5.0 (Linux; U; en-gb) AppleWebKit/418.9.1 (KHTML, like Gecko) SiteCon/8.10.22

• 5x detected (2x unique)
  WeSEE:Search,WeSEE:Search/0.1 (Alpha)

  Example user agent
  WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)

• 4x detected (1x unique)
  EveryoneSocialBot

  Example user agent
  Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)

• 6x detected (1x unique)
  Cliqzbot

  Example user agent
  Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)

• 4x detected (3x unique)
  WeSEE_Bot,WeSEE Bot,WeSEE:Search

  Example user agent
  WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/)

• 3x detected (1x unique)
  NerdyBot

  Example user agent
  NerdyBot

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.814

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.36

• 2x detected (1x unique)
  dead link checker

  Example user agent
  www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36

• 4x detected (2x unique)
  Google Page Speed,Google PageSpeed Insights

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36

• 8x detected (4x unique)
  Ezooms Crawler,Ezooms,ezooms.bot,Ezooms/1.0

  Example user agent
  Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)

• 3x detected (1x unique)
  memorybot

  Example user agent
  Mozilla/5.0 (compatible; memorybot/1.20.207 +http://mignify.com/bot.html)

• 3x detected (1x unique)
  hivaBot

  Example user agent
  hivaBot/hivaBot-0.2 (info@gorgor.ir)

• 4x detected (1x unique)
  Wget

  Example user agent
  Wget/1.12 (linux-gnu)

• 4x detected (3x unique)
  AboutUsBot,AboutUs Bot Johnny5,AboutUsBot Johnny5/2.0

  Example user agent
  Mozilla/5.0 (compatible; AboutUsBot Johnny5/2.0; +http://www.AboutUs.org/)

• 2x detected (1x unique)
  vSentry

  Example user agent
  Mozilla/5.0 (Windows; chromeframe/2.4.8.5746) AppleWebKit/1.0 (KHTML, like Gecko) Bromium Safari/1.0

• 2x detected (1x unique)
  DuckDuck App

  Example user agent
  DDG-Android-3.0.11

• 2x detected (1x unique)
  InfegyAtlas

  Example user agent
  Mozilla/5.0 (compatible; Linux; InfegyAtlas/1.0; en-US; collection@infegy.com)

• 5x detected (2x unique)
  YodaoBot,YodaoBot/1.0

  Example user agent
  Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )

• 4x detected (2x unique)
  Daumoa,Daum

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0

• 4x detected (2x unique)
  200PleaseBot,200PleaseBot/1.0

  Example user agent
  Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)

• 5x detected (3x unique)
  meanpathbot,Meanpath Bot,meanpathbot/1.0

  Example user agent
  Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)

• 4x detected (2x unique)
  Yahoo Mobile App,Yahoo

  Example user agent
  YahooMobile/1.0 (Yahoo! Search; 3.1.2) (Apple; iPod touch; iPhone OS/7.1.1); YHOO_Search_App/3.1.2

• 2x detected (1x unique)
  Adbeat Bot

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86; de-de) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4

• 3x detected (1x unique)
  XoviBot

  Example user agent
  Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)

• 4x detected (2x unique)
  Java,Java Standard Library

  Example user agent
  Java/1.17

• 5x detected (2x unique)
  IRLbot,IRLbot/2.0

  Example user agent
  IRLbot/2.0 (+http://irl.cs.tamu.edu/crawler)

• 3x detected (2x unique)
  LivelapBot,Livelap Crawler

  Example user agent
  LivelapBot/0.2 (http://site.livelap.com/crawler)

• 5x detected (3x unique)
  genieBot,GenieKnows,genieBot b

  Example user agent
  genieBot ((http://64.5.245.11/faq/faq.html))

• 6x detected (3x unique)
  MJ12bot,MJ12 Bot,MJ12bot/v1.4.4

  Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)

• 2x detected (1x unique)
  profiller

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.21 (KHTML, like Gecko) profiller Safari/537.21

• 2x detected (1x unique)
  WebThumb

  Example user agent
  Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.19) Gecko WebThumb/1.0

• 2x detected (1x unique)
  Web Glance

  Example user agent
  Mozilla/5.0 (X11; U; Linux; C -) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Webglance/0.1

• 4x detected (2x unique)
  SEOkicks-Robot,SEOkicks Robot

  Example user agent
  Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/2.0a1

• 2x detected (1x unique)
  mbot

  Example user agent
  mbot v.1.16

• 3x detected (1x unique)
  GIDBot

  Example user agent
  GIDBot/2.0 (+http://www.gidnetwork.com/tools/gzip-test.php)

• 4x detected (2x unique)
  CRAZYWEBCRAWLER,Crazy Webcrawler

  Example user agent
  CRAZYWEBCRAWLER 0.9.0, http://www.crazywebcrawler.com

• 5x detected (3x unique)
  Spinn3r,Spinn3r RSS Aggregator,Spinn3r 3.1

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19

• 4x detected (2x unique)
  SemrushBot,Semrush Bot

  Example user agent
  Mozilla/5.0 (compatible; SemrushBot/0.97sa; +http://www.semrush.com/bot.html)

• 2x detected (1x unique)
  python-requests

  Example user agent
  python-requests/2.2.1 CPython/2.7.3 Linux/3.2.0-56-generic

• 4x detected (2x unique)
  EasouSpider,Easou Spider

  Example user agent
  Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)

• 5x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)

• 3x detected (2x unique)
  SkypeUriPreview,Skype URI Preview

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5

• 4x detected (2x unique)
  Comodo,Comodo Spider

  Example user agent
  Comodo Spider 1.2

• 6x detected (1x unique)
  Cliqzbot

  Example user agent
  Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)

• 5x detected (1x unique)
  DotBot

  Example user agent
  Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)

• 6x detected (2x unique)
  OpenWebSpider,OpenWebSpider v0.1.4

  Example user agent
  OpenWebSpider v0.1.4 (http://www.openwebspider.org/)

• 2x detected (1x unique)
  Anonymizied

  Example user agent
  Mozilla/5.0 (Randomized by FreeSafeIP.com/upgrade-to-remove; compatible; MSIE 9.0; Windows NT 5.0WOW64) Chrome/18.0.1229.79

• 4x detected (2x unique)
  PagesInventory ,PagesInventory Bot

  Example user agent
  PagesInventory (robot http://www.pagesinventory.com)

• 5x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/)

• 2x detected (1x unique)
  Adbeat Bot

  Example user agent
  Mozilla/5.0 (X11; U; Linux x86; nl-NL) adbeat.com/policy Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 AppleWebKit/532.4 Safari/532.4

• 4x detected (3x unique)
  Perl,lwp-trivial,lwp

  Example user agent
  lwp-trivial/1.35

• 6x detected (3x unique)
  Googlebot,Google Bot,Google

  Example user agent
  googlebot

• 4x detected (3x unique)
  Mail.RU_Bot,Mail.Ru,Mail.Ru Bot

  Example user agent
  Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.800 (+http://passoire.afraid.org/mylittlewebsurvey/index.html)

• 3x detected (1x unique)
  Finderlein Research Crawler

  Example user agent
  Finderlein Research Crawler 1.0

• 6x detected (3x unique)
  Aboundex,Aboundexbot,Aboundexbot/0.2

  Example user agent
  Aboundex/0.2 (http://www.aboundex.com/crawler/)

• 4x detected (1x unique)
  Wget

  Example user agent
  Wget/1.14 (linux-gnu)

• 3x detected (2x unique)
  com crawler,semantic-visions.com crawler

  Example user agent
  Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/2.7

• 2x detected (1x unique)
  DomainScan Server Monitoring

  Example user agent
  Mozilla/5.0 (compatible; DomainSCAN/2.0; +http://domainscan.info/robot/)

• 4x detected (2x unique)
  Ruby,Curb

  Example user agent
  Ruby/Curb

• 3x detected (1x unique)
  SoftListBot

  Example user agent
  Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)

• 6x detected (4x unique)
  Speedy Spider,Entireweb,Speedy,Speedy Spider Beta/1.2

  Example user agent
  Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)

• 7x detected (4x unique)
  Bing,MsnBot,msnbot-media,BingBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b

• 4x detected (2x unique)
  JamesBOT,WebCrawler

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html

• 4x detected (3x unique)
  OOZBOT,veoozbot,Veooz Bot

  Example user agent
  Mozilla/5.0 ( compatible ; Veooz/1.0 ; +http://www.veooz.com/veoozbot.html )

• 4x detected (1x unique)
  GurujiBot

  Example user agent
  GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)

• 6x detected (1x unique)
  BacklinkCrawler

  Example user agent
  BacklinkCrawler (http://www.backlinktest.com/crawler.html)

• 3x detected (1x unique)
  AntBot

  Example user agent
  AntBot/1.0 (http://www.ant.com)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.821

• 2x detected (1x unique)
  Lightspeed Systems Crawler

  Example user agent
  Mozilla/4.0 (compatible; MSIE 7.0;Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727;.NET CLR 3.0.04506.30) Lightspeedsystems

• 5x detected (2x unique)
  CatchBot,CatchBot/1.0

  Example user agent
  CatchBot/1.0; +http://www.catchbot.com

• 4x detected (3x unique)
  Baiduspider,Baidu Mobile Search,Baidu Spider

  Example user agent
  Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html)

• 2x detected (1x unique)
  Proxy Gear Pro

  Example user agent
  Proxy Gear Pro/2.1 (Windows; N; Windows NT 6.1; en)

• 3x detected (1x unique)
  oBot

  Example user agent
  Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/)

• 2x detected (1x unique)
  Werbefreie Deutsche Suchmaschine

  Example user agent
  Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.de/robot.html)

• 5x detected (4x unique)
  Mail.RU_Bot,Mail.Ru,Mail.Ru Bot,Mail.RU_Bot/2.0

  Example user agent
  Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.816

• 3x detected (1x unique)
  fr-crawler

  Example user agent
  Mozilla/5.0 (compatible; fr-crawler/1.1)

• 3x detected (1x unique)
  AjaxSnapBot

  Example user agent
  Mozilla/5.0 (compatible; AjaxSnapBot; +http://ajaxsnapshots.com/bot.html)

• 2x detected (1x unique)
  JobBoerse Bot

  Example user agent
  Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; Jobboerse.com; http://www.xn--jobbrse-d1a.com) Gecko/20100401 Firefox/24.0

• 3x detected (2x unique)
  fastbot,fastbot crawler

  Example user agent
  fastbot crawler beta 4.0 (+http://www.fastbot.de)

• 6x detected (2x unique)
  ScoutJet,Blekkobot

  Example user agent
  Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)

• 2x detected (1x unique)
  Shrook

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Shrook/1290y

• 4x detected (2x unique)
  Esribot,Esribot/1.0

  Example user agent
  Mozilla/5.0 (compatible; Esribot/1.0; http://www.esrihu.hu/)

• 6x detected (4x unique)
  Bing,msnbot-media,BingBot,MSNBot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X; en-us) AppleWebKit/534+ (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 MsnBot-Media /1.0b

• 4x detected (1x unique)
  Applebot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

• 3x detected (1x unique)
  SMTBot

  Example user agent
  Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• 4x detected (1x unique)
  GurujiBot

  Example user agent
  GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/2.6

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.64

• 3x detected (2x unique)
  SetLinks bot,SetLinks.ru Crawler

  Example user agent
  SetLinks bot 1.0

• 2x detected (1x unique)
  HTTrack

  Example user agent
  Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

• 5x detected (2x unique)
  SISTRIX Crawler,sistrix

  Example user agent
  Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)

• 3x detected (1x unique)
  MetaGeneratorCrawler

  Example user agent
  MetaGeneratorCrawler/1.3.4 (www.metagenerator.info)

• 3x detected (2x unique)
  pagefreezer crawler,PageFreezer

  Example user agent
  PageFreezer (pagefreezer crawler; http://pagefreezer.com/pagefreezer-crawler/; info@pagefreezer.com)

• 3x detected (2x unique)
  CommonCrawler,CommonCrawler Node

  Example user agent
  Mozilla/5.0 CommonCrawler Node MNATI3RZFBD6ZTQNRVQGXBDCQOEXIA4CIDEUKNYZDF3MJSH5G7BCQ5EMADE5TLF.MHCZ3LLTWGHIHMYLM.YFBJBHE6R2YQTHOTJ3N5BGFP457JRX4AJRXGAJYGDAANRY24.cdn0.common.crawl.zone

• 6x detected (3x unique)
  Speedy Spider,Entireweb,Speedy

  Example user agent
  Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)

• 2x detected (1x unique)
  akregator

  Example user agent
  Mozilla/5.0 (X11;) AppleWebKit/534.34 (KHTML, like Gecko) akregator/4.14.1 Safari/534.34

• 3x detected (2x unique)
  imrbot,Mignify Bot

  Example user agent
  Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)

• 2x detected (1x unique)
  Crawler

  Example user agent
  Mozilla/5.0 (compatible; Crawler/0.9; http://linkfluence.net/)

• 2x detected (1x unique)
  Dolphin smalltalk http client

  Example user agent
  Dolphin http client/10.3.0(280) (Android)

• 5x detected (2x unique)
  TweetmemeBot,Tweetmeme Bot

  Example user agent
  Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)

• 3x detected (1x unique)
  SMTBot

  Example user agent
  Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

• 5x detected (1x unique)
  Gluten Free Crawler

  Example user agent
  Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)

• 2x detected (1x unique)
  Bot.AraTurka.com

  Example user agent
  Bot.AraTurka.com/0.0.1

• 5x detected (2x unique)
  IRLbot,IRLbot/2.0 b

  Example user agent
  IRLbot/2.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)

• 5x detected (1x unique)
  360spider

  Example user agent
  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider

• 4x detected (2x unique)
  linkdexbot,Linkdex Bot

  Example user agent
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

• 4x detected (1x unique)
  FyberSpider

  Example user agent
  FyberSpider

• 5x detected (1x unique)
  Vagabondo

  Example user agent
  Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)

• 3x detected (1x unique)
  TestCrawler

  Example user agent
  Mozilla/5.0 (compatible; TestCrawler)

• 3x detected (2x unique)
  online-webceo-bot,webceo Bot

  Example user agent
  Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)

• 5x detected (1x unique)
  thumbshots-de-bot

  Example user agent
  thumbshots-de-bot (+http://www.thumbshots.de/)

• 5x detected (2x unique)
  Magpie RSS,MagpieRSS

  Example user agent
  MagpieRSS/0.7 ( http://magpierss.sf.net)

• 3x detected (1x unique)
  AdvBot

  Example user agent
  Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)

• 3x detected (1x unique)
  DBot

  Example user agent
  Mozilla/5.1 (compatible; DBot/7.0; +http://getdownload.ws)

• 2x detected (1x unique)
  Baidu Mobile Search

  Example user agent
  Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spi_der.html)

• 3x detected (1x unique)
  GarlikCrawler

  Example user agent
  GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)

• 4x detected (3x unique)
  spbot,SEOprofiler,OpenLinkProfiler

  Example user agent
  Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )

• 4x detected (3x unique)
  Perl,lwp-trivial,lwp

  Example user agent
  lwp-trivial/1.41

• 3x detected (1x unique)
  JobdiggerSpider

  Example user agent
  Mozilla/5.0 (compatible; JobdiggerSpider +http://www.jobdigger.nl/spider)

• 3x detected (2x unique)
  UA crawler,Internet Archive

  Example user agent
  Mozilla/2.0 (compatible; ARCHIVE.ORG.UA crawler)

• 6x detected (3x unique)
  80legs,008,80legs/0.83 b

  Example user agent
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620

• 5x detected (1x unique)
  360spider

  Example user agent
  Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider

• 2x detected (1x unique)
  Android Download Manager

  Example user agent
  AndroidDownloadManager

• 3x detected (1x unique)
  IstellaBot

  Example user agent
  Mozilla/5.0 (compatible; IstellaBot/1.18.81 +http://www.tiscali.it/)

• 5x detected (1x unique)
  Tiny Tiny RSS

  Example user agent
  Tiny Tiny RSS/1.10 (http://tt-rss.org/)

• 4x detected (3x unique)
  Bing Preview,BingPreview,BingBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko

• 4x detected (1x unique)
  PHP

  Example user agent
  PHP/5.3.77

• 3x detected (2x unique)
  Web Crawler,Arachnida Web Crawler

  Example user agent
  Arachnida Web Crawler

• 4x detected (1x unique)
  IRLbot

  Example user agent
  IRLbot/2.0 ( http://irl.cs.tamu.edu/crawler)

• 6x detected (3x unique)
  PaperLiBot,Paper.li Bot,PaperLiBot/2.1

  Example user agent
  Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

• 8x detected (3x unique)
  Pingdom,PingdomBot,Pingdom Bot

  Example user agent
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com)

• 3x detected (2x unique)
  Wayback Archive,Wayback Archive Bot

  Example user agent
  Mozilla/5.0 (compatible; waybackarchive.org/1.0; +spider@waybackarchive.org)

• 5x detected (2x unique)
  Python,Python-urllib

  Example user agent
  Python-urllib/2.1

• 3x detected (1x unique)
  NetzCheckBot

  Example user agent
  NetzCheckBot/1.0 (security seal; private-krankenversicherung.de.netzcheck.com; bot@netzcheck.com)

• 4x detected (2x unique)
  SurveyBot,Survey Bot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)

• 2x detected (1x unique)
  Google Toolbar

  Example user agent
  Mozilla/4.0 (compatible; GoogleToolbar 7.5.5111.1712; Windows XP 5.1; MSIE 8.0.6001.18702)

• 2x detected (1x unique)
  FlipboardProxy

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)

• 2x detected (1x unique)
  bot

  Example user agent
  bot

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.65

• 3x detected (1x unique)
  nbot

  Example user agent
  Mozilla/5.0 (compatible; nbot/2.0)

• 5x detected (3x unique)
  ZyBorg,WiseNut search engine crawler,ZyBorg/1.0 Dead Link Checker

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)

• 2x detected (1x unique)
  Android Download Manager

  Example user agent
  AndroidDownloadManager/4.3 (Linux; U; Android 4.3; ME302C Build/JSS15Q)

• 4x detected (1x unique)
  MetaJobBot

  Example user agent
  Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.at/crawler)

• 3x detected (1x unique)
  HyperCrawl

  Example user agent
  Mozilla/5.0 (compatible; HyperCrawl/0.2; +http://www.seograph.net/bot.html)

• 4x detected (2x unique)
  Orangebot,Orange Bot

  Example user agent
  Mozilla/5.0 (compatible; Orangebot/2.0; support.orangebot@orange.com)

• 3x detected (1x unique)
  uMBot

  Example user agent
  Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto: crawling@ubermetrics-technologies.com)

• 3x detected (2x unique)
  NewsBlur,NewsBlur Feed Fetcher

  Example user agent
  NewsBlur Feed Fetcher - 0 subscribers - http://www.newsblur.com/site/1719739/nouvelles-cinema (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• 6x detected (3x unique)
  CommonCrawl,CCBot,ccBot crawler

  Example user agent
  CCBot/2.0 (http://commoncrawl.org/faq/)

• 3x detected (1x unique)
  Kraken

  Example user agent
  Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)

• 4x detected (1x unique)
  woriobot

  Example user agent
  Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)

• 5x detected (2x unique)
  IRLbot,IRLbot/3.0 b

  Example user agent
  IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)

• 3x detected (2x unique)
  Perl,libwww

  Example user agent
  libwww-perl/5.800

• 4x detected (2x unique)
  Girafabot,Girafabot b

  Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0; Girafabot; girafabot at girafa dot com; http://www.girafa.com)

• 4x detected (2x unique)
  linkdexbot,Linkdex Bot

  Example user agent
  Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)

• 9x detected (3x unique)
  Yandex Bot,YandexBot,YandexBot/3.0

  Example user agent
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

• 4x detected (2x unique)
  Java,Java Standard Library

  Example user agent
  Java/1.7

• 3x detected (1x unique)
  JoobleBot

  Example user agent
  JoobleBot (compatible; http://jooble.org/jooble-bot)

• 4x detected (1x unique)
  MetaHeadersBot

  Example user agent
  MetaHeadersBot (+http://www.metaheaders.com/bot.html)

• 3x detected (1x unique)
  MerchantCentricBot

  Example user agent
  Mozilla/5.0 (compatible; MerchantCentricBot/1.0)

• 4x detected (2x unique)
  ZyBorg,WiseNut search engine crawler

  Example user agent
  Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)

• 3x detected (2x unique)
  OpenindexSpider,Openindex Spider

  Example user agent
  Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)

• 4x detected (2x unique)
  Linkdex Bot,linkdexbot

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/)

• 6x detected (1x unique)
  TurnitinBot

  Example user agent
  TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)

• 7x detected (4x unique)
  Exabot Thumbnails,Exabot,Exalead,Exabot-Thumbnails

  Example user agent
  Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)

• 6x detected (1x unique)
  QuerySeekerSpider

  Example user agent
  QuerySeekerSpider ( http://queryseeker.com/bot.html )

• 9x detected (3x unique)
  Googlebot,Google Bot,Google

  Example user agent
  Googlebot/2.1 (http://www.googlebot.com/bot.html)

• 6x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  Your Nutch Spider/Nutch-2.2.1

• 5x detected (1x unique)
  Tiny Tiny RSS

  Example user agent
  Tiny Tiny RSS/1.11.4c63934 (http://tt-rss.org/)

• 4x detected (1x unique)
  Screaming Frog SEO Spider

  Example user agent
  Screaming Frog SEO Spider/2.22

• 6x detected (3x unique)
  Archive.org,heritrix,archive.org bot

  Example user agent
  Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot)

• 3x detected (2x unique)
  StatusCake Error Detector,Detector Crawler

  Example user agent
  StatusCake Error Detector Crawler V0.01

• 4x detected (3x unique)
  Backlink-Check,de/bot,Backlink-Ceck.de

  Example user agent
  Backlink-Ceck.de (+http://www.backlink-check.de/bot.html)

• 1x detected (1x unique)
  Daum

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1

• 9x detected (5x unique)
  Apache Nutch,msnbot,Nutch,MSN,BingBot

  Example user agent
  MSNBot/Nutch-1.5.1

• 2x detected (2x unique)
  BountiiBot,Bountii Bot

  Example user agent
  Mozilla/5.0 (compatible; BountiiBot/1.1; +http://bountii.com/contact.php)

• 3x detected (2x unique)
  rogerBot,rogerbot/1.0

  Example user agent
  Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)

• 4x detected (2x unique)
  Daumoa,Daum

  Example user agent
  Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0;

• 2x detected (2x unique)
  UptimeRobot,Uptime Robot

  Example user agent
  Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

• 4x detected (2x unique)
  OrangeBot,Orange Bot

  Example user agent
  Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com)

• 4x detected (3x unique)
  NewsBlur,Safari,NewsBlur Favicon Fetcher

  Example user agent
  NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• 1x detected (1x unique)
  Baidu Spider

  Example user agent
  Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; baidu Transcoder;)

• 5x detected (1x unique)
  MojeekBot

  Example user agent
  Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)

• 5x detected (2x unique)
  Voilabot,Orange Bot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)

• 7x detected (2x unique)
  Sogou Web Spider,sogou spider

  Example user agent
  Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

• 9x detected (4x unique)
  Gocrawl,Googlebot,Google Bot,Google

  Example user agent
  Googlebot (gocrawl v0.4)

• 5x detected (2x unique)
  Magpie RSS,MagpieRSS

  Example user agent
  MagpieRSS/0.72 (+http://magpierss.sf.net)

• 3x detected (1x unique)
  DotBot

  Example user agent
  dotbot

• 6x detected (3x unique)
  Voilabot,Orange Bot,VoilaBot BETA 1.2

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)

• 4x detected (2x unique)
  Amorank Spider,AmorankSpider

  Example user agent
  AmorankSpider/0.1; +http://amorank.com/webcrawler.html

• 7x detected (4x unique)
  Yandex Blogs,YandexBlogs,Yandex Bot,YandexBot

  Example user agent
  Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots)

• 4x detected (3x unique)
  Analytics SEO,Analytics SEO Crawler,Curious George

  Example user agent
  Curious George - www.analyticsseo.com/crawler

• 5x detected (2x unique)
  Zookabot,Zookabot/2.5

  Example user agent
  Zookabot/2.5;++http://zookabot.com

• 6x detected (4x unique)
  Archive.org,archive.org_bot,Internet Archive,archive.org bot

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)

• 9x detected (4x unique)
  Googlebot,Nutch,Google Bot,Google

  Example user agent
  Googlebot/Nutch-1.7

• 6x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  My Nutch Spider/Nutch-1.6

• 1x detected (1x unique)
  Googlebot

  Example user agent
  Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36

• 3x detected (2x unique)
  NewsBlur,NewsBlur Feed Fetcher

  Example user agent
  NewsBlur Feed Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• 3x detected (1x unique)
  Kouio

  Example user agent
  kouio.com RSS reader

• 6x detected (2x unique)
  Wotbox,Wotbox/2.01

  Example user agent
  Wotbox/2.01 (+http://www.wotbox.com/bot/)

• 3x detected (2x unique)
  websitepulse checker,WebSitePulse

  Example user agent
  websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux)

• 5x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7

• 4x detected (2x unique)
  Google Page Speed,Google PageSpeed Insights

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/4.0.5 Mobile/8A306 Safari/6531.22.7

• 7x detected (5x unique)
  Archive.org,archive.org_bot,Internet Archive,archive.org bot,Archive

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

• 1x detected (1x unique)
  Google Partner Monitoring

  Example user agent
  Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1 google_partner_monitoring FWSzVTDDBz14547302713138T

• 5x detected (3x unique)
  Bing Preview,BingPreview,BingBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

• 3x detected (2x unique)
  NewsBlur,NewsBlur Favicon Fetcher

  Example user agent
  NewsBlur Page Fetcher - 7 subscribers - http://www.newsblur.com/site/3966817/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• 6x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  spider/Nutch-1.5.1 (spider; http://www.xxx.com)

• 2x detected (2x unique)
  EasouSpider,Easou Spider

  Example user agent
  Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us; EasouSpider; +http://www.easou.com/search/spider.html)

• 2x detected (2x unique)
  SputnikImageBot,Sputnik Bot

  Example user agent
  Mozilla/5.0 (compatible; SputnikImageBot/2.2)

• 8x detected (4x unique)
  Google Ads Bot,AdsBot-Google,AdsBot Google,Googlebot

  Example user agent
  AdsBot-Google (+http://www.google.com/adsbot.html)

• 6x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  NutchSpider/Nutch-1.4

• 6x detected (3x unique)
  SeznamBot,Seznam Bot,SeznamBot/3.0

  Example user agent
  SeznamBot/3.0 (+http://fulltext.sblog.cz/)

• 4x detected (2x unique)
  OrangeBot,Orange Bot

  Example user agent
  Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com)

• 4x detected (1x unique)
  360spider

  Example user agent
  360spider-image

• 5x detected (3x unique)
  Sogou Web Spider,sogou spider,Sogou inst spider

  Example user agent
  Sogou inst spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

• 4x detected (1x unique)
  rogerbot

  Example user agent
  rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)

• 8x detected (5x unique)
  Alexa,ia_archiver,Internet Archive,Alexa Crawler,ia_archiver alexa

  Example user agent
  ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)

• 2x detected (1x unique)
  NewsBlur

  Example user agent
  NewsBlur Feed Finder (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

• 5x detected (2x unique)
  AhrefsBot,aHrefs Bot

  Example user agent
  AhrefsBot.Feeds v0.1; http://ahrefs.com/

• 5x detected (3x unique)
  sogou spider,Sogou Pic Spider,Sogou Image Crawler

  Example user agent
  Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

• 3x detected (1x unique)
  SimplePie

  Example user agent
  freshrss/0.8-dev (Linux; http://freshrss.org) SimplePie/1.4-dev-FreshRSS

• 7x detected (3x unique)
  YaCy Bot,yacy,yacybot

  Example user agent
  yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html

• 9x detected (5x unique)
  Googlebot Video,Googlebot,Googlebot-Video,Googlebot-Video/1.0,Google

  Example user agent
  Googlebot-Video/1.0

• 4x detected (1x unique)
  ICC-Crawler

  Example user agent
  ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html)

• 2x detected (2x unique)
  YYSpider,Yunyun Bot

  Example user agent
  Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)

• 5x detected (2x unique)
  ZumBot,ZumBot/1.0

  Example user agent
  Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)

• 5x detected (2x unique)
  Linkedin Bot,LinkedInBot

  Example user agent
  LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)

• 5x detected (4x unique)
  TelegramBot,TwitterBot,TelgramBot,Twitter

  Example user agent
  TelegramBot (like TwitterBot)

• 3x detected (2x unique)
  yrspider,Yunyun Bot

  Example user agent
  Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html)

• 6x detected (4x unique)
  Archive.org,archive.org_bot,Internet Archive,archive.org bot

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html)

• 2x detected (2x unique)
  SputnikBot,Sputnik Bot

  Example user agent
  Mozilla/5.0 (compatible; SputnikBot/2.2)

• 4x detected (2x unique)
  Domain Re-Animator,Domain Re-Animator Bot

  Example user agent
  Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com

• 3x detected (3x unique)
  SEOENGWorldBot,SEOENGBot,SEOENGBot/1.0

  Example user agent
  SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)

• 6x detected (2x unique)
  CiteSeerX,heritrix

  Example user agent
  flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)

• 4x detected (1x unique)
  Applebot

  Example user agent
  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)

• 5x detected (2x unique)
  rogerbot,rogerbot/1.0

  Example user agent
  rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)

• 6x detected (3x unique)
  psbot,Picsearch Bot,psbot/0.1

  Example user agent
  psbot/0.1 (+http://www.picsearch.com/bot.html)

• 6x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  MySpider/Nutch-2.2

• 4x detected (1x unique)
  SimplePie

  Example user agent
  SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20130514092120

• 3x detected (3x unique)
  Gazeta Bot,Yandex Bot,YandexBot

  Example user agent
  Mozilla/5.0 (compatible; Linux i686; Yandex.Gazeta Bot/1.0; +http://gazeta.yandex.ru)

• 2x detected (2x unique)
  YRSpider,Yunyun Bot

  Example user agent
  Mozilla/5.0 (compatible; YRSpider; +http://www.yunyun.com/SiteInfo.php?r=about)

• 7x detected (4x unique)
  Pingdom,PingdomBot,Pingdom Bot,pingdom.com_bot 1.4

  Example user agent
  Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)

• 4x detected (1x unique)
  WebbCrawler

  Example user agent
  WebbCrawler 1.0 ( http://badcheese.com/crawler.html )

• 6x detected (2x unique)
  magpie-crawler,magpie-crawler/1.1

  Example user agent
  magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)

• 5x detected (1x unique)
  360spider

  Example user agent
  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider

• 3x detected (1x unique)
  Fever

  Example user agent
  Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)

• 3x detected (1x unique)
  TweetedTimes Bot

  Example user agent
  TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, +http://tweetedtimes.com)

• 2x detected (1x unique)
  Dazoobot

  Example user agent
  Mozilla/5.0 (compatible; Dazoobot/0.1; +http://dazoo.fr)

• 2x detected (1x unique)
  NalezenCzBot

  Example user agent
  NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler)

• 7x detected (2x unique)
  Exabot,Exabot/3.0/BiggerBetter

  Example user agent
  Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)

• 3x detected (2x unique)
  omgilibot,Omgili bot

  Example user agent
  omgilibot/0.3 +http://www.omgili.com/Crawler.html

• 2x detected (1x unique)
  nlcrawler

  Example user agent
  nlcrawler/1.0 (+http://northernlight.com/)

• 6x detected (2x unique)
  CiteSeerX,heritrix

  Example user agent
  Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)

• 9x detected (4x unique)
  Google News,Googlebot,Googlebot-News,Google

  Example user agent
  Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))

• 3x detected (1x unique)
  Kouio

  Example user agent
  kouio.com RSS reader - 6 subscribers

• 4x detected (3x unique)
  redditbot,Reddit,Reddit Bot

  Example user agent
  Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)

• 6x detected (4x unique)
  Archive.org,archive.org_bot,Internet Archive,archive.org bot

  Example user agent
  Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)

• 1x detected (1x unique)
  Baidu Spider

  Example user agent
  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8;baidu Transcoder) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729)

• 6x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  Spider/Nutch-2.3-SNAPSHOT (Webcrawler)

• 2x detected (2x unique)
  VSMCrawler,Visual Site Mapper Crawler

  Example user agent
  Mozilla/5.0 (compatible; VSMCrawler; http://www.visualsitemapper.com/crawler/)

• 7x detected (3x unique)
  YaCy Bot,yacy,yacybot

  Example user agent
  yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.html

• 4x detected (1x unique)
  SimplePie

  Example user agent
  SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20121030175911

• 2x detected (2x unique)
  SensikaBot,Sensika Bot

  Example user agent
  SensikaBot/x.33 (+http://sensika.com)

• 7x detected (3x unique)
  ShopWiki,Help:Bot,ShopWiki/1.0

  Example user agent
  ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)

• 4x detected (2x unique)
  discobot,discobot/1.0

  Example user agent
  Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)

• 6x detected (3x unique)
  ExactSeek,Nutch,ExactSeek Crawler

  Example user agent
  ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com)

• 4x detected (1x unique)
  Superfeedr bot

  Example user agent
  Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!

• 6x detected (2x unique)
  CiteSeerX,heritrix

  Example user agent
  mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)

• 6x detected (3x unique)
  YoudaoBot,Youdao Bot,YoudaoBot/1.0

  Example user agent
  Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )

• 5x detected (3x unique)
  Bing Preview,BingPreview,BingBot

  Example user agent
  Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/538.14 (KHTML, like Gecko) BingPreview/1.0b

• 6x detected (2x unique)
  CiteSeerX,heritrix

  Example user agent
  Mozilla/5.0 (compatible; heritrix/1.14.4 +http://netarkivet.dk/webcrawler/)

• 3x detected (1x unique)
  FTRF: Friendly Robot

  Example user agent
  FTRF: Friendly robot/1.3

• 4x detected (1x unique)
  Wget

  Example user agent
  Wget/1.6

• 8x detected (4x unique)
  Google News,Googlebot,Googlebot-News,Google

  Example user agent
  Googlebot-News

• 4x detected (1x unique)
  Wget

  Example user agent
  Wget/1.15 (linux-gnu)

• 7x detected (5x unique)
  Google Ads Bot,AdsBot-Google,AdsBot Google-Mobile,Googlebot,AdsBot Google

  Example user agent
  AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36

• 6x detected (2x unique)
  FAST Web Crawler,FAST-WebCrawler

  Example user agent
  FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)

• 3x detected (2x unique)
  Mapian News Bot,mapion-news-bot

  Example user agent
  mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)

• 3x detected (2x unique)
  OrangeBot,Orange Bot

  Example user agent
  Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)

• 2x detected (2x unique)
  PowerMapper,CrawlerProcess

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0) CrawlerProcess (http://www.PowerMapper.com) /5.10.736.0

• 2x detected (2x unique)
  Bing Preview,BingBot

  Example user agent
  Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Nokia; Lumia 520) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Mobile Safari/537.36 Edge/12.0 BingPreview/1.0b

• 7x detected (3x unique)
  Googlebot,Google PageSpeed Insights,Google

  Example user agent
  Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1

• 9x detected (3x unique)
  Googlebot,Google Bot,Google

  Example user agent
  Googlebot/2.1(+http://www.googlebot.com/bot.html)

• 9x detected (3x unique)
  Googlebot,Google Bot,Google

  Example user agent
  Googlebot/2.1 ( http://www.googlebot.com/bot.html)

• 9x detected (4x unique)
  Googlebot,Google Bot,Googlebot/2.1,Google

  Example user agent
  Googlebot/2.1 (+http://www.google.com/bot.html)

• 6x detected (1x unique)
  FAST Enterprise Crawler

  Example user agent
  FAST Enterprise Crawler/5.3.4 (crawler@fast.no)

• 2x detected (2x unique)
  FAST Web Crawler,FAST-WebCrawler

  Example user agent
  Opera/30.0.1835.59 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/29.16 AppleWebKit/599.0+ (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 FAST-WebCrawler/3.8

• 9x detected (2x unique)
  Baiduspider,Baidu Spider

  Example user agent
  Baiduspider ( http://www.baidu.com/search/spider.htm)

• 3x detected (2x unique)
  Flamingo Search,Flamingo_SearchEngine

  Example user agent
  Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)

• 4x detected (1x unique)
  Wget

  Example user agent
  Wget/1.16.1 (darwin14.0.0)

• 6x detected (2x unique)
  80legs,008

  Example user agent
  Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620

• 2x detected (2x unique)
  Bing Preview,BingBot

  Example user agent
  Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)

• 4x detected (1x unique)
  Wget

  Example user agent
  Wget/1.11.4

• 3x detected (2x unique)
  Blogram,BlogramCrawler

  Example user agent
  BlogramCrawler/1.0.1(+http://blogram.jp/)

• 9x detected (2x unique)
  Baiduspider,Baidu Spider

  Example user agent
  Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)

• 8x detected (4x unique)
  Google Ads Bot,AdsBot-Google,AdsBot Google,Googlebot

  Example user agent
  AdsBot-Google ( http://www.google.com/adsbot.html)

• 2x detected (2x unique)
  Bing Preview,BingBot

  Example user agent
  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.11067 BingPreview/1.0b

• 2x detected (1x unique)
  GomezAgent

  Example user agent
  Mozilla/4.0 (compatible; MSIE 5.01; GomezAgent 2.0; Windows NT)

• 4x detected (2x unique)
  ArchiveBot,ArchiveTeam ArchiveBot

  Example user agent
  ArchiveTeam ArchiveBot/20140807.01 (wpull 0.37a1) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36

• 2x detected (1x unique)
  Data Hotel Watchdog

  Example user agent
  Data-Hotel-Watchdog/1.1

• 1x detected (1x unique)
  HeartRails Capture

  Example user agent
  Mozilla/5.0 (X11; U; Linux i686; xx; rv:1.9.2.18) Gecko/20110622 HeartRails_Capture/1.0.5 (+http://capture.heartrails.com/) Namoroka/3.6.18

• 3x detected (3x unique)
  GNIP,UnwindFetchor/1.0,UnwindFetchor

  Example user agent
  UnwindFetchor/1.0 (+http://www.gnip.com/)

• 6x detected (3x unique)
  Apache Nutch,Nutch,Nutch-based Bot

  Example user agent
  emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)

• 6x detected (3x unique)
  MJ12bot,MJ12 Bot,MJ12bot/v1.4.0

  Example user agent
  Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)

• 1x detected (1x unique)
  ADJUSTbot

  Example user agent
  Mozilla/5.0 (compatible; ADJUSTbot/2.0; +http://www.ad-just.jp/)

• 2x detected (1x unique)
  Summify

  Example user agent
  Summify (Summify/1.0.1; +http://summify.com)

• 0x detected (0x unique)

  Example user agent
  SearQuBot/SearQuBot v1.0

• 2x detected (1x unique)
  PostRank

  Example user agent
  PostRank/2.0 (postrank.com; 1 subscribers)

• 0x detected (0x unique)

  Example user agent
  Rome Client (http://tinyurl.com/64t5n)

• 3x detected (2x unique)
  Mapian News Bot,mapion-news-bot

  Example user agent
  "mapion-news-bot/1.0 (http://www.mapion.co.jp/news/)"

• 4x detected (1x unique)
  SimplePie

  Example user agent
  SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20111118194710

• 1x detected (1x unique)
  Rakutenbot

  Example user agent
  Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)

• 5x detected (2x unique)
  TweetmemeBot,Tweetmeme Bot

  Example user agent
  Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)

• 1x detected (1x unique)
  kizasi-spider

  Example user agent
  kizasi-spider/1.0 (+http://kizasi.jp/)

• 1x detected (1x unique)
  Feed Crawler

  Example user agent
  www2.apserver.net ASP-Ranker Feed Crawler

• 1x detected (1x unique)
  cloudforecastbot

  Example user agent
  cloudforecastbot

• 0x detected (0x unique)

  Example user agent
  Rainmeter WebParser plugin

About this comparison