| User Agent | | Date Added |
| AboutUsBot |  | 29/06/2007 23:34:04 |
| The AboutUs:Bot gathers descriptive information about a website from several sources to build a Wiki Page. This pre-built wiki page gives website owners and AboutUs.org contributors a head-start in creating a useful and informative AboutUs.org page. |
| ADSARobot | | 5/02/2004 20:50:18 |
|
| AhrefsBot | | 29/08/2011 11:10:48 |
| Ahrefs.com provides different services based on backlinks analysis. |
| Atlas | | 11/02/2004 0:25:57 |
| This robot will transverse a web site and map every web page found. It will index them in a coma separate file. (.csv) Atlas will construct a site map of your web site, logging all web pages and locations. This will enable you to analyze the site and monitor size. |
| CCBot | | 1/12/2008 1:14:23 |
|
| CommonCrawler | | 14/12/2015 13:51:45 |
|
| Computer_and_Automation_Research_Institute_Crawler | | 5/02/2004 20:50:38 |
|
| DepSpid | | 5/11/2006 22:43:42 |
| DepSpid is a distributed kind of a web crawler. The DepSpid spider visits domains, analyses links and finally calculates scores about the link dependencies between individual domains. Each spider job starts at the main page of a domain and then follows each link on that page retrieving more pages and analysing them, too. The spider stays within one domain. If it finds an external link it only checks if the linked domain is reachable but doesn't continue crawling into the external domain. Every unknown domain will be visited from another spider job at a later time. |
| DNS-Digger | | 29/06/2007 23:30:42 |
| DNS-Digger.com is a utility that lists what other domain a host shares DNS-server with. |
| DomainMacroCrawler | | 23/10/2015 10:33:14 |
|
| EbiNess | | 11/02/2004 17:14:36 |
| Used to build a url relationship database, to be viewed in 3D |
| e-SocietyRobot | | 27/04/2006 0:14:47 |
The research project "Technologies for the Knowledge Discovery from the Internet" is one of the year 2003 leading projects of Ministry of Education, Culture, Sports, Science and Technology, Japan. The project contractor is Waseda university. The project goal is to gather all the Web pages in the world efficiently and to apply data mining to the gathered Web pages to discover the Knowledge. The following is the detailed sub goals.
1. R&D on new Web page crawler
　Gather all the Web pages (about 12 billion pages) and keep their freshness in one month in average.
2. R&D on discovering the knowledge
　Discover the knowledge that the user want to know. |
| hosterstats.com | | 17/12/2015 11:30:14 |
|
| HTTP Header Survey | | 23/03/2015 9:04:14 |
|
| HTTP Header Survey By Benjojo | | 6/02/2014 15:17:29 |
From Benjojo:
This is a little service I am developing in my free time to keep a eye on the status headers of all the .com domain names.
Because I get daily updates from Verisign about the .com domain file I request them and store the headers they send back. |
| IceCat | | 13/12/2009 13:13:18 |
|
| InfoSpiders | | 8/03/2004 0:09:51 |
| application of artificial life algorithm to adaptive distributed information retrieval |
| Kilroy | | 8/03/2004 0:55:58 |
| Used to collect data for several projects. Runs constantly and visits site no faster than once every 90 seconds. |
| knowledge | | 12/09/2006 0:36:05 |
|
| MediaFox | | 24/07/2005 23:01:53 |
| The robot is used to index meta information of a specified set of documents and update a database accordingly. |
| MetaGeneratorCrawler | | 14/10/2013 11:01:17 |
| Data mining meta="generator" tags |
| NetCarta WebMap Engine | | 25/07/2005 0:59:30 |
| The NetCarta WebMap Engine is a general purpose, commercial spider. Packaged with a full GUI in the CyberPilo Pro product, it acts as a personal spider to work with a browser to facilitiate context-based navigation. The WebMapper product uses the robot to manage a site (site copy, site diff, and extensive link management facilities). All versions can create publishable NetCarta WebMaps, which capture the crawled information. If the robot sees a published map, it will return the published map rather than continuing its crawl. Since this is a personal spider, it will be launched from multiple domains. This robot tends to focus on a particular site. No instance of the robot should have more than one outstanding request out to any given site at a time. The User-agent field contains a coded ID identifying the instance of the spider; specific users can be blocked via robots.txt using this ID. |
| NetTrack Anonymous Web Statistics | | 5/10/2016 23:42:25 |
|
| nominet.org.uk | | 25/09/2015 10:31:26 |
|
| OntoSpider | | 25/07/2005 1:10:05 |
| Focused crawler for research purposes |
| OpenIntelligenceData | | 3/09/2005 17:15:02 |
| Open Intelligence Data ™ is a project by Tortuga Group LLC to provide free tools for collecting information for millions of Internet domains. |
| Pack Rat | | 25/07/2005 1:13:56 |
| Used for local maintenance and for gathering web pages so that local statisistical info can be used in artificial intelligence programs. Funded by NEMOnline. |
| PHP version tracker | | 9/09/2005 18:00:05 |
This is the bot used to produce Nexen.net's PHP web statistics. You may find one of the produced stats here.
PHP version tracker has a 20 millions domain name database that he scans every month. For each web site, it retrieve the headers from the main page, using the HEAD HTTP command. Then, from those headers, it gathers information about PHP adoption in the world, and some other stats.
|
| Pioneer | | 25/07/2005 1:26:45 |
| Pioneer is part of an undergraduate research project. |
| Pita | | 5/02/2004 20:54:44 |
|
| PolyBot | | 5/02/2004 20:55:20 |
| Polybot is a part of an academic research project that aims to improve search and analysis techniques for the World Wide Web. We crawl data for academic/research purposes ONLY. |
| Popular Iconoclast | | 7/03/2004 23:50:18 |
| statistics |
| RBSE Spider | | 27/07/2005 23:01:45 |
| Developed and operated as part of the NASA-funded Repository Based Software Engineering Program at the Research Institute for Computing and Information Systems, University of Houston - Clear Lake. |
| Riddler | | 9/05/2014 8:37:36 |
| Riddler is an online research project which investigates algorithms for mapping the topology of the Internet. Riddler collects data about public systems via crawling and port mapping common ports. |
| Scrapy | | 23/05/2012 14:41:28 |
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
fell for bad bot trap!
|
| servernfo.com | | 7/03/2016 9:52:21 |
|
| Shim-Crawler | | 6/02/2006 15:38:26 |
| Shim-crawler was written by Shim Wonbo of Chikayama-Taura laboratory.The main goal behind writing the crawler is to collect web pages for researches related to web-search and data mining. Recently, we are planning to use it for crawling weblogs too.The Crawler is used by the members of Chikayama-Taura Laboratory to crawl web-pages only for the research purposes.Our crawling policy distinctly respects the general crawling norm.Though we duely understand the concern of the webmasters, we would like to assure that our crawler is only crawling pages for performing researches and not for any business use.Please have a glance at our crawling policy for better understanding.We sincerely appriciate your co-operation and support. |
| Snappy | | 5/11/2006 22:54:45 |
|
| special_archiver | | 7/09/2011 14:24:41 |
bypasses robots.txt and we did not receive an e-mail to notice about the crawl as they say they do on their site!
Library of Congress Web Archives |
| Steeler | | 7/06/2004 11:46:38 |
| Steeler is a web crawler (aka robot), software to surf the web automatically. It is being developed and operated at Kitsuregawa Laboratory, The University of Tokyo. We are working on analysis and understanding of the nature of cyberspace based on the documents collected through the surfing. |
| vURL Online | | 18/12/2015 13:58:10 |
|
| Wappalyzer | | 17/12/2016 23:19:39 |
|
| Web Core / Roots | | 11/02/2004 16:38:26 |
| Parallel robot developed in Minho University in Portugal to catalog relations among URLs and to support a special navigation aid. |
| WebStatsChecker.com | | 25/09/2015 11:53:50 |
Referrer:
http://www.webstatschecker.com/stats/domain/yourdomainhere.com
WebStatsChecker.com is a free service, allowing website owners and Search Engine Optimization specialists to:
Retrieve Stats and Search Engine Performance for their own and competitor domains.
Retrieve ranking information for a specific keyword, and then view and compare the competitor's Google PageRank, Yahoo Backlinks, website title, and many more.
Do keyword research, using our unique keyword research engine, that will find similar keywords, all used by the same domains.
|
| WebVac | | 5/02/2004 20:56:55 |
|
| Wild Ferret Web Hopper #1, #2, #3 | | 11/02/2004 22:57:48 |
| The wild ferret web hopper's are designed as specific agents to retrieve data from all available sources on the internet. They work in an onion format hopping from spot to spot one level at a time over the internet. The information is gathered into different relational databases, known as "Hazel's Horde". The information is publicly available and will be free for the browsing at www.greenearth.com. Effective date of the data posting is to be announced. |
| Zao | | 5/02/2004 20:57:23 |
|
