| User Agent | | Verified | Date Added |
| AboutUsBot | [edit] | | 29/06/2007 23:34:04 |
| The AboutUs:Bot gathers descriptive information about a website from several sources to build a Wiki Page. This pre-built wiki page gives website owners and AboutUs.org contributors a head-start in creating a useful and informative AboutUs.org page. |
| ADSARobot | [edit] | | 5/02/2004 20:50:18 |
|
| Atlas | [edit] | | 11/02/2004 0:25:57 |
| This robot will transverse a web site and map every web page found. It will index them in a coma separate file. (.csv) Atlas will construct a site map of your web site, logging all web pages and locations. This will enable you to analyze the site and monitor size. |
| Computer_and_Automation_Research_Institute_Crawler | [edit] | | 5/02/2004 20:50:38 |
|
| DepSpid | [edit] | | 5/11/2006 22:43:42 |
| DepSpid is a distributed kind of a web crawler. The DepSpid spider visits domains, analyses links and finally calculates scores about the link dependencies between individual domains. Each spider job starts at the main page of a domain and then follows each link on that page retrieving more pages and analysing them, too. The spider stays within one domain. If it finds an external link it only checks if the linked domain is reachable but doesn't continue crawling into the external domain. Every unknown domain will be visited from another spider job at a later time. |
| DNS-Digger | [edit] | | 29/06/2007 23:30:42 |
| DNS-Digger.com is a utility that lists what other domain a host shares DNS-server with. |
| EbiNess | [edit] | Yes | 11/02/2004 17:14:36 |
| Used to build a url relationship database, to be viewed in 3D |
| e-SocietyRobot | [edit] | Yes | 27/04/2006 0:14:47 |
The research project "Technologies for the Knowledge Discovery from the Internet" is one of the year 2003 leading projects of Ministry of Education, Culture, Sports, Science and Technology, Japan. The project contractor is Waseda university. The project goal is to gather all the Web pages in the world efficiently and to apply data mining to the gathered Web pages to discover the Knowledge. The following is the detailed sub goals.
1. R&D on new Web page crawler
　Gather all the Web pages (about 12 billion pages) and keep their freshness in one month in average.
2. R&D on discovering the knowledge
　Discover the knowledge that the user want to know. |
| IceCat | [edit] | | 13/12/2009 13:13:18 |
|
| InfoSpiders | [edit] | | 8/03/2004 0:09:51 |
| application of artificial life algorithm to adaptive distributed information retrieval |
| Kilroy | [edit] | | 8/03/2004 0:55:58 |
| Used to collect data for several projects. Runs constantly and visits site no faster than once every 90 seconds. |
| knowledge | [edit] | | 12/09/2006 0:36:05 |
|
| MediaFox | [edit] | | 24/07/2005 23:01:53 |
| The robot is used to index meta information of a specified set of documents and update a database accordingly. |
| NetCarta WebMap Engine | [edit] | | 25/07/2005 0:59:30 |
| The NetCarta WebMap Engine is a general purpose, commercial spider. Packaged with a full GUI in the CyberPilo Pro product, it acts as a personal spider to work with a browser to facilitiate context-based navigation. The WebMapper product uses the robot to manage a site (site copy, site diff, and extensive link management facilities). All versions can create publishable NetCarta WebMaps, which capture the crawled information. If the robot sees a published map, it will return the published map rather than continuing its crawl. Since this is a personal spider, it will be launched from multiple domains. This robot tends to focus on a particular site. No instance of the robot should have more than one outstanding request out to any given site at a time. The User-agent field contains a coded ID identifying the instance of the spider; specific users can be blocked via robots.txt using this ID. |
| OntoSpider | [edit] | | 25/07/2005 1:10:05 |
| Focused crawler for research purposes |
| OpenIntelligenceData | [edit] | | 3/09/2005 17:15:02 |
| Open Intelligence Data ™ is a project by Tortuga Group LLC to provide free tools for collecting information for millions of Internet domains. |
| Pack Rat | [edit] | | 25/07/2005 1:13:56 |
| Used for local maintenance and for gathering web pages so that local statisistical info can be used in artificial intelligence programs. Funded by NEMOnline. |
| PHP version tracker | [edit] | Yes | 9/09/2005 18:00:05 |
This is the bot used to produce Nexen.net's PHP web statistics. You may find one of the produced stats here.
PHP version tracker has a 20 millions domain name database that he scans every month. For each web site, it retrieve the headers from the main page, using the HEAD HTTP command. Then, from those headers, it gathers information about PHP adoption in the world, and some other stats.
|
| Pioneer | [edit] | | 25/07/2005 1:26:45 |
| Pioneer is part of an undergraduate research project. |
| Pita | [edit] | | 5/02/2004 20:54:44 |
|
| PolyBot | [edit] | Yes | 5/02/2004 20:55:20 |
| Polybot is a part of an academic research project that aims to improve search and analysis techniques for the World Wide Web. We crawl data for academic/research purposes ONLY. |
| Popular Iconoclast | [edit] | | 7/03/2004 23:50:18 |
| statistics |
| RBSE Spider | [edit] | | 27/07/2005 23:01:45 |
| Developed and operated as part of the NASA-funded Repository Based Software Engineering Program at the Research Institute for Computing and Information Systems, University of Houston - Clear Lake. |
| Shim-Crawler | [edit] | Yes | 6/02/2006 15:38:26 |
| Shim-crawler was written by Shim Wonbo of Chikayama-Taura laboratory.The main goal behind writing the crawler is to collect web pages for researches related to web-search and data mining. Recently, we are planning to use it for crawling weblogs too.The Crawler is used by the members of Chikayama-Taura Laboratory to crawl web-pages only for the research purposes.Our crawling policy distinctly respects the general crawling norm.Though we duely understand the concern of the webmasters, we would like to assure that our crawler is only crawling pages for performing researches and not for any business use.Please have a glance at our crawling policy for better understanding.We sincerely appriciate your co-operation and support. |
