| User Agent | | Date Added |
| MerzScope |  | 24/07/2005 23:03:08 |
| Robot is part of a Web-Mapping package called MerzScope, to be used mainly by consultants, and web masters to create and publish maps, on and of the World wide web. |
| Mnogosearch | | 7/02/2004 23:13:37 |
| mnoGoSearch (formerly known as UdmSearch) is a full-featured web search engine software for intranet and internet servers. mnoGoSearch for UNIX is a free software covered by the GNU General Public License and mnoGoSearch for Windows is a commercial search software version. |
| Motor | | 25/07/2005 0:48:09 |
| The Motor robot is used to build the database for the www.webindex.de search service operated by CyberCon. The robot is under development - it runs in random intervals and visits site in a priority driven order (.de/.ch/.at first, root and robots.txt first) |
| MS Sharepoint Portal Server | | 7/02/2004 23:12:33 |
|
| MSNBot Media | | 13/06/2006 0:06:56 |
|
| Muncher | | 25/07/2005 0:50:11 |
| Used to build the index for www.goodlookingcooking.co.uk. Seeks out cooking and recipe pages. |
| Muscat Ferret | | 25/07/2005 0:54:52 |
| Used to build the database for the EuroFerret |
| Mwd.Search | | 25/07/2005 0:55:50 |
| Robot for indexing finnish (toplevel domain .fi) webpages for search engine called Fifi. Visits sites in random order. |
| NDSpider | | 25/07/2005 0:57:50 |
| It is designed to index the web. |
| NEC-MeshExplorer | | 24/07/2005 23:04:16 |
The NEC-MeshExplorer robot is used to build database for the NETPLAZA search service operated by NEC Corporation. The robot searches URLs around sites in japan(JP domain). The robot runs every day, and visits sites in a random order.
Prototype version of this robot was developed in C&C Research Laboratories, NEC Corporation. Current robot (Version 1.0) is based on the prototype and has more functions. |
| NetCarta WebMap Engine | | 25/07/2005 0:59:30 |
| The NetCarta WebMap Engine is a general purpose, commercial spider. Packaged with a full GUI in the CyberPilo Pro product, it acts as a personal spider to work with a browser to facilitiate context-based navigation. The WebMapper product uses the robot to manage a site (site copy, site diff, and extensive link management facilities). All versions can create publishable NetCarta WebMaps, which capture the crawled information. If the robot sees a published map, it will return the published map rather than continuing its crawl. Since this is a personal spider, it will be launched from multiple domains. This robot tends to focus on a particular site. No instance of the robot should have more than one outstanding request out to any given site at a time. The User-agent field contains a coded ID identifying the instance of the spider; specific users can be blocked via robots.txt using this ID. |
| NetResearchServer | | 8/02/2004 16:31:37 |
NRS crawls pages all over the world in order to build full-text
search indexes and/or to compile lists of search engine forms. |
| NetScoop | | 25/07/2005 1:01:18 |
The NetScoop robot is used to build the database for the NetScoop search engine.
The robot has been used in the research project at the Faculty of Engineering, Tokushima University, Japan., since Dec. 1996. |
| newscan-online | | 25/07/2005 1:02:31 |
The newscan-online robot is used to build a database for the newscan-online news search service operated by smart information services. The robot runs daily and visits predefined sites in a random order.
This robot finds its roots in a prereleased software for news filtering for Lotus Notes in 1995. |
| NextopiaBOT | | 7/02/2004 23:16:00 |
|
| NHSE Web Forager | | 25/07/2005 1:03:15 |
| to generate a Resource Discovery database |
| Nomad | | 25/07/2005 1:04:17 |
| Developed in 1995 at Colorado State University. |
| NutchCVS | | 7/02/2004 23:18:12 |
| When we crawl to populate our index, we advertise the "User-agent" string "NutchOrg". If you see the agent "Nutch" or "NutchCVS", that's probably a developer testing a new version of our robot, or someone running their own instance. |
| Occam | | 25/07/2005 1:08:07 |
The robot takes high-level queries, breaks them down into multiple web requests, and answers them by combining disparate data gathered in one minute from numerous web sites, or from the robots cache.
The robot is a descendant of Rodney, an earlier project at the University of Washington. |
| omgilibot | | 31/03/2008 17:10:34 |
| crawls forums |
| OpenIntelligenceData | | 3/09/2005 17:15:02 |
| Open Intelligence Data ™ is a project by Tortuga Group LLC to provide free tools for collecting information for millions of Internet domains. |
| Oracle Ultra Search | | 7/02/2004 23:33:14 |
| Ultra Search can be used to search across Collaboration Suite Components, corporate Web servers, databases, mail servers, fileservers and Oracle10g Portal instances. |
| Orb Search | | 25/07/2005 1:12:12 |
| Orbsearch builds the database for Orb Search Engine. It runs when requested. |
| Origin | | 7/02/2004 23:40:28 |
| Empty user agent |
| PageBoy | | 25/07/2005 1:14:43 |
| The robot visits at regular intervals. |
| Panoptic | | 7/02/2004 23:15:22 |
| Panoptic is a new generation search engine offering very high quality results. It offers a unique combination of metadata and full text indexing from a variety of sources and does a great job of finding home pages. It can support your web site, portal, e-commerce and customer service initiatives. |
| ParaSite | | 7/02/2004 23:39:09 |
| ParaSite is an incredibly powerful spider which went through several different versions over the course of two years. It is designed to index a substantial portion of the web quickly. ParaSite runs using a server and multiple downloaders. Each downloader runs a number of threads, capable of indexing five to ten documents per second. Since this is a parallel implementation, multiple downloaders can be run simultaneously. The server sorts the incoming urls into queues and hands of batches of urls to the downloaders for indexing. |
| pegasus | | 25/07/2005 1:18:21 |
pegasus gathers information from HTML pages (7 important tags). The indexing process can be started based on starting URL(s) or a range of IP address.
This robot was created as an implementation of a final project on Informatics Engineering Department, Institute of Technology Bandung, Indonesia. |
| PerlCrawler | | 25/07/2005 1:22:03 |
| The PerlCrawler robot is designed to index and build a database of pages relating to the Perl programming language. |
| PGP Key Agent | | 25/07/2005 1:29:29 |
This program search the pgp public key for the specified user.
Originated as a research project at Salerno University in 1995. |
| Phantom | | 25/07/2005 1:22:53 |
| Designed to allow webmasters to provide a searchable index of their own site as well as to other sites, perhaps with similar content. |
| PhpDig | | 25/07/2005 1:23:47 |
| Small robot and search engine written in php. |
| Picsearch | | 8/02/2004 19:52:18 |
| Picsearch is indexing pictures from the web. To do this we use a web-crawler which identifies itself as 'Psbot'. |
| Pimptrain.com's robot | | 25/07/2005 1:26:02 |
| Crawls remote sites as part of a search engine program |
| Pioneer | | 25/07/2005 1:26:45 |
| Pioneer is part of an undergraduate research project. |
| pipeLiner | | 10/11/2004 12:33:34 |
|
| PlumtreeWebAccessor | | 25/07/2005 1:35:23 |
| The Plumtree Web Accessor is a component that customers can add to the Plumtree Server to index documents on the World Wide Web. |
| Poppi | | 27/07/2005 22:54:07 |
Poppi is a crawler to index the web that runs weekly gathering and indexing hypertextual, multimedia and executable file formats.
Created by Antonio Provenzano in the april of 2000, has been acquired from Tomi Officine Multimediali srl and it is next to release as service and commercial. |
| Portal Juice Spider | | 25/07/2005 1:27:47 |
Indexing web documents for Portal Juice vertical portal search engine
Indexing the web since 1998 for the purposes of offering our commerical Portal Juice search engine services. |
| PortalB Spider | | 27/07/2005 22:55:04 |
| The PortalB Spider indexes selected sites for high-quality business information. |
| Project XP5 | | 7/02/2004 23:37:56 |
|
| proximic | | 9/12/2009 0:50:47 |
|
| Raven Search | | 27/07/2005 23:00:25 |
| Raven was written for the express purpose of indexing the web. It can parallel process hundreds of URLS's at a time. It runs on a sporadic basis as testing continues. It is really several programs running concurrently. It takes four computers to run Raven Search. Scalable in sets of four. |
| RBSE Spider | | 27/07/2005 23:01:45 |
| Developed and operated as part of the NASA-funded Repository Based Software Engineering Program at the Research Institute for Computing and Information Systems, University of Houston - Clear Lake. |
| RDSIndexer | | 7/02/2004 23:44:14 |
| Information Resource Management Tool/Web Portal |
| Resume Robot | | 27/07/2005 23:02:38 |
|
| Road Runner: The ImageScape Robot | | 27/07/2005 23:23:18 |
|
| Robbie the Robot | | 27/07/2005 23:24:37 |
Used to define document collections for the DISCO system. Robbie is still under development and runs several times a day, but usually only for ten minutes or so. Sites are visited in the order in which references are found, but no host is visited more than once in any two-minute period.
The DISCO system is a resource-discovery component in the OLLA system, which is a prototype system, developed under DARPA funding, to support computer-based education and training. |
| RoboCrawl Spider | | 27/07/2005 23:26:43 |
The Canadian Content robot indexes for it's search database.
Our robot is a newer project at Canadian Content. |
| Robot Francoroute | | 11/02/2004 23:17:13 |
| Part of the RISQ's Francoroute project for researching francophone. Uses the Accept-Language tag and reduces demand accordingly |
