| User Agent | | Date Added |
| ACHE |  | 12/07/2016 6:34:40 |
| ACHE is a web crawler for domain-specific search |
| admantx-euaspb | | 29/05/2018 17:41:31 |
The ADmantX Semantic Analysis Service provides real-time analysis of the context of the article or text on a web page. The ADmantX SA Service utilizes Natural Language Processing (NLP) technology to analyze text the way a human does when reading. The output of the ADmantX SA Service is the “context” (automatic categorization) of the web page. The ADmantX SA Service then provides the “context” output to the requesting client (our customers and partners).
Our customers are web site publishers who use our Semantic Analysis Service to provide “contextual ad targeting” as well as provide brand safety for advertisers (blocking ads on inappropriate or questionable content). |
| AdvertDex | | 15/11/2005 0:54:39 |
| Advertising Web Directory |
| AlkalineBOT | | 7/02/2004 23:04:26 |
|
| ananzi | | 11/02/2004 22:43:44 |
|
| ASpider (Associative Spider) | | 10/02/2004 21:01:11 |
| ASpider is a CGI script that searches the web for keywords given by the user through a form. |
| ASPseek | | 7/02/2004 23:03:13 |
| ASPseek is an Internet search engine software developed by SWsoft and licensed as free software under GNU GPL. |
| AURESYS | | 10/02/2004 23:56:08 |
| The AURESYS is used to build a personnal database for somebody who search information. The database is structured to be analysed. AURESYS can found new server by IP incremental. It generate statistics... |
| Blackboard Safeassign | | 8/04/2016 21:54:08 |
|
| BLEXBot | | 27/06/2013 10:21:15 |
SEO
fell for bad bot trap! |
| Building-X | | 30/04/2006 0:54:50 |
| Building and Construction Web Directory |
| BullsEye | | 7/09/2006 23:53:37 |
| Intelliseek robot |
| CatchBot | | 9/09/2008 13:55:03 |
CatchBot is the web crawler for Catch, the online division of Reed Business Information Australia. Reed Business Information is Australia’s leading and largest business to business publisher and information provider.
CatchBot investigates websites for publicly available information about companies, such as a company’s name, address, telephone number and keyword data about a company’s products and services. CatchBot is not designed to access or index any personal information or any information about individuals.
Information gathered by CatchBot is stored on our password protected servers and the security of this information is of the highest importance to us.
Information gathered by CatchBot may be used for business activities that are undertaken by Catch. Examples of this include publishing and maintaining business directories in various countries around the world, industry specific websites and online portals |
| charlotte betaspider.com | | 5/11/2006 22:51:16 |
We are a stealth-mode startup that is indexing the web for a novel application. We plan to release this new service to the public very soon.
We are not attempting to steal any copyrighted information from your site and will not be re-distributing your content. We will only be allowing users to find for your website more easily. |
| CloudServerMarketSpider | | 6/02/2014 8:50:29 |
| CloudServerMarketSpider is Cloud Server Market's web crawling spider. Crawling is the process by which CloudServerMarketSpider discovers new and updated pages to be added to the Cloud Server Market index. |
| CMC/0.01 | | 11/02/2004 16:16:32 |
| This CMC/0.01 robot collects the information of the page that was registered to the music specialty searching service. |
| Combine | | 7/02/2004 23:05:39 |
| Combine is an open system for harvesting and threshing (indexing) Internet resources. |
| ComputingSite Robi/1.0 | | 27/07/2005 23:25:37 |
Intelligent agent used to build the ComputingSite Search Directory.
It was born on August 1997. |
| Conceptbot | | 11/02/2004 16:26:22 |
| The Conceptbot spider is used to research concept-based search indexing techniques. It uses a breadth first search to spread out the number of hits on a single site over time. The spider runs at irregular intervals and is still under construction. |
| Contacts Crawler | | 24/11/2016 9:07:21 |
Scrapy Cloud, our cloud-based web crawling platform, allows you to easily deploy crawlers and scale them on demand – without needing to worry about servers, monitoring, backups, or cron jobs. It helps developers like you turn over two billion web pages per month into valuable data.
Our platform's many add-ons let you extend your spiders in clicks. Among them, our smart proxy rotator (Crawlera) helps you bypass bot counter-measures so you can crawl large sites faster.
Your data gets safely stored in a high-availability database. You can browse it and share it with your team from your dashboard, or consume your data in your app using our API. |
| DeWeb(c) Katalog/Index | | 11/02/2004 16:53:41 |
| Its purpose is to generate a Resource Discovery database, perform mirroring, and generate statistics. Uses combination of Informix(tm) Database and WN 1.11 serversoftware for indexing/ressource discovery, fulltext search, text excerpts. |
| DNAbot | | 11/02/2004 17:07:07 |
| A search robot in 100 java, with its own built-in database engine and web server . Currently in Japanese. |
| DocSpotBot | | 19/09/2017 16:23:43 |
| DocSpotBot is DocSpot's software to crawl pages to help people find out more about doctors and other medical providers. Provider profiles link back to the sources of the information. |
| elefent | | 30/11/2017 9:27:32 |
| ELEFENT is a web crawler providing data for a number of publicly accessible web search engines. You can freely search data indexed by Elefent here. |
| Emacs-w3 Search Engine | | 11/02/2004 22:42:30 |
| Its purpose is to generate a Resource Discovery database This code has not been looked at in a while, but will be spruced up for the Emacs-w3 2.2.0 release sometime this month. It will honor the /robots.txt file at that time. |
| Enterprise_Search | | 7/02/2004 23:08:44 |
|
| ES.NET | | 5/07/2005 11:02:08 |
| Innerprise develops full-text indexing search engine software technology enabling search for your Web site, Intranet, or the Web. Advanced crawler features ensure that only documents you want indexed are indexed. Key features provide support for common file types, secure servers, multiple servers, and complete automation through built-in schedulers. |
| FAST Enterprise Crawler | | 18/03/2004 21:41:19 |
|
| Felix IDE | | 11/02/2004 22:55:39 |
| Felix IDE is a retail personal search spider sold by The Pentone Group, Inc. It supports the proprietary exclusion "Frequency: ??????????" in the robots.txt file. Question marks represent an integer indicating number of milliseconds to delay between document requests. This is called VDRF(tm) or Variable Document Retrieval Frequency. Note that users can re-define the useragent name. |
| FetchRover | | 11/02/2004 23:00:32 |
| FetchRover fetches Web Pages. It is an automated page-fetching engine. FetchRover can be used stand-alone or as the front-end to a full-featured Spider. Its database can use any ODBC compliant database server, including Microsoft Access, Oracle, Sybase SQL Server, FoxPro, etc. |
| Fish search | | 11/02/2004 23:13:23 |
| Its purpose is to discover resources on the fly a version exists that is integrated into the Tübingen Mosaic 2.4.2 browser (also written in C) |
| Fluid Dynamics Search Engine robot (FDSE) | | 7/02/2004 23:10:47 |
| FDSE is an easy-to-install search engine for local and remote sites. It returns fast, accurate results from a template-driven architecture. |
| gammaSpider/FocusedCrawler | | 11/02/2004 23:23:05 |
| Information gathering. Focused crawling on specific topic. Uses gammaFetcherServer Product for selling. RobotUserAgent may changed by the user. More features are being added. The product is constatnly under development. AKA FocusedCrawler |
| GastroGnome | | 29/01/2018 1:43:26 |
|
| GetBot | | 12/02/2004 20:44:16 |
| GetBot's purpose is to index all the sites it can find that contain Shockwave movies. It is the first bot or spider written in Shockwave. The bot was originally written at Macromedia on a hungover Sunday as a proof of concept. - Alex Zavatone 3/29/96 |
| Google Search Appliance | | 8/02/2004 0:02:23 |
|
| Googlebot-Image | | 8/04/2004 22:35:33 |
|
| Grapnel/0.01 Experiment | | 12/02/2004 20:53:07 |
| Resource Discovery Experimentation |
| Grub | | 8/02/2004 0:00:25 |
| Leveraging the power of distributed computing, Grub allows everyone with an Internet connection to participate in the last frontier of discovery. By downloading the unique screensaver, you can donate your computer's unused bandwidth to probing the hidden depths of the Web. |
| havIndex | | 14/02/2004 0:15:57 |
| havIndex allows individuals to build searchable word index of (user specified) lists of URLs. havIndex does not crawl - rather it requires one or more user supplied lists of URLs to be indexed. havIndex does (optionally) save urls parsed from indexed pages. |
| Heritrix | | 8/02/2004 0:05:08 |
| Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. |
| HKU WWW Octopus | | 25/07/2005 1:08:54 |
| HKU Octopus is an ongoing project for resource discovery in the Hong Kong and China WWW domain . It is a research project conducted by three undergraduate at the University of Hong Kong |
| ht://Dig | | 14/02/2004 0:24:00 |
The ht://Dig system is a complete world wide web indexing and searching system for a domain or intranet. This system is not meant to replace the need for powerful internet-wide search systems like Lycos, Infoseek, Google and AltaVista. Instead it is meant to cover the search needs for a single company, campus, or even a particular sub section of a web site.
As opposed to some WAIS-based or web-server based search engines, ht://Dig can easily span several web servers. The type of these different web servers doesn't matter as long as they understand common protocols like HTTP. |
| Hyper-Decontextualizer | | 14/02/2004 0:27:03 |
| Perl 5 Takes an input sentence and marks up each word with an appropriate hyper-text link. |
| IBM_Planetwide | | 7/03/2004 23:48:26 |
| Restricted to IBM owned or related domains. |
| image.kapsi.net | | 8/03/2004 0:50:22 |
The image.kapsi.net robot is used to build the database for the image.kapsi.net search service. The robot runs currently in a random times.
The Robot was build for image.kapsi.net's database in year 2001. |
| Imagelock | | 7/03/2004 23:54:21 |
| searches for image links |
| Ingrid | | 7/03/2004 23:52:52 |
| Commercial as part of search engine package |
| InnerpriseBot | | 25/07/2005 3:14:07 |
| Innerprise develops full-text indexing search engine software technology enabling search for your Web site, Intranet, or the Web. Advanced crawler features ensure that only documents you want indexed are indexed. Key features provide support for common file types, secure servers, multiple servers, and complete automation through built-in schedulers. |
| IXE Crawler | | 7/02/2004 23:11:02 |
|
