| User Agent | | Date Added |
| AskTbWME |  | 31/01/2011 0:53:58 |
|
| AskTbWSV5 | | 5/09/2010 1:29:10 |
|
| AskTbX | | 25/05/2012 14:48:31 |
|
| AskTbX-SD | | 6/07/2010 20:23:08 |
|
| AskTbYLC | | 23/04/2010 1:20:20 |
|
| AskTbYSI2 | | 12/10/2012 11:43:23 |
|
| AskTbZMR | | 9/12/2010 16:37:16 |
|
| AskTbZTV | | 6/07/2010 19:53:32 |
|
| ASpider (Associative Spider) | | 10/02/2004 21:01:11 |
| ASpider is a CGI script that searches the web for keywords given by the user through a form. |
| Asterias | | 8/02/2004 19:38:37 |
| Singingfish uses an innovative, multimedia-oriented spider or robot to crawl the Web, looking for media files to index for the Singingfish Streambase. Our spider, called Asterias®, obeys spider Netiquette rules: It checks the Robot Exclusion File and it avoids hitting each site too frequently. Asterias® identifies multimedia (Real Media, Windows Media, Quicktime, MP3 and Flash) files and adds them to our index. Sometimes this process results in many sequential accesses to a Site. While this process results in a complete index, the behavior is atypical of other spiders found on the web. |
| ATN Worldwide | | 10/02/2004 23:52:53 |
| The ATN robot is used to build the database for the AllThatNet search service operated by All That Net. The robot runs weekly, and visits sites in a random order. |
| Atomz.com Search Robot | | 10/02/2004 23:54:53 |
| Robot used for web site search service. |
| Baiduspider | | 8/02/2004 15:21:41 |
|
| Baiduspider-image | | 21/12/2009 0:59:06 |
|
| BBot | | 11/02/2004 0:02:44 |
| Mainly intended for site level search, sometimes set loose. |
| BebopBot | | 1/12/2008 0:46:59 |
| BebopBot is an experimental web crawler under development since January 2007. BebopBot is owned and operated by A Passion for Jazz!™ as part of an effort to develop a niche based searchable Web index. We are not attempting to steal any copyrighted information from your site and will not be re-distributing your content. We will only be allowing users to find your website more easily. The information gathered will be indexed and made accessible via one or more publicly accessible web sites in the near future. |
| BecomeBot | | 17/01/2005 10:17:49 |
| BecomeBot is the user-agent for Become's new web crawler. Become is crawling the web to build a next generation search engine. |
| bingbot | | 29/09/2010 11:55:13 |
|
| boitho.com | | 5/01/2006 10:05:08 |
| The boitho.com-robot is a distributed crawler that downloads web pages to build the database used by boitho.com to search in. |
| Borg-Bot | | 11/02/2004 0:27:02 |
| Developmental crawler to feed a search engine |
| BSpider | | 11/02/2004 0:29:42 |
| BSpider is crawling inside of Japanese domain for indexing. |
| Businessjet | | 8/02/2004 15:22:06 |
|
| Calif | | 11/02/2004 16:04:50 |
| Used to build searchable index |
| Cassandra | | 11/02/2004 16:06:54 |
| Cassandra search robot is used to create and maintain indexed database for widespread Information Retrieval System |
| Charlotte | | 8/02/2004 15:23:24 |
|
| Charlotte Searchme | | 8/01/2008 16:36:23 |
|
| ChristCrawler.com | | 11/02/2004 16:14:07 |
| A Christian internet spider that searches web sites to find Christian Related material |
| cIeNcIaFiCcIoN.nEt | | 11/02/2004 16:17:52 |
| Robot encargado de la indexación de las páginas para www.cienciaficcion.net |
| CindooSpider | | 22/02/2009 23:52:09 |
|
| CipinetBot | | 8/02/2004 15:25:43 |
|
| Climate Change Spider | | 8/02/2004 15:26:47 |
|
| cmsworldmap.com | | 4/02/2011 12:39:31 |
Search Engine for CMS System from linkfactory.dk
fell for bad bot trap
|
| coccocbot-image | | 3/11/2025 11:39:24 |
|
| Collective | | 11/02/2004 16:21:03 |
Collective is the most cleverest Internet search engine, With all found urls guaranteed to have your search terms.
Collective is a highly configurable program designed to interrogate online search engines and online databases, it will ignore web pages that lie about there content, and dead url's, it can be super strict, it searches each web page it finds for your search terms to ensure those terms are present, any positive urls are added to a html file for your to view at any time even before the program has finished. Collective can wonder the web for days if required. |
| Conceptbot | | 11/02/2004 16:26:22 |
| The Conceptbot spider is used to research concept-based search indexing techniques. It uses a breadth first search to spread out the number of hits on a single site over time. The spider runs at irregular intervals and is still under construction. |
| CoolBot | | 11/02/2004 16:27:43 |
| The CoolBot robot is used to build and maintain the directory of the german search engine Suchmaschine21. |
| Cowbot | | 8/02/2004 15:27:55 |
|
| Crawlson | | 10/09/2020 14:28:39 |
|
| CrazyWebCrawler | | 24/09/2015 23:55:45 |
|
| Cusco | | 11/02/2004 16:42:30 |
| The Cusco robot is part of the CUCE indexing sistem. It gathers information from several sources: HTTP, Databases or filesystem. At this moment, it's universe is the .pt domain and the information it gathers is available at the Portuguese search engine Cusco http://www.cusco.pt/. |
| DeepIndex | | 8/02/2004 15:29:32 |
|
| Desert Realm Spider | | 8/02/2004 15:30:24 |
| The spider indexes fantasy and science fiction sites by using a customizable keyword algorithm. Only home pages are indexed, but all pages are looked at for links. Pages are visited randomly to limit impact on any one webserver. |
| DeuSu | | 9/10/2015 20:50:57 |
| DeuSu is my personal project. It is a search-engine which has its very own search-index. To create this search-index, over a billion web-pages have to be crawled. The DeuSu robot is the software used to do this. |
| Die Blinde Kuh | | 11/02/2004 0:10:22 |
| The robot is use for indexing and proofing the registered urls in the german language search engine for kids. Its a non-commercial one-woman-project of Birgit Bachmann living in Hamburg, Germany. |
| DIE-KRAEHE | | 8/02/2004 15:28:58 |
|
| DienstSpider | | 11/02/2004 16:54:43 |
| Indexing and searching the NCSTRL(Networked Computer Science Technical Report Library) and ERCIM Collection |
| Digger | | 11/02/2004 16:56:06 |
| indexing web sites for the Diggit! search engine |
| Digital Integrity Robot | | 11/02/2004 17:03:37 |
|
| Direct Hit Grabber | | 11/02/2004 17:04:55 |
| Direct Hit Grabber indexes documents and collects Web statistics for the Direct Hit Search Engine (available at www.directhit.com and our partners' sites) |
| DittoSpyder | | 8/09/2006 0:13:47 |
| Ditto image search engine |
