| User Agent | | Date Added |
| AhrefsBot |  | 29/08/2011 11:10:48 |
| Ahrefs.com provides different services based on backlinks analysis. |
| aiHitBot | | 14/01/2010 2:42:44 |
| aiHit has created and maintains a database of over 18m company profiles from across the world and new profiles are being added at the rate of over half-a-million per month. Our engine automatically identifies corporate and educational web-sites and extracts structured data from these sites to create a profile including a description of the organisation, the names and titles of key executives, contact information, clients, investors etc. For UK Limited companies, our web profiles are consolidated with Companies House profiles. These profiles are regularly updated each time our engine revisits the sites. Because we keep every historic profile, our database makes it possible for users to see how an organisation has changed over time. aiHit is unique in providing a structured history of changes to organisations, based on the historic snapshots of their web-site. Our database is used by academics, corporate researchers, business and market analysts, credit scoring companies and more. |
| Almaden | | 5/02/2004 18:49:27 |
| Harnessing WebFountain's power will help enterprises gain insightful, highly synthesized, timely, and customized information that is not readily perceptible or available today. This includes information such as emerging trends and patterns, competitive activities, “buzz” about products, relationships affecting customers’ businesses, and pathways to discovery. |
| Arachnophilia | | 10/02/2004 20:42:21 |
| The purpose (undertaken by HaL Software) of this run was to collect approximately 10k html documents for testing automatic abstract generation |
| Barkrowler | | 26/08/2017 10:00:51 |
|
| BDCbot | | 10/01/2014 10:02:00 |
|
| CACTVS Chemistry Spider | | 11/02/2004 0:31:27 |
| Locates chemical structures in Chemical MIME formats on WWW and FTP servers and downloads them into database searchable with structure queries (substructure, fullstructure, formula, properties etc.) |
| Cerfinfo | | 5/02/2004 20:32:25 |
| CERFinfo.com is a dynamic directory of tens of thousands of carefully selected, information-rich, safe K-12 websites |
| CJNetworkQuality | | 5/02/2004 20:31:11 |
| The network quality utility tool searches each publisher Web site that is registered in the Commission Junction network that generates traffic to monitor compliance to the Publisher Service Agreement, specifically, Sections 1 and/or 2.2. |
| Conceptbot | | 11/02/2004 16:26:22 |
| The Conceptbot spider is used to research concept-based search indexing techniques. It uses a breadth first search to spread out the number of hits on a single site over time. The spider runs at irregular intervals and is still under construction. |
| Contacts Crawler | | 24/11/2016 9:07:21 |
Scrapy Cloud, our cloud-based web crawling platform, allows you to easily deploy crawlers and scale them on demand – without needing to worry about servers, monitoring, backups, or cron jobs. It helps developers like you turn over two billion web pages per month into valuable data.
Our platform's many add-ons let you extend your spiders in clicks. Among them, our smart proxy rotator (Crawlera) helps you bypass bot counter-measures so you can crawl large sites faster.
Your data gets safely stored in a high-availability database. You can browse it and share it with your team from your dashboard, or consume your data in your app using our API. |
| Contacts-Crawler | | 2/08/2017 11:27:27 |
|
| Covario-IDS | | 3/10/2011 15:17:59 |
| Covario IDS is a proprietary crawler used as part of Covario's Organic Search Insight solution. Covario helps global organizations manage and measure their SEO opportunities through automated on-page audits and external link analysis. Part of this external link analysis depends upon Covario IDS to help call out external pages that are relevant to the same topics in which SEO Marketers are trying to build links. |
| DataMiner | | 8/04/2017 8:58:48 |
|
| DeWeb(c) Katalog/Index | | 11/02/2004 16:53:41 |
| Its purpose is to generate a Resource Discovery database, perform mirroring, and generate statistics. Uses combination of Informix(tm) Database and WN 1.11 serversoftware for indexing/ressource discovery, fulltext search, text excerpts. |
| DomainSigmaCrawler | | 31/10/2014 13:00:30 |
| This crawler is run by DomainSigma to provide data about various domains on our website. If our crawler is causing your problem, then please contact us using our Contacts page. |
| EdisterBot | | 1/03/2012 11:18:45 |
EdisterBot is the web crawler for Edister.com. A "crawler" is simply software that fetches web pages, records details about those pages, and examines them for links to other web pages -- which it then fetches and examines, etc.
fell for bad bot trap |
| electricmonk | | 29/10/2016 10:59:38 |
|
| FunnelWeb | | 11/02/2004 23:19:50 |
| Its purpose is to generate a Resource Discovery database, and generate statistics. Localised South Pacific Discovery and Search Engine, plus distributed operation under development. |
| GCreep | | 12/02/2004 20:42:55 |
| Indexing robot to learn SQL |
| IRLbot | | 2/05/2006 0:42:29 |
| IRL-crawler is a Texas A&M research project sponsored in part by the National Science Foundation that investigates algorithms for mapping the topology of the Internet and discovering the various parts of the web. The crawler downloads random web pages (text only) and follows certain links to find other websites. |
| IssueCrawler | | 5/02/2004 20:36:25 |
|
| IUSA Browser | | 5/02/2004 20:33:57 |
|
| Kilroy | | 8/03/2004 0:55:58 |
| Used to collect data for several projects. Runs constantly and visits site no faster than once every 90 seconds. |
| knowledge | | 12/09/2006 0:36:05 |
|
| legs | | 8/03/2004 1:07:28 |
| The legs robot is used to build the magazine article database for MagPortal.com. |
| Lipperhey-Kaus-Australis | | 16/04/2015 8:22:49 |
|
| MADBbot | | 14/07/2023 16:02:52 |
|
| Mediapartners | | 5/02/2004 20:40:07 |
Google AdSense is for web publishers who want to make more revenue from advertising on their site while maintaining editorial quality.
Mediapartners-Google/2.1 (via babelfish.yahoo.com): this one will look like it is coming from the Yahoo IP range, but the X-Forwarded-For header will contain a Google IP address. |
| MerzScope | | 24/07/2005 23:03:08 |
| Robot is part of a Web-Mapping package called MerzScope, to be used mainly by consultants, and web masters to create and publish maps, on and of the World wide web. |
| miniRank | | 3/05/2006 0:21:23 |
| miniRank is an online tool that ranks websites by popularity in their respective country.The rank is calculated from a wide range of qualitative factors. Webmasters can't pay for a higher score. |
| Miva | | 3/05/2006 0:04:10 |
| MIVA is the new name for Espotting and the FindWhat.com Group. We are now one company, with one brand and one mission - to help businesses grow. |
| MOMspider | | 25/07/2005 0:44:17 |
to validate links, and generate statistics. It's usually run from anywhere
Originated as a research project at the University of California, Irvine, in 1993. Presented at the First International WWW Conference in Geneva, 1994. |
| MorMor TCP Spider | | 29/02/2012 14:17:53 |
Topclick Prospects Spider is a program built by Topclick Prospects to collect data on the web. We save down information such as how well the websites are constructed for search engines. The spider crawls nonstop.
fell for bad bot trap |
| MSRBOT | | 5/02/2004 20:37:06 |
| Microsoft is using the MSRBot web crawler to collect data from the web for further study. |
| NZexplorer | | 11/02/2004 22:48:30 |
| Started in 1995 to provide a comprehensive index to WWW pages within New Zealand. Now also used in Malaysia and other countries. |
| panscient.com | | 9/11/2006 22:42:55 |
At Panscient Technologies we design, build and operate custom internet search engines that unlock the hidden structure of web data.
Using state of the art AI technology, Panscient Technologies' software analyzes web sites for their information content and compiles the data into a searchable index. Our software can be trained to recognize specific entities and relations, so whatever your application, from searching product reviews to detecting new job ads, Panscient Technologies can supply a custom search engine for the task. |
| Patric | | 25/07/2005 1:17:25 |
|
| PiltdownMan | | 25/07/2005 1:25:14 |
The PiltdownMan robot is used to get a list of links from the search engines in our database. These links are followed, and the page that they refer is downloaded to get some statistics from them. The robot runs once a month, more or less, and visits the first 10 pages listed in every search engine, for a group of keywords.
To maintain a database of search engines, we needed an automated tool. That's why we began the creation of this robot. |
| Robot Francoroute | | 11/02/2004 23:17:13 |
| Part of the RISQ's Francoroute project for researching francophone. Uses the Accept-Language tag and reduces demand accordingly |
| RufusBot | | 13/12/2005 1:07:06 |
|
| SafeDNS Search Bot | | 18/10/2015 8:24:58 |
| The main reason for us at SafeDNS to collect web pages, is to correctly categorize the Internet resources and to develop new technologies and products for SafeDNS. |
| SandCrawler | | 1/05/2006 0:06:59 |
| Microsoft's SandCrawler: used for monitoring what server you are running so that they'll know their market share. |
| SBIder | | 29/04/2006 23:50:25 |
SiteSell is gathering a statistical representation of topics presented on the Web as a whole. Each Web page visited is categorized under the topics that it represents, allowing our customers to know the percentage of Web pages that are about any particular topic.
The actual content of all Web pages is removed from all SiteSell systems after being spidered, categorized and scored, usually within 48 hours of being visited by SBIder. |
| ScooperBot | | 13/07/2014 10:51:49 |
| CustomScoop provides an all-in-one monitoring solution that covers traditional and social media. Every account includes on-demand analytics, distribution tools, and personalized service. |
| semanticdiscovery | | 5/02/2004 20:48:19 |
|
| Site Valet | | 31/07/2005 23:45:09 |
| a deluxe site monitoring and analysis service |
| SiteIntel.net Bot | | 25/09/2015 10:56:48 |
|
| SMPU | | 3/05/2006 21:41:15 |
Referer: http://www.norhaus.com/smpu.html
SMPU is a HTTP/1.0 URI parser and spider. The purpose of SMPU is resource collection and web site analysis.
- SMPU does not request any page more than once on any crawl.
- We will send you any information we have collected by request.
What does it do?
More often than not SMPU is used as a download utility, as it can recursively download some (or all) resources on a website. If you are seeing many requests that are all different then your server's contents are being either wholly or partially mirrored by the user.
If you are seeing occasional requests the chances are SMPU is being used as a spider to traverse the internet looking for something, and found a reference to your site.
What can I make it do?
Plenty of things, as a download util it's pretty good but it is more powerful as an analysis tool. You should familiarise yourself with the arguments for an idea of what it can do. It is free to download, and if you are a regular command prompt user it's a pretty useful tool to have around. |
| SMTBot | | 21/09/2014 9:14:57 |
| fell for bad bot trap |
