| User Agent | | Date Added |
| JBot Java Web Robot |  | 8/03/2004 0:36:08 |
Java web crawler to download web sites
User agent can be changed by user |
| JoBo Java Web Robot | | 8/03/2004 0:40:18 |
JoBo is a web site download tool. The core web spider can be used for any purpose.
User agent can be changed by user |
| JOC Web Spider | | 6/02/2004 21:57:49 |
| Download websites to your HD and navigate offline! |
| JoeBot | | 8/03/2004 0:44:38 |
| JoeBot is a generic web crawler implemented as a collection of Java classes which can be used in a variety of applications, including resource discovery, link validation, mirroring, etc. It currently limits itself to one visit per host per minute. |
| JPluck | | 6/02/2004 21:34:04 |
| JPluck converts web sites and RSS feeds to Plucker documents for offline reading on your handheld. |
| JRTS Check Favorites | | 7/02/2004 14:04:27 |
| Check Favorites is a full-featured solution for maintaining all of the Internet-based links in your Favorites list (bookmarks). Check Favorites can check multiple links simultaneously and can optionally remove all of the broken links for you. Check Favorites also supports the ability to export your Favorites in a variety of formats, as well as the ability to extract, check, and export the links contained on any HTML page on your system, or accessible via the Internet (thus acting like a kind of link checker and ripper). |
| Kongulo | | 11/10/2006 0:54:49 |
| A simple web spider that lets you keep copies of web sites in your Google Desktop Search index. |
| Kontiki Client | | 6/02/2004 22:00:30 |
|
| LightSpeed | | 15/06/2023 10:43:16 |
|
| memorybot | | 20/05/2014 13:40:21 |
|
| Mirror Checking | | 27/09/2005 11:49:46 |
|
| MixnodeCache | | 23/03/2019 13:44:09 |
| We create a copy of the web so that bots and crawlers come to us and not your website, dramatically reducing your non-human traffic and hosting costs. |
| Monster | | 25/07/2005 0:46:52 |
| The Monster has two parts - Web searcher and Web analyzer. Searcher is intended to perform the list of WWW sites of desired domain (for example it can perform list of all WWW sites of mit.edu, com, org, etc... domain) In the User-agent field $TYPE is set to 'Mapper' for Web searcher and 'StAlone' for Web analyzer. |
| Mozilla/5.0 | | 15/06/2006 22:33:17 |
very aggressive bot (+5 requests/sec) + fell for bad bot trap when copying a client site.
Second encounter: went directly for url with "guestbook" in it. |
| MSIECrawler | | 6/02/2004 0:33:24 |
| To provide users with the best browsing experience, Microsoft® Internet Explorer 4.0 introduced offline browsing to the Microsoft Win32 platform. Internet Explorer 5 extends offline browsing, supporting "smarter" offline Favorites. |
| NavRoad | | 8/09/2006 12:08:53 |
| NavRoad HTML Viewer is a small, fast, powerful off-line HTML browser designed for viewing HTML and web image files (GIF, JPG, PNG, BMP) anytime, anywhere. |
| NearSite | | 8/09/2006 12:11:49 |
| You can get more out of your Internet connection withNearSite. Keep your favourite Web pages and sites close at hand and up-to-date with Autobrowse - NearSite can automatically collect your Web browsing while you get on with other tasks, ready for you browse offline whenever you wish, wherever you are. |
| NetCarta WebMap Engine | | 25/07/2005 0:59:30 |
| The NetCarta WebMap Engine is a general purpose, commercial spider. Packaged with a full GUI in the CyberPilo Pro product, it acts as a personal spider to work with a browser to facilitiate context-based navigation. The WebMapper product uses the robot to manage a site (site copy, site diff, and extensive link management facilities). All versions can create publishable NetCarta WebMaps, which capture the crawled information. If the robot sees a published map, it will return the published map rather than continuing its crawl. Since this is a personal spider, it will be launched from multiple domains. This robot tends to focus on a particular site. No instance of the robot should have more than one outstanding request out to any given site at a time. The User-agent field contains a coded ID identifying the instance of the spider; specific users can be blocked via robots.txt using this ID. |
| NetSpider | | 8/09/2006 12:18:39 |
| The primary objective of NetSpider is to extract and display all the links and local references from a selected page and to allow the user to download them. All the extracted links to other pages can be processed further in the same manner. The program supports resume and has a facility for sites requiring user names and passwords. It is able to accept pages for processing and files for downloading from clipboard. |
| Offline Explorer | | 6/02/2004 22:11:42 |
| Download Web sites to your hard disk for offline browsing |
| Offline Navigator | | 7/09/2006 23:40:39 |
|
| Pack Rat | | 25/07/2005 1:13:56 |
| Used for local maintenance and for gathering web pages so that local statisistical info can be used in artificial intelligence programs. Funded by NEMOnline. |
| pavuk | | 3/05/2006 0:43:52 |
Pavuk is a multifunctional open source web grabber with slow but continous development. This page informs about important news regarding pavuk (usually new releases).
Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set. |
| pcBrowser | | 9/09/2006 0:10:59 |
pcBrowser is offline browsing at its finest, especially since it recognizes more than 40 fully tested filetypes!
With slideshow capability - integrated with a Windows Explorer appeal - pcBrowser is a primo program as an all-around multimedia player/image viewer, taking offline browsing a step higher. |
| PostFavorites | | 3/05/2006 1:26:28 |
Yahoo Search My Web
- Save what you like to build your own personal web
- "Re-find" pages instantly when you need them again
- Share your personal web
- Better than bookmarks |
| puf | | 6/02/2004 22:43:32 |
| puf is a download tool for UNIX-like systems. You may use it to download single files or to mirror entire servers. |
| ReGet | | 9/09/2006 0:18:57 |
|
| retriever | | 12/09/2006 0:38:02 |
|
| ripper | | 12/09/2006 0:38:13 |
|
| RoboFox | | 27/07/2005 23:27:49 |
| scheduled utility to download and database a domain |
| Robot Francoroute | | 11/02/2004 23:17:13 |
| Part of the RISQ's Francoroute project for researching francophone. Uses the Accept-Language tag and reduces demand accordingly |
| SBL-BOT | | 17/10/2014 12:30:55 |
| SoftByte Labs BlackWidow |
| Shai'Hulud | | 31/07/2005 23:40:57 |
Used to build mirrors for internal use.
This robot finds its roots in a research project at RDTeX Perspective Projects Group in 1996. |
| SiteCopy | | 6/02/2004 22:20:31 |
| SiteCopy arkiverer din hjemmeside og sikrer virksomhedens digitale historie. |
| SiteSnagger | | 9/09/2006 0:25:17 |
|
| SiteSucker | | 6/02/2004 22:21:24 |
| SiteSucker can be used to make local copies of your web sites for easy maintenance. SiteSucker can "localize" the files it downloads, allowing you to browse a site off-line. |
| SMPU | | 3/05/2006 21:41:15 |
Referer: http://www.norhaus.com/smpu.html
SMPU is a HTTP/1.0 URI parser and spider. The purpose of SMPU is resource collection and web site analysis.
- SMPU does not request any page more than once on any crawl.
- We will send you any information we have collected by request.
What does it do?
More often than not SMPU is used as a download utility, as it can recursively download some (or all) resources on a website. If you are seeing many requests that are all different then your server's contents are being either wholly or partially mirrored by the user.
If you are seeing occasional requests the chances are SMPU is being used as a spider to traverse the internet looking for something, and found a reference to your site.
What can I make it do?
Plenty of things, as a download util it's pretty good but it is more powerful as an analysis tool. You should familiarise yourself with the arguments for an idea of what it can do. It is free to download, and if you are a regular command prompt user it's a pretty useful tool to have around. |
| stripper | | 12/09/2006 0:40:24 |
|
| sucker | | 12/09/2006 0:40:06 |
|
| Sunrise XP | | 30/06/2007 0:24:38 |
| Sunrise XP converts web sites and newsfeeds to Plucker documents for offline reading on your handheld. |
| SuperBot | | 9/09/2006 0:28:11 |
| By using SuperBot to save your important and frequently used websites directly on your PC, you will ensure that you never lose access to vital Web information. |
| SuperHTTP | | 9/09/2006 0:30:21 |
| A full featured personal web spider. Use it to download entire websites for offline viewing, collect images or other file types off the internet, grab entire collections of files, Query search engines, or check links on your web site. |
| Teleport Webspiders | | 6/02/2004 22:27:40 |
|
| thief | | 12/09/2006 0:40:48 |
|
| thieves | | 12/09/2006 0:40:58 |
|
| vb wininet | | 9/02/2004 23:24:21 |
vbcode example to fetch page from web server.
Proof of monitoring available (log files)
NOT VERIFIED: probably also used in Igrabber Internet content grabber (http://www.aldostools.com/igrabber.html). Website did not exist anymore, google cache was still available:
A powerful internet content grabber that lets you automatically download web server contents off the net, extract specific information and integrate data from multiple pages for your later review, reduce download size or for transfer to portable devices, by simply placing all the 'Content Filters' you want fetched into one directory and pressing a button, using the command line or directly from the Favorites Links using the integration with the new AWeb (Aldo's Web Server).
Includes AWeb 1.5 letting browse filtered sites using filter templates. To filter images enter from MSIE address bar an URL like: http://localhost/filter=filter.dat/http://www.anyweb.com/. |
| Web Downloader | | 6/02/2004 22:29:24 |
|
| Web Sucker | | 11/09/2006 23:13:40 |
|
| WebAuto | | 24/05/2004 11:13:21 |
| WebAuto/3.42ƒÀ1 (WinNT; I) |
| WebCopier | | 6/02/2004 22:30:13 |
| Use our products to record websites and store them locally until you are ready to view them. |
