| User Agent | | Date Added |
| StackRambler |  | 25/07/2005 2:51:59 |
|
| stat | | 3/05/2006 22:18:23 |
Experimental search engine
stat is an experimental crawler for a next generation search engine like service, which would try to be webmaster friendly. It's interesting that it only got noticed by now after it has crawled tens of millions of pages. I haven't seen any complains (aside from a couple of curious inquiries) yet regarding it's behavior.
It adheres to the robots.txt instructions. The current minimum page fetch interval for a site is 30 seconds (there is no scheduled delay between the fetch of robots.txt and the first page.) I suspect most webmasters won't mind as there are zero complaints so far. The intention is to create a service that is mutually beneficial to the sites crawled and the service provider.
If you think it's misbehaving on your site, send a note to statcrawler@gmail.com and it'll be dealt with. |
| StractBot | | 15/10/2023 11:48:19 |
|
| SUCHPROGRAMM | | 4/06/2017 23:05:30 |
|
| suzuran | | 9/09/2006 0:33:05 |
| Yokogao Search Engine |
| SwissSearch | | 10/02/2004 1:38:50 |
|
| sygol | | 8/02/2004 19:54:20 |
|
| Szukacz | | 8/02/2004 17:00:12 |
|
| t6labs | | 24/12/2006 22:54:34 |
| T6 Labs is an R&D lab which is into higher order tensor analysis to solve variety of industry related problems. Philosophically all problems of this world where there is an information overload, be it web or computational fluid dynamics, needs higher order tensor analysis for better abstraction. Higher order tensor analysis techniques developed by T6 Labs is being currently used for developing SPAC – a search engine personalization and collaboration platform. |
| Teradex Mapper | | 8/02/2004 17:01:26 |
|
| The Informant | | 10/02/2004 1:40:03 |
| The Informant robot continually checks the Web pages that are relevant to user queries. Users are notified of any new or updated pages. The robot runs daily, but the number of hits per site per day should be quite small, and these hits should be randomly distributed over several hours. Since the robot does not actually follow links (aside from those returned from the major search engines such as Lycos), it does not fall victim to the common looping problems. The robot will support the Robot Exclusion Standard by early December, 1996. |
| The Jubii Indexing Robot | | 8/03/2004 0:46:39 |
| Its purpose is to generate a Resource Discovery database, and validate links. Used for indexing the .dk top-level domain as well as other Danish sites for aDanish web database, as well as link validation. |
| timboBot | | 8/02/2004 19:56:31 |
| timboBot is a bot that scans recently updated weblogs to be included in the BreakingBlogs.com database. |
| Tkensaku | | 8/02/2004 17:01:51 |
| Tkensaku is a web robot, which is surfing the web automatically and make indexes for tkensaku.com |
| Toweyabot | | 9/10/2016 17:05:52 |
|
| TutorGigBot | | 16/11/2004 19:26:27 |
| TutorGigBot collects content from the web for use by TutorGig's Search. It fetches only HTML documents. It does not fetch more than one document each 10 seconds from a website. |
| Tutorial Crawler | | 8/02/2004 17:03:57 |
| TutorGig's Tutorial Crawler collects content from the web for use by TutorGig's Search. |
| TygoBot | | 8/02/2004 17:05:14 |
| TYGO is a powerful search engine and directory designed for anyone who wants fast search results with higher relevancy. |
| Unitek UniEngine | | 8/02/2004 17:08:29 |
|
| UTSE | | 8/02/2004 17:07:16 |
| The UTSE search engine covers Performing Arts and supporting industries worldwide. |
| Vagabondo | | 8/02/2004 17:10:31 |
|
| VoilaBot | | 8/02/2004 17:12:49 |
| Bienvenue sur www.voila.com ! Le moteur de recherche pour les geeks |
| WbSrch | | 17/10/2015 7:22:49 |
|
| WebCrawler | | 10/02/2004 1:44:41 |
|
| WebGo | | 8/02/2004 19:13:50 |
|
| WebRankSpider | | 16/04/2007 0:25:02 |
| WebRankSpider is an experimental web crawler under development since September 2004. WebRankSpider is operated as part of an effort to develop a state-of-the-art searchable Web index. The information gathered by WebRankSpider will be indexed and made accessible via one or more publicly-accessible web sites in the near future. |
| WebSearch | | 8/02/2004 19:15:44 |
|
| WebSearch.COM.AU | | 14/02/2004 13:04:57 |
|
| WebSpider | | 8/02/2004 19:16:17 |
|
| WeViKa | | 6/02/2014 15:09:49 |
|
| WhatchaBot | | 8/02/2004 19:17:23 |
|
| Wild Ferret Web Hopper #1, #2, #3 | | 11/02/2004 22:57:48 |
| The wild ferret web hopper's are designed as specific agents to retrieve data from all available sources on the internet. They work in an onion format hopping from spot to spot one level at a time over the internet. The information is gathered into different relational databases, known as "Hazel's Horde". The information is publicly available and will be free for the browsing at www.greenearth.com. Effective date of the data posting is to be announced. |
| WiseWire | | 10/02/2004 1:46:41 |
|
| woriobot | | 28/03/2008 15:27:27 |
|
| Wotbox | | 8/02/2004 19:18:44 |
|
| yacy | | 5/11/2005 23:26:10 |
p2p-based distributed Web Search Engine
Proxy agent.
Referer header is creator's website, so this is spamming.
The YaCy project is a new approach to build a P2P-based Web indexing network.
* Search your own or the global index
* Crawl your own pages or start distributed crawling
* Run your peer to support other YaCy crawlers
* Provide Information on your peer using the built-in http-server, file-sharing zone and wiki
* Built-in caching http proxy
* Indexing benefits from the proxy cache; private information is not stored or indexed
* Usage of the proxy is not a requisite for web indexing, but it enables you to access the new top-level-domains '.yacy'
* Filter unwanted content like ad- or spyware; share your web-blacklist with other peers
* Easy installation! No additional database required!
* No central server!
* GPL'ed, freeware
|
| Yahoo Mindset | | 1/03/2007 23:40:43 |
|
| Yahoo Slurp | | 18/05/2018 16:36:51 |
|
| Yahoo-MMCrawler | | 8/02/2004 19:20:33 |
| Crawler for Yahoo! paid results supplied by Overture(?) |
| YahooSeeker | | 8/02/2004 19:21:41 |
| Yahoo! crawls hundreds of thousands of web sites for product information to include within Yahoo! Shopping. We extract product information like product names, prices, images, and more and store them within our Yahoo! Product Search index. |
| Yandex | | 8/02/2004 19:19:54 |
| Yandex is the leading Russian web-resource. Yandex sells indexing and search toolkit applicable to a wide range of search and retrieval applications. |
| YandexBot | | 29/06/2010 13:14:33 |
|
| Yanga WorldSearch Bo | | 23/02/2009 0:33:25 |
|
| Yeti | | 13/05/2006 12:21:48 |
|
| YodaoBot | | 26/12/2006 2:46:59 |
|
| Yooo! Search Engine | | 3/09/2018 14:36:51 |
|
| ZipppBot | | 13/05/2004 18:18:21 |
|
| zitebot | | 15/02/2014 12:42:29 |
| Stop searching and get only what you care about. Zite delivers the best of your favorite magazines, newspapers, authors, blogs, and videos. |
| ZyBorg | | 8/02/2004 19:29:37 |
|
