|
|
|
|
|
User-Agents Database |
User Agent Details of Krugle
| Name: | Krugle |
| URL: | http://www.krugle.com/crawler/info.html |
| User Agent Key: | Krugle |
| User Agents: | Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com)
|
| IP Ranges: | |
| Adheres robots.txt: | Yes |
| Adheres META tag: | No |
| Type(s): |
IndexingTool
|
| Verified: | | | Info: | The Krugle spider crawls web pages, documents, and archives looking for technical information that would be of value to programmers.
We use the results of the crawl to provide a vertical search service for programmers. Our product page explains how Krugle helps programmers find code and answers to technical questions.
Our spider is based on Nutch (version 0.8 as of April 2006), and uses various open source components to pull down publicly available information via HTTP(S) and FTP.
To be polite, our spider tries to only access a given domain via one thread at any time. In addition, we impose a 5 second minimum delay between requests. | | Date Added: | 29/04/2006 23:52:36 | | |  |
User Agents - Search
|
|
