|The Krugle spider crawls web pages, documents, and archives looking for technical information that would be of value to programmers.|
We use the results of the crawl to provide a vertical search service for programmers. Our product page explains how Krugle helps programmers find code and answers to technical questions.
Our spider is based on Nutch (version 0.8 as of April 2006), and uses various open source components to pull down publicly available information via HTTP(S) and FTP.
To be polite, our spider tries to only access a given domain via one thread at any time. In addition, we impose a 5 second minimum delay between requests.