| Singingfish uses an innovative, multimedia-oriented spider or robot to crawl the Web, looking for media files to index for the Singingfish Streambase. Our spider, called Asterias®, obeys spider Netiquette rules: It checks the Robot Exclusion File and it avoids hitting each site too frequently. Asterias® identifies multimedia (Real Media, Windows Media, Quicktime, MP3 and Flash) files and adds them to our index. Sometimes this process results in many sequential accesses to a Site. While this process results in a complete index, the behavior is atypical of other spiders found on the web. |
