Recursively checks all links on a site, looking for broken or redirected links. Checks all off-site links using HEAD requests and does not progress further. Designed to behave well and to be very configurable.
Linkdex is an enterprise-class platform that combines team management software with search engine optimization tools. Sometimes we crawl websites to understand backlink profiles or gather information for our users so they can improve their websites and SEO strategies.