(Quote: WebMasterWorld) They probably are just categorizing sites so that they know which ads to show visitors to your site... or which sites to pop-under an advertisers ad... whenever a visitor with their "enhancements" surfs to your site.
Scrapy Cloud, our cloud-based web crawling platform, allows you to easily deploy crawlers and scale them on demand – without needing to worry about servers, monitoring, backups, or cron jobs. It helps developers like you turn over two billion web pages per month into valuable data.
Our platform's many add-ons let you extend your spiders in clicks. Among them, our smart proxy rotator (Crawlera) helps you bypass bot counter-measures so you can crawl large sites faster.
Your data gets safely stored in a high-availability database. You can browse it and share it with your team from your dashboard, or consume your data in your app using our API.