I am use to work web crawling for several project. 1. Journalistpub - it was news aggregator for several world news agencies 2. pinteresting - health blogs aggregator more than 300 blogs 3. pronajdi.com - local ads and shop agregator 4. booking agregator for some hotel from tipical location. (still in function) http://diybotix.com/index.php?module=hotels&action=index it daily collect data for 18 hotels for 365 days