We have good experience in scraping and crawling the web. Tech stack we use- 1. Selenium + Phantomjs+ xvfb 2. Python + celery + beautifulsoup + supervisord 3. PHP + nginx + Mysql We have built a free legal website (lexofy.com) by scraping and crawling the web.