I have several years of experience working on web crawling based projects. I have worked on very complex projects including parsing PDF files, download images and files, processing huge amounts of data, solving blocking issues, etc. I focus on performance and reliable data, also I can run the crawlers on my own servers periodically so you don't need to worry about infrastructure or difficult setups.