A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot." Entire sites or specific pages can be selectively visited and indexed. The best way to extract huge data that is spread out across many pages of a website is by building a Crawler. Based on requirement, a Crawler travel to every page of that site looking for other pages that match. Crawlers are best used for when you want lots of data, but don’t know all the URLs for that site.