Responsible for demand analysis based on the data to be crawled, analyzing the website structure of the target website and some anti-crawling methods;Responsible for writing crawler programs tocapture content through requests, scrapy, scrapy-redis, xpath and other technical means, cleaning and filtering the captured data, and storing them in separate tables for use by other departments;Responsible for regularly maintaining the ip proxy pool, clearing invalid proxy ip, and optimizing your own program according to some anti-climbing methods encountered during the crawling process;Responsible for using the Django framework of the web to participate in the development of other companies' internal software;Responsible for deduplication and cleaning of data, anti-climbing of Amazon IP, detection of verification codes, and formulation of specific countermeasures.