I have 3.4 years of experience in IT field. Overcame challenges of storing & processing data via Hadoop Framework & Apach PySpark Automated and scheduled the Sqoop jobs in a timely manner using Python Scripts. Filtered out bad records on the basis of requirement. Perform validation at different labels and Ingest data into hive table Created replica of hive tables based of security requirements (different level of hive tables). Ingested 70+ sources into HDFS using ELF framework. Written Pyspark script to extract data from staging/Row tables. Created transformation layer from multiple source table based on Policy type. Worked on Hadoop framework to process data at multiple layer based on client requirements. Deployed Apache Spark and Python script for data processing and Hive to store data. Designed solutions & codes using the Hadoop Framework to create Classic layer and transformation layer where PIT table acting as bridge. Independently designed framework like "DQ ,Security and MD5 generation for Struct data type as well as flat data type " using python and spark.