Working in MNC as data Engineer

I have 3.4 years of experience in IT field.
I am currently working in MNC from last 3.2 year as Technology Analyst on Big data domain as a Hadoop developer.
If you have any opening in Big data field please let me know. I have vast knowledge of Big data,Hadoop,Spark & Python.




Brief: Worked collaboratively with the clients and the onsite team to move the data from
Row layer to transformation layer along with the necessary transformations operations
and modifications .The responsibilities included design, development.
Environment: Spark , Hive, Python, HDFS (for storage).

Overcame challenges of storing & processing data via Hadoop Framework & Apach PySpark
Automated and scheduled the Sqoop jobs in a timely manner using Python Scripts.
Filtered out bad records on the basis of requirement.
Perform validation at different labels and Ingest data into hive table Created replica of hive tables based of security requirements (different level of hive tables).
Ingested 70+ sources into HDFS using ELF framework.
Written Pyspark script to extract data from staging/Row tables.
Created transformation layer from multiple source table based on Policy type.
Worked on Hadoop framework to process data at multiple layer based on client requirements.
Deployed Apache Spark and Python script for data processing and Hive to store data.
Designed solutions & codes using the Hadoop Framework to create Classic layer and transformation layer where PIT table acting as bridge.
Independently designed framework like "DQ ,Security and MD5 generation for Struct data type as well as flat data type " using python and spark.