1) Maintained data pipeline up-time of 99.9% while ingesting streaming and transactional data across 7 different primary data sources using Spark, Redshift, S3, and Python.
2) Ingested data from disparate data sources using a combination of SQL, Google Analytics API, and Salesforce API using Python to create data views to be used in BI tools like Tableau.
3) Automated ETL processes across billions of rows of data which reduced manual workload by 33% monthly.
4) Designed and implemented a real-time data pipeline to process semi- structured data by integrating 150 million raw records from 30+ data sources using Kafka and Spark and stored processed data in Redshift.
5) Designed the data pipeline architecture for a new product that quickly scaled from 0 to 115,000 daily active users.
6) Designed and implemented a real-time data pipeline to process semi- structured data by integrating 150 million raw records from 30+ data sources using Kafka and Spark and stored processed data in Redshift.
7) Designed the data pipeline architecture for a new product that quickly scaled from 0 to 115,000 daily active users
8) Utilized Spark in Python to distribute data processing on large streaming datasets to improve ingestion and processing speed of that data by 87%.
9) Experienced in Data Entry, analyzing huge amount of data and making visualization.
10) Use and improve ETL frameworks, continuous data quality frameworks and other automation in data pipeline.
11) Participate in daily standups, lead design reviews and offshore coordination