Data Engineer – Confidential -World-leading shipping company

• Create Databrick’s workspace on a cluster then configure the cluster in Azure to run notebooks.

• Create Python/Scala/SQL code within Databricks notebooks that read and write data from files, which

store in various formats like CVS, JSON, Parquet, and XML, located in different data sources.

• Create notebooks in Databricks that ingested data from Azure Data Lake Storage into Databricks

pipelines that shaped and curated data for

• Prepare advanced SQL queries within notebooks that run jobs, which extract, transform and load (ETL/

ELT) data, that can infer schema change, modify Delta Live Table, and monitor data loads.

• Prepare Azure Data Factory (ADF) pipelines that ingest and move data from Azure Data Lake Storage

(ADLS) into SQL Server Database,

• Prepare Data warehouse star schema using slow changing dimension (SCD) both type 1 and type 2

method

• Create Azure Synapse workspace using serverless SQL pools to query parquet files existing on ADLS

Azure Data Engineer – Confidential -American leading donut company

• Create pipelines with data flow activities to move data from Rest API to Azure Data Lake based on

Franchisee’s location and HQ’s data requirements.

• Build data movement and data transformation logic within the data pipeline’s activities, which utilize

complex and iterative processing logic, for data ingestion and preparation in both Azure Data Factory

and Databrick’s Notebooks.

• Monitor data pipelines and activities output to identify data within different tables, files, folders, and

documents.

• Create data models of business’ information to translate to the logical data model layer to create

tables and schema in snowflake.

• Detailed oriented data engineering professional with over 15 years of experience who undertakes complex

assignments and delivers consistent customer value-focus performance.

• Design integration layer from Azure Data Factory and Azure Data Lake to move transformed data into

Snowflake data warehouse and Azure Synapse Analytics.

• Design a data warehouse architecture of a star schema that adopted fact and dimensional tables with

Snowflake for business intelligence tool consumption.

• Build a Type 2 Slowly Changing Dimension (SCD) using Snowflake’s Stream functionality and automate

the process using Snowflake’s Task functionality.