Data Engineer with 2 years of experience in Databricks and Azure ecosystem, specializing in scalable data engineering, Spark-based ETL development, Delta Lake, data quality, and cloud data integration. Experienced in developing efficient data pipelines, optimizing Spark workloads, and supporting analytics solutions.
Expertise & skills
Big Data
DatabricksApache SparkPySparkDelta Lake
Programming
PythonSQL
Cloud
Microsoft Azure
Azure
Azure Data FactoryADLS Gen2Azure Synapse AnalyticsAzure Key Vault
Databases
Azure SQL DatabaseSQL Server
DevOps
GitAzure DevOps
Visualization
Power BI
Methodology
Agile / Scrum
Core Skills
Databricks DevelopmentPySparkDelta LakeSpark SQLAzure Data FactoryETL DevelopmentPerformance TuningData ValidationTeam Collaboration
Selected projects
Insurance Data Lake Modernization
DatabricksPySparkDelta LakeAzure Data FactoryADLS Gen2
Worked on building cloud-based data engineering solutions using Databricks and Azure services.
Responsibilities
- Developed PySpark ETL pipelines for insurance policy data.
- Created Bronze, Silver, and Gold Delta Lake architecture.
- Automated daily ingestion using Azure Data Factory.
- Implemented data quality validation rules.
- Optimized Spark jobs using partitioning and caching.
- Supported production deployment and issue resolution.
E-commerce Customer Analytics
DatabricksSpark SQLPythonAzure SQL
Worked on building cloud-based data engineering solutions using Databricks and Azure services.
Responsibilities
- Built customer analytics datasets using Databricks notebooks.
- Developed reusable Spark SQL transformations.
- Created incremental data loading processes.
- Improved query performance through Delta optimization.
- Collaborated with BI developers for reporting datasets.
- Maintained technical documentation.
Manufacturing Supply Chain Reporting
DatabricksPySparkAzure Data LakeGit
Worked on building cloud-based data engineering solutions using Databricks and Azure services.
Responsibilities
- Processed supply-chain datasets from multiple sources.
- Developed scalable ETL workflows.
- Implemented logging and exception handling.
- Performed source-to-target reconciliation.
- Participated in Agile ceremonies.
- Provided production support.
Core competencies
Databricks DevelopmentPySparkDelta LakeSpark SQLAzure Data FactoryETL DevelopmentPerformance TuningData ValidationTeam Collaboration
Certifications
- Databricks Fundamentals Accreditation
- Microsoft Azure Fundamentals (AZ-900)
- DP-900: Microsoft Azure Data Fundamentals
Education
Bachelor of Technology (Information Technology)
Parul University
Similar engineers
Other Data Engineering engineers with comparable experience.
