Experienced Databricks Data Engineer with 8 years of expertise in Azure Databricks, Apache Spark, PySpark, Delta Lake, ETL/ELT pipelines, Lakehouse Architecture, Spark optimization, and real-time data processing solutions.
Expertise & skills
Databricks
DatabricksApache SparkPySparkSpark SQLDelta LakeDelta Live TablesUnity CatalogDatabricks WorkflowsMLflowAuto LoaderStructured Streaming
Cloud Technologies
Microsoft AzureAzure Data FactoryAzure Data Lake Storage Gen2Azure Blob StorageAzure Key VaultAzure DevOpsAWS S3
Programming Languages
PythonSQLScalaShell Scripting
Databases
SQL ServerPostgreSQLMySQLSnowflake
DevOps & Version Control
GitGitHubAzure DevOpsJenkinsTerraform
Reporting Tools
Power BITableau
Selected projects
Enterprise Lakehouse Modernization
DatabricksPySparkDelta LakeAzure Data FactoryUnity CatalogAzure DevOps
Responsibilities
- Designed scalable ETL pipelines using PySpark.
- Implemented Delta Lake and optimized Spark workloads.
- Built reusable transformation frameworks.
- Automated workflows and deployments.
- Improved data quality and processing performance.
Customer 360 Analytics Platform
DatabricksKafkaStructured StreamingDelta Live TablesPower BI
Responsibilities
- Designed scalable ETL pipelines using PySpark.
- Implemented Delta Lake and optimized Spark workloads.
- Built reusable transformation frameworks.
- Automated workflows and deployments.
- Improved data quality and processing performance.
Financial Reporting Platform
DatabricksSpark SQLPythonAzure Data Lake Storage Gen2
Responsibilities
- Designed scalable ETL pipelines using PySpark.
- Implemented Delta Lake and optimized Spark workloads.
- Built reusable transformation frameworks.
- Automated workflows and deployments.
- Improved data quality and processing performance.
Healthcare Claims Data Processing Platform
DatabricksPySparkDelta LakeAzure Data Factory
Responsibilities
- Designed scalable ETL pipelines using PySpark.
- Implemented Delta Lake and optimized Spark workloads.
- Built reusable transformation frameworks.
- Automated workflows and deployments.
- Improved data quality and processing performance.
Supply Chain Analytics Platform
DatabricksPythonSpark SQLAzure Blob Storage
Responsibilities
- Designed scalable ETL pipelines using PySpark.
- Implemented Delta Lake and optimized Spark workloads.
- Built reusable transformation frameworks.
- Automated workflows and deployments.
- Improved data quality and processing performance.
Certifications
- Databricks Certified Data Engineer Associate
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
Education
Bachelor of Engineering (Computer Engineering)
Gujarat Technological University
Achievements
- Reduced Spark processing time by 42%.
- Delivered 40+ enterprise data engineering solutions.
- Developed 200+ production Databricks notebooks.
- Migrated 80+ TB of enterprise data.
- Implemented reusable ETL frameworks reducing development effort by 35%.
Similar engineers
Other Data Engineering engineers with comparable experience.
