Experienced Databricks Data Engineer with 8 years of expertise in architecting modern Lakehouse solutions using Azure Databricks. Strong background in large-scale ETL development, real-time streaming, Delta Lake optimization, data governance, cloud migration, and analytics engineering.
Expertise & skills
Databricks
DatabricksDatabricks SQLDelta LakeUnity CatalogDelta Live TablesMLflowAuto Loader
Big Data
Apache SparkPySparkSpark SQLStructured StreamingKafkaHadoop
Cloud
Microsoft AzureAzure Data FactoryADLS Gen2Azure Synapse Analytics
Programming
PythonSQLScala
DevOps
GitAzure DevOpsJenkinsTerraform
Reporting
Power BITableau
Selected projects
Insurance Risk Analytics Platform
DatabricksPySparkDelta LakePower BI
Responsibilities
- Developed enterprise ETL pipelines for policy and claims data.
- Implemented Medallion Architecture.
- Optimized Spark jobs using partitioning and caching.
- Built reusable PySpark libraries.
- Improved reporting performance by 46%.
Fraud Detection Data Platform
DatabricksStructured StreamingKafkaMLflow
Responsibilities
- Built streaming pipelines for fraud detection.
- Processed high-volume transactional events.
- Implemented Delta Live Tables.
- Integrated MLflow for model lifecycle management.
- Configured monitoring and alerts.
Customer Behavior Analytics
DatabricksSpark SQLUnity Catalog
Responsibilities
- Unified customer datasets from multiple systems.
- Implemented secure access using Unity Catalog.
- Created Gold-layer business datasets.
- Optimized SQL workloads.
- Supported self-service analytics.
Supply Chain Optimization
DatabricksPythonADLS Gen2
Responsibilities
- Designed scalable data ingestion framework.
- Automated ETL orchestration.
- Improved data validation process.
- Reduced batch processing duration.
- Created reusable transformation framework.
Enterprise Cloud Migration
DatabricksDelta LakeAzure Data Factory
Responsibilities
- Migrated legacy ETL jobs to Databricks.
- Converted SQL workflows into PySpark.
- Implemented CI/CD deployment pipelines.
- Validated migrated data.
- Optimized storage and compute costs.
Certifications
- Databricks Certified Data Engineer Associate
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Azure Fundamentals (AZ-900)
Education
Bachelor of Engineering (Computer Science)
Achievements
- Delivered 60+ enterprise data engineering solutions.
- Reduced Spark execution time by 46% through optimization.
- Migrated over 120 TB of enterprise data to Lakehouse architecture.
- Built more than 300 production-ready Databricks notebooks.
- Standardized enterprise ETL framework adopted across multiple teams.
Similar engineers
Other Data Engineering engineers with comparable experience.
