Databricks Data Engineer with 6 years of experience delivering enterprise data engineering solutions for the insurance and financial services domain. Experienced in Azure Databricks, Delta Lake, PySpark, Spark SQL, Unity Catalog, Databricks Asset Bundles, Delta Live Tables, Auto Loader, Azure Data Factory, and cloud-native Lakehouse implementations. Strong expertise in building scalable ETL pipelines, performance tuning, data governance, and production support.
Expertise & skills
Selected projects
Responsibilities
- Designed a Medallion Architecture for policy, claims, and billing datasets.
- Developed scalable ETL pipelines using PySpark and Delta Lake.
- Implemented Delta Live Tables to automate ingestion and transformation.
- Optimized Spark jobs using partitioning and Z-Ordering, reducing execution time by 47%.
- Established enterprise-wide governance with Unity Catalog.
Environment · Azure Databricks, Delta Lake, Delta Live Tables, Unity Catalog, Azure Data Factory, ADLS Gen2
Responsibilities
- Built Auto Loader pipelines for ingesting structured and semi-structured claim data.
- Created reusable Spark SQL models for fraud analytics.
- Implemented data quality validation and monitoring framework.
- Delivered curated datasets for Power BI dashboards and self-service analytics.
Environment · Azure Databricks, Auto Loader, PySpark, Spark SQL, Power BI, Azure Monitor
Responsibilities
- Migrated legacy ETL workloads from SQL Server to Databricks Lakehouse.
- Implemented CI/CD using Azure DevOps and Databricks Asset Bundles.
- Enhanced pipeline reliability through automated testing and deployment.
- Worked with business teams to improve reporting performance and data availability.
Environment · Azure Databricks, Azure DevOps, Databricks Asset Bundles, SQL Server, Snowflake
Core competencies
Certifications
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate
Education
Bachelor of Engineering (Information Technology)
Savitribai Phule Pune University
Similar engineers
Other Data Engineering engineers with comparable experience.
