Databricks Data Engineer with 6 years of experience building scalable Lakehouse solutions on Microsoft Azure for insurance and healthcare organizations. Strong expertise in Azure Databricks, Delta Lake, PySpark, Spark SQL, Unity Catalog, Delta Live Tables, MLflow, Azure Data Factory, and performance optimization.
Expertise & skills
Databricks
DatabricksDelta LakeDelta Live TablesUnity CatalogMLflowDatabricks SQLDatabricks Workflows
Big Data
Apache SparkPySparkSpark SQLStructured Streaming
Cloud
Microsoft AzureAzure Data FactoryADLS Gen2Azure Event HubsAzure Key Vault
Programming
PythonSQLScala
Databases
Azure SQL DatabaseCosmos DBPostgreSQL
DevOps
GitAzure DevOpsTerraform
Reporting
Power BI
Selected projects
Responsibilities
- Developed end-to-end ETL pipelines using PySpark and Delta Lake.
- Implemented Delta Live Tables for automated data ingestion.
- Optimized Spark jobs, reducing execution time by 50%.
- Configured Unity Catalog for centralized governance.
- Processed over 9 TB of claims and policy data.
Environment · Azure Databricks, Delta Lake, DLT, ADLS Gen2, Azure Data Factory, Unity Catalog
Responsibilities
- Built scalable ingestion pipelines from multiple healthcare systems.
- Implemented incremental processing using Auto Loader.
- Created reusable transformation framework in PySpark.
- Integrated dashboards with Power BI for operational reporting.
- Improved data quality through automated validation checks.
Environment · Azure Databricks, Auto Loader, PySpark, Event Hubs, Power BI
Responsibilities
- Migrated legacy SQL Server ETL workloads to Azure Databricks.
- Designed Medallion Architecture for analytics workloads.
- Implemented CI/CD using Azure DevOps.
- Collaborated with business teams to modernize reporting.
Environment · Azure Databricks, Spark SQL, Azure DevOps, Azure SQL Database
Core competencies
Lakehouse ArchitecturePerformance OptimizationEnterprise ETL DevelopmentCloud Data EngineeringData GovernanceAgile ScrumProduction Support
Certifications
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate
Education
Bachelor of Engineering (Information Technology)
Rajasthan Technical University
Similar engineers
Other Data Engineering engineers with comparable experience.
