Results-driven Databricks Developer with 4 years of experience building cloud-native data platforms on Azure. Expertise in PySpark, Delta Lake, Databricks Workflows, Structured Streaming, MLflow, Unity Catalog, and Azure Data Factory. Experienced in designing scalable Lakehouse solutions, optimizing Spark workloads, and delivering enterprise analytics solutions.
Expertise & skills
Cloud
Microsoft Azure
Databricks
DatabricksDelta LakeDatabricks SQLUnity CatalogAuto LoaderDatabricks WorkflowsMLflowStructured Streaming
Languages
PythonPySparkSQL
Data Engineering
Apache SparkETLData LakehouseData Modeling
Azure Services
ADLS Gen2Azure Data FactoryAzure Key VaultAzure DevOps
Visualization
Power BI
Version Control
Git
Selected projects
Healthcare Claims Analytics Platform
- Designed scalable ETL pipelines for healthcare claims processing.
- Built Bronze, Silver and Gold layers using Delta Lake.
- Implemented Structured Streaming for near real-time claim ingestion.
- Optimized Spark jobs reducing execution time by 35%.
- Created reporting datasets consumed by Power BI.
Manufacturing IoT Data Platform
- Developed Auto Loader pipelines to ingest IoT sensor files.
- Created reusable PySpark notebooks for transformation.
- Implemented Unity Catalog for governance and access control.
- Improved storage efficiency using Optimize and Vacuum operations.
- Automated orchestration using Databricks Workflows.
ML Feature Engineering Platform
- Developed feature engineering pipelines using PySpark.
- Tracked model experiments using MLflow.
- Created reusable feature tables for data science teams.
- Implemented CI/CD deployment using Azure DevOps.
- Provided production support and performance tuning.
Certifications
- Databricks Fundamentals
- Microsoft Azure Data Fundamentals (DP-900)
Education
Bachelor of Technology (Information Technology)
Nirma University
Achievements
- 4 years of Azure Databricks experience.
- Hands-on expertise with PySpark, Delta Lake and Structured Streaming.
- Experience implementing Lakehouse architecture for enterprise projects.
- Strong understanding of Spark optimization and production support.
- Worked in Agile Scrum environment.
Similar engineers
Other Data Engineering engineers with comparable experience.
