Azure Databricks Developer with 4 years of experience building enterprise-scale data engineering solutions on Microsoft Azure. Experienced in PySpark, Spark SQL, Delta Lake, Unity Catalog, Databricks Asset Bundles, Auto Loader, Azure Data Factory, and Azure Data Lake Storage Gen2. Skilled in developing high-performance ETL pipelines, data governance, and analytics-ready Lakehouse platforms.
Expertise & skills
Cloud
Microsoft Azure
Databricks
DatabricksDelta LakeUnity CatalogAuto LoaderDatabricks SQLDatabricks Asset Bundles
Programming
PythonPySparkSQL
Big Data
Apache SparkSpark SQL
Azure Services
Azure Data FactoryADLS Gen2Azure Key VaultAzure FunctionsAzure Monitor
Databases
SQL ServerPostgreSQL
DevOps
GitAzure DevOpsCI/CD
Reporting
Power BI
Selected projects
Insurance Fraud Detection Lakehouse
- Developed PySpark pipelines to ingest policy, claims, and fraud investigation datasets.
- Designed Bronze, Silver, and Gold layers using Delta Lake architecture.
- Implemented incremental ingestion with Auto Loader for daily claim files.
- Optimized Spark jobs using adaptive execution and partition pruning.
- Published analytics-ready datasets for fraud investigation dashboards.
Retail Inventory Intelligence Platform
- Integrated ERP, warehouse, and POS datasets into Azure Databricks.
- Developed reusable transformation notebooks using PySpark.
- Implemented Unity Catalog for centralized data governance.
- Automated orchestration through Azure Data Factory.
- Improved query performance using Optimize and Z-Ordering.
Travel Booking Analytics Platform
- Built ETL pipelines for flight, hotel, and booking transaction data.
- Created business-ready Gold tables for KPI reporting.
- Implemented monitoring, logging, and exception handling.
- Supported production releases through Azure DevOps CI/CD pipelines.
- Worked closely with product owners and QA teams in Agile sprints.
Certifications
- Databricks Fundamentals
- Microsoft Azure Fundamentals (AZ-900)
- Microsoft Azure Data Engineer Associate (DP-203)
Education
Bachelor of Technology (Information Technology)
Ganpat University
Achievements
- 4 years of Azure Databricks development experience.
- Expertise in PySpark, Delta Lake, and Unity Catalog.
- Hands-on experience building enterprise Lakehouse solutions.
- Strong Spark performance optimization and troubleshooting skills.
- Experienced with Agile delivery, code reviews, and CI/CD.
Similar engineers
Other Data Engineering engineers with comparable experience.
