Databricks Engineer with 4 years of experience delivering enterprise data engineering solutions on Azure. Proficient in Azure Databricks, PySpark, Spark SQL, Delta Lake, Databricks Workflows, Unity Catalog, Auto Loader, and Azure Data Factory. Experienced in building scalable Lakehouse architectures, optimizing Spark workloads, and enabling self-service analytics.
Expertise & skills
Cloud
Microsoft Azure
Databricks
DatabricksDelta LakeUnity CatalogAuto LoaderDatabricks WorkflowsDatabricks SQL
Programming
PythonPySparkSQL
Big Data
Apache SparkSpark SQL
Azure Services
Azure Data FactoryADLS Gen2Azure Key VaultEvent Hubs
Databases
Azure SQL DatabasePostgreSQL
Visualization
Power BI
DevOps
GitAzure DevOps
Selected projects
E-Commerce Recommendation Data Platform
- Developed scalable ETL pipelines for customer, product, and clickstream data.
- Implemented incremental ingestion using Auto Loader.
- Created curated Gold datasets for recommendation analytics.
- Optimized Spark jobs using broadcast joins and caching.
- Automated workflows with Databricks Workflows.
Banking Risk Analytics Solution
- Built Delta Lake-based data pipelines for transaction processing.
- Developed reusable PySpark notebooks for data transformation.
- Implemented Unity Catalog for secure data governance.
- Created SQL views for reporting and regulatory analytics.
- Supported production deployments through Azure DevOps.
Energy Consumption Analytics
- Processed smart meter data using Azure Databricks.
- Designed Medallion Architecture for scalable analytics.
- Performed Spark tuning using partitioning and file optimization.
- Integrated business-ready datasets with Power BI.
- Resolved production issues and monitored pipeline health.
Certifications
- Databricks Fundamentals
- Microsoft Azure Fundamentals (AZ-900)
- Microsoft Azure Data Fundamentals (DP-900)
Education
Bachelor of Engineering (Computer Science)
Parul University
Achievements
- 4 years of hands-on Azure Databricks development.
- Experience implementing enterprise Lakehouse architecture.
- Strong expertise in PySpark, Spark SQL, and Delta Lake.
- Worked with structured and semi-structured data sources.
- Experienced in Agile delivery and production support.
Similar engineers
Other Data Engineering engineers with comparable experience.
