Databricks Data Engineer with 6 years of experience in designing, developing, and optimizing enterprise-scale data platforms on Microsoft Azure. Experienced in building Lakehouse architectures using Azure Databricks, Delta Lake, Unity Catalog, Delta Live Tables, and Apache Spark. Strong expertise in scalable ETL pipelines using PySpark and Spark SQL, data governance, orchestration, and CI/CD best practices.

Expertise & skills

Databricks

DatabricksDatabricks WorkflowsDelta LakeDelta Live Tables (DLT)Unity CatalogPhoton EngineDatabricks SQL

Big Data

Apache SparkPySparkSpark SQLStructured Streaming

Cloud

Microsoft AzureAzure Data FactoryAzure Data Lake Storage Gen2Azure Key VaultAzure Monitor

Programming

PythonSQLShell Scripting

Databases

SQL ServerAzure Synapse AnalyticsPostgreSQL

DevOps & Version Control

Azure DevOpsGitCI/CD Pipelines

Reporting

Power BI

Selected projects

Responsibilities

Designed Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
Developed scalable ETL pipelines using PySpark.
Created Delta Live Tables for automated data processing.
Optimized Spark jobs using partitioning and Z-Ordering.
Implemented Unity Catalog for centralized data governance.
Automated workflows using Databricks Workflows.
Reduced ETL execution time by 45%.
Processed nearly 12 TB of enterprise sales data.

Environment · Azure Databricks, PySpark, Delta Lake, Delta Live Tables, Azure Data Lake Storage Gen2, Azure Data Factory, Unity Catalog

Responsibilities

Integrated CRM, ERP, and Marketing datasets.
Built reusable PySpark transformation framework.
Designed dimensional data models.
Created automated incremental data pipelines.
Implemented role-based access using Unity Catalog.
Automated deployments through Azure DevOps.
Improved customer analytics data availability.

Environment · Azure Databricks, PySpark, Spark SQL, Azure Data Factory, Azure DevOps, Power BI

Responsibilities

Migrated legacy SQL ETL jobs to Azure Databricks.
Developed incremental processing pipelines.
Implemented Delta Lake optimization techniques.
Created data quality validation framework.
Improved reporting frequency from daily to hourly.
Optimized Spark jobs for large-scale datasets.
Collaborated with BI teams for reporting solutions.

Environment · Azure Databricks, Delta Lake, Spark SQL, Python, Azure Synapse Analytics, Power BI

Core competencies

Lakehouse Architecture DesignEnterprise Data EngineeringPerformance OptimizationETL & ELT DevelopmentData Pipeline DevelopmentData GovernanceData ModelingAgile Scrum MethodologyProduction SupportCross-functional Collaboration