Innovative Databricks Data Engineer with 8 years of experience building enterprise data platforms using Azure Databricks, Apache Spark, Delta Lake, and cloud technologies. Skilled in data lake modernization, batch and streaming pipelines, performance tuning, governance, and analytics.
Expertise & skills
Databricks
DatabricksDatabricks SQLDelta LakeDelta Live TablesUnity CatalogMLflowAuto Loader
Big Data
Apache SparkPySparkSpark SQLStructured StreamingKafka
Cloud
Microsoft AzureAzure Data FactoryADLS Gen2Azure SynapseAWS S3
Programming
PythonSQLScalaShell Scripting
DevOps
GitGitHubAzure DevOpsJenkinsTerraform
Visualization
Power BITableau
Selected projects
Energy Consumption Analytics
DatabricksPySparkDelta LakePower BI
Responsibilities
- Built scalable ETL pipelines for smart meter data.
- Designed Bronze, Silver, and Gold layers.
- Optimized Spark transformations.
- Automated daily ingestion workflows.
- Reduced analytics latency by 43%.
Real-Time Payment Processing
DatabricksKafkaStructured StreamingDelta Live Tables
Responsibilities
- Developed low-latency streaming pipelines.
- Processed millions of payment events.
- Implemented checkpointing and fault tolerance.
- Created monitoring dashboards.
- Improved streaming throughput.
Marketing Intelligence Platform
DatabricksSpark SQLUnity Catalog
Responsibilities
- Created curated business datasets.
- Implemented secure governance using Unity Catalog.
- Optimized Spark SQL queries.
- Developed reusable notebook framework.
- Enabled self-service analytics.
HR Workforce Analytics
DatabricksPythonADLS Gen2
Responsibilities
- Integrated workforce data from multiple HR systems.
- Automated data quality validation.
- Developed reusable PySpark modules.
- Improved batch processing efficiency.
- Delivered reporting datasets.
Legacy Data Warehouse Modernization
DatabricksAzure Data FactoryDelta Lake
Responsibilities
- Migrated legacy ETL jobs to Databricks.
- Converted SQL scripts into PySpark.
- Implemented CI/CD deployment pipelines.
- Validated migrated datasets.
- Optimized compute and storage costs.
Certifications
- Databricks Certified Professional Data Engineer
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Azure Fundamentals (AZ-900)
Education
Bachelor of Technology (Computer Engineering)
Achievements
- Delivered 55+ enterprise data engineering solutions.
- Reduced Spark processing time by 43% through optimization.
- Processed over 110 TB of enterprise data.
- Developed 275+ production Databricks notebooks.
- Established reusable Lakehouse standards adopted across multiple projects.
Similar engineers
Other Data Engineering engineers with comparable experience.
