Accomplished Databricks engineer with 10 years of experience delivering modern Lakehouse solutions, enterprise data platforms, streaming architectures, and cloud-native analytics. Expertise includes Databricks, Delta Lake, Unity Catalog, Delta Live Tables, Spark optimization, MLflow, and Azure cloud services.
Expertise & skills
Core Skills
DatabricksDelta LakeUnity CatalogDelta Live TablesDatabricks WorkflowsDatabricks SQLApache SparkPySparkSpark SQLStructured StreamingKafkaPythonSQLScalaAzure Data FactoryADLS Gen2Azure SynapseAzure DevOpsGitJenkinsTerraformCI/CDSnowflakeSQL ServerPostgreSQL
Selected projects
Cloud Data Platform Transformation
- Designed enterprise Lakehouse architecture.
- Migrated 650+ ETL jobs to Databricks.
- Implemented Auto Loader and Delta Lake.
- Reduced processing cost by 40%.
- Built reusable ingestion framework.
Environment · Azure Databricks, Delta Lake, Auto Loader, PySpark, ADLS Gen2
IoT Streaming Analytics
- Built near real-time IoT data pipelines.
- Processed over 15 million sensor events daily.
- Implemented Structured Streaming with Kafka.
- Optimized streaming performance.
Environment · Databricks, Kafka, Structured Streaming, Python
Financial Reporting Lakehouse
- Developed medallion architecture.
- Automated data validation.
- Created optimized Delta tables.
- Improved report generation by 50%.
Environment · Azure Databricks, Spark SQL, Delta Live Tables
Enterprise Data Governance
- Implemented Unity Catalog.
- Configured fine-grained access control.
- Established metadata standards.
- Enabled enterprise data lineage.
Environment · Unity Catalog, Azure AD, Databricks
ML Data Preparation Framework
- Built reusable feature engineering pipelines.
- Integrated MLflow.
- Automated training dataset preparation.
Environment · MLflow, Databricks, PySpark
Certifications
- Databricks Certified Data Engineer Professional
- Databricks Certified Associate Developer for Apache Spark
- Microsoft Certified: Azure Data Engineer Associate
Education
Anna University
Achievements
- Delivered 35+ Databricks implementations.
- Handled 25+ TB of daily data processing.
- Reduced cloud costs by 40%.
- Improved Spark job performance by 65%.
- Migrated 650+ ETL workflows to Databricks.
Similar engineers
Other Data Engineering engineers with comparable experience.
