Results-oriented Databricks professional with 10 years of experience delivering enterprise data platforms, cloud migrations, real-time analytics, and Lakehouse solutions using Apache Spark, Delta Lake, Unity Catalog, and Azure Databricks. Experienced in building scalable data pipelines, optimizing performance, and implementing governance for large organizations.
Expertise & skills
Core Skills
DatabricksDatabricks SQLDelta LakeDelta Live TablesUnity CatalogAuto LoaderApache SparkPySparkSpark SQLStructured StreamingKafkaPythonSQLScalaAzure Data FactoryADLS Gen2Azure SynapseGitAzure DevOpsJenkinsTerraformSnowflakeSQL ServerPostgreSQL
Selected projects
Responsibilities
- Built enterprise Lakehouse using Medallion Architecture.
- Developed incremental ETL pipelines with Auto Loader.
- Reduced batch processing time by 57%.
- Implemented Delta Lake optimization and partitioning.
- Automated deployments using CI/CD.
Environment · Azure Databricks, Delta Lake, Auto Loader, PySpark, Azure DevOps
Responsibilities
- Created real-time event processing pipelines.
- Integrated Kafka with Structured Streaming.
- Processed over 18 million events daily.
- Improved dashboard latency significantly.
Environment · Databricks, Kafka, Structured Streaming, Python
Responsibilities
- Migrated legacy Hadoop workloads to Databricks.
- Converted Hive jobs into optimized PySpark notebooks.
- Reduced infrastructure costs by 42%.
- Improved reliability using Delta Live Tables.
Environment · Azure Databricks, Spark, Hive, Delta Live Tables
Responsibilities
- Developed reusable data quality framework.
- Implemented Unity Catalog governance.
- Configured row-level access controls.
- Created centralized metadata repository.
Environment · Unity Catalog, Databricks, SQL
Responsibilities
- Created reusable feature pipelines.
- Integrated MLflow for experiment tracking.
- Automated feature refresh processes.
- Improved model readiness for analytics teams.
Environment · MLflow, Databricks, PySpark
Certifications
- Databricks Certified Data Engineer Professional
- Databricks Certified Associate Developer for Apache Spark
- Microsoft Certified: Azure Data Engineer Associate
- Microsoft Certified: Azure Solutions Architect Expert
Education
Vellore Institute of Technology (VIT)
Achievements
- Completed 40+ Databricks enterprise implementations.
- Optimized Spark jobs improving performance by 57%.
- Migrated 700+ ETL workflows to Databricks.
- Processed 30+ TB of data daily.
- Reduced cloud infrastructure costs by 42%.
- Implemented enterprise-wide data governance using Unity Catalog.
Similar engineers
Other Data Engineering engineers with comparable experience.
