Highly skilled Databricks Data Engineer with 8 years of experience designing modern data platforms using Azure Databricks, Apache Spark, Delta Lake, and cloud-native technologies. Experienced in data lake modernization, enterprise ETL/ELT development, streaming analytics, governance, and performance optimization.
Expertise & skills
Databricks
DatabricksDatabricks SQLDelta LakeDelta Live TablesUnity CatalogMLflowAuto LoaderDatabricks Workflows
Big Data
Apache SparkPySparkSpark SQLStructured StreamingKafka
Cloud
Microsoft AzureAzure Data FactoryAzure Data Lake Storage Gen2Azure Synapse AnalyticsAWS S3
Programming
PythonSQLScalaShell Scripting
DevOps
GitGitHubAzure DevOpsJenkinsTerraform
Visualization
Power BITableau
Selected projects
Pharmaceutical Data Lake Platform
DatabricksPySparkDelta LakeAzure Data Factory
Responsibilities
- Designed scalable ETL pipelines for clinical and operational datasets.
- Implemented Medallion Architecture for structured data processing.
- Optimized Spark jobs using partitioning and caching techniques.
- Automated data ingestion with Databricks Workflows.
- Improved pipeline execution performance by 44%.
E-Commerce Recommendation Engine
DatabricksStructured StreamingKafkaMLflow
Responsibilities
- Developed streaming pipelines for customer activity data.
- Processed high-volume clickstream events.
- Integrated MLflow for model lifecycle management.
- Implemented Delta Live Tables for reliable data ingestion.
- Created monitoring dashboards for pipeline health.
Enterprise Finance Analytics
DatabricksSpark SQLUnity Catalog
Responsibilities
- Built curated financial reporting datasets.
- Implemented secure data governance using Unity Catalog.
- Optimized complex Spark SQL transformations.
- Automated reconciliation processes.
- Enabled self-service BI reporting.
Telecommunications Network Analytics
DatabricksPythonADLS Gen2
Responsibilities
- Developed ETL pipelines for network performance metrics.
- Automated data quality validation.
- Built reusable PySpark components.
- Reduced batch processing duration.
- Delivered analytics-ready datasets.
Cloud Lakehouse Transformation
DatabricksDelta LakeAzure DevOps
Responsibilities
- Migrated legacy warehouse workloads to Lakehouse architecture.
- Converted SQL procedures into PySpark notebooks.
- Implemented CI/CD pipelines using Azure DevOps.
- Validated data consistency after migration.
- Optimized compute and storage utilization.
Certifications
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Azure Fundamentals (AZ-900)
Education
Bachelor of Engineering (Information Technology)
Achievements
- Delivered more than 65 enterprise data engineering solutions.
- Reduced Spark execution time by 44% through advanced optimization techniques.
- Processed over 130 TB of enterprise data across cloud platforms.
- Developed 320+ production-ready Databricks notebooks.
- Established reusable Lakehouse frameworks adopted across multiple business units.
Similar engineers
Other Data Engineering engineers with comparable experience.
