Databricks Data Engineer with 6 years of experience delivering cloud-native data engineering solutions for pharmaceutical and healthcare organizations. Experienced in Azure Databricks, Delta Lake, PySpark, Spark SQL, Unity Catalog, Auto Loader, Delta Live Tables, Azure Data Factory, and data governance. Skilled in building scalable Lakehouse architectures, optimizing ETL pipelines, and enabling advanced analytics.
Expertise & skills
Selected projects
Responsibilities
- Designed Medallion Architecture for clinical and patient datasets.
- Built scalable PySpark ETL pipelines for batch and incremental processing.
- Implemented Delta Live Tables to automate data ingestion.
- Reduced ETL execution time by 44% using Spark optimization techniques.
- Configured Unity Catalog for centralized governance and secure data access.
Environment · Azure Databricks, Delta Lake, DLT, PySpark, Azure Data Factory, ADLS Gen2
Responsibilities
- Developed Auto Loader pipelines for sales and distributor data.
- Created reusable Spark SQL models for enterprise reporting.
- Built Power BI dashboards for sales performance analytics.
- Implemented automated data validation and reconciliation processes.
Environment · Azure Databricks, Auto Loader, Spark SQL, Power BI, Azure Monitor
Responsibilities
- Migrated legacy ETL workloads to Azure Databricks Lakehouse.
- Implemented CI/CD pipelines using Azure DevOps.
- Optimized Delta tables through partitioning and file compaction.
- Collaborated with business users to improve reporting performance.
Environment · Azure Databricks, Azure DevOps, Delta Lake, Snowflake, Python
Core competencies
Certifications
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate
Education
Bachelor of Technology (Information Technology)
Dr. A.P.J. Abdul Kalam Technical University
Similar engineers
Other Data Engineering engineers with comparable experience.
