Databricks Data Engineer with 6 years of experience delivering modern data platforms for retail and e-commerce organizations. Hands-on expertise in Azure Databricks, Apache Spark, PySpark, Delta Lake, Unity Catalog, Auto Loader, Databricks SQL, Azure Data Factory, and CI/CD implementation. Experienced in designing high-performance Lakehouse architectures and optimizing large-scale ETL workloads.
Expertise & skills
Selected projects
Responsibilities
- Designed Medallion Architecture for enterprise sales analytics.
- Developed scalable PySpark ETL pipelines for daily and hourly loads.
- Optimized Delta Lake tables using Z-Ordering and file compaction.
- Reduced data processing time by 48% through Spark optimization.
- Enabled secure data access using Unity Catalog.
Environment · Azure Databricks, PySpark, Delta Lake, Unity Catalog, Azure Data Factory, ADLS Gen2
Responsibilities
- Built incremental ingestion pipelines using Auto Loader.
- Integrated customer, product, and transaction datasets.
- Developed reusable transformation framework in PySpark.
- Created Databricks SQL dashboards for business reporting.
- Improved pipeline reliability with automated monitoring.
Environment · Azure Databricks, Auto Loader, Spark SQL, Python, Databricks SQL, Azure Monitor
Responsibilities
- Migrated legacy ETL workflows to Databricks Lakehouse.
- Implemented CI/CD pipelines using Azure DevOps.
- Created data quality validation and reconciliation processes.
- Collaborated with business stakeholders to enhance reporting accuracy.
Environment · Azure Databricks, Delta Lake, Azure DevOps, Snowflake, Power BI
Core competencies
Certifications
- Databricks Certified Data Engineer Associate
- Microsoft Certified: Azure Data Engineer Associate
Education
Bachelor of Technology (Computer Science)
Jawaharlal Nehru Technological University
Similar engineers
Other Data Engineering engineers with comparable experience.
