Databricks Data Engineer with 6 years of experience designing modern data platforms for manufacturing and supply chain organizations. Expertise in Azure Databricks, PySpark, Spark SQL, Delta Lake, Unity Catalog, Auto Loader, Delta Live Tables, Azure Data Factory, and enterprise ETL modernization. Strong experience optimizing large-scale data processing and implementing secure Lakehouse architectures.
Expertise & skills
Selected projects
Responsibilities
- Designed Medallion Architecture for production and quality datasets.
- Developed scalable PySpark ETL pipelines for batch and incremental loads.
- Implemented Delta Live Tables for automated data transformation.
- Reduced processing time by 46% through Spark optimization.
- Configured Unity Catalog to enforce enterprise data governance.
Environment · Azure Databricks, Delta Lake, DLT, PySpark, Azure Data Factory, ADLS Gen2
Responsibilities
- Built Auto Loader pipelines to ingest logistics and warehouse data.
- Developed reusable Spark SQL views for operational reporting.
- Created Power BI dashboards for supply chain KPIs.
- Implemented automated data quality validation and reconciliation.
Environment · Azure Databricks, Auto Loader, Spark SQL, Power BI, Azure Monitor
Responsibilities
- Migrated legacy ERP ETL workflows to Azure Databricks.
- Implemented CI/CD pipelines using Azure DevOps.
- Optimized Delta tables using partitioning and Z-Ordering.
- Collaborated with business users to enhance reporting performance.
Environment · Azure Databricks, Azure DevOps, Delta Lake, Snowflake, Python
Core competencies
Certifications
- Databricks Certified Data Engineer Associate
- Microsoft Certified: Azure Data Engineer Associate
Education
Bachelor of Engineering (Computer Engineering)
Gujarat Technological University
Similar engineers
Other Data Engineering engineers with comparable experience.
