Azure Databricks Data Engineer with 4 years of experience designing cloud-native data platforms and scalable ETL solutions. Strong expertise in PySpark, Spark SQL, Delta Lake, Unity Catalog, Databricks SQL, Azure Data Factory, Azure Data Lake Storage Gen2, and workflow orchestration. Experienced in building high-performance data pipelines for analytics and business intelligence.
Expertise & skills
Cloud
Microsoft Azure
Databricks
DatabricksDelta LakeDatabricks SQLUnity CatalogAuto LoaderPhoton
Programming
PythonPySparkSQL
Big Data
Apache SparkSpark SQL
Azure Services
Azure Data FactoryADLS Gen2Azure Key VaultAzure Functions
Databases
Azure SQLPostgreSQLMongoDB
DevOps
GitAzure DevOps
Reporting
Power BI
Selected projects
Telematics Data Lakehouse
- Developed PySpark pipelines to process vehicle telemetry data from multiple sources.
- Implemented Bronze, Silver, and Gold architecture using Delta Lake.
- Optimized large-scale Spark jobs using Photon and partition pruning.
- Built reusable ingestion framework with Auto Loader.
- Delivered curated datasets for fleet analytics dashboards.
Media Streaming Analytics Platform
- Created ETL pipelines for user activity and content consumption data.
- Developed parameterized notebooks for reusable transformations.
- Implemented Unity Catalog for secure data governance.
- Automated orchestration with Azure Data Factory.
- Enhanced query performance using Optimize and Z-Ordering.
Customer Loyalty Analytics
- Integrated CRM, POS, and marketing datasets into a centralized Lakehouse.
- Developed business-ready Gold tables for executive reporting.
- Implemented data validation and reconciliation checks.
- Performed production support, monitoring, and incident resolution.
- Collaborated with QA and business teams in Agile delivery.
Certifications
- Databricks Fundamentals
- Microsoft Azure Fundamentals (AZ-900)
- Microsoft Azure Data Engineer Associate (DP-203)
Education
Bachelor of Engineering (Information Technology)
L.J. Institute of Engineering and Technology
Achievements
- 4 years of hands-on Azure Databricks development experience.
- Expertise in Lakehouse architecture and enterprise ETL solutions.
- Strong skills in Spark optimization, Delta Lake, and Unity Catalog.
- Experience with Azure Data Factory and CI/CD deployments.
- Worked on Telecom, Media, and Customer Analytics domains.
Similar engineers
Other Data Engineering engineers with comparable experience.
