Databricks Data Engineer with 2 years of experience in designing, developing, and optimizing scalable data pipelines using Databricks, Apache Spark, Delta Lake, and Azure Data Factory. Hands-on experience in building ETL/ELT workflows, data transformation, data validation, and performance optimization. Familiar with Azure cloud services, SQL, Python, and data warehousing concepts. Passionate about building reliable big data solutions and improving data processing efficiency.
Expertise & skills
Selected projects
Retail Sales Analytics Platform
Developed scalable ETL pipelines for retail sales analytics.
Responsibilities
- Developed PySpark notebooks.
- Created Delta Lake tables.
- Built incremental data loads.
- Performed data cleansing.
- Optimized Spark jobs.
- Integrated ADF with Databricks.
- Monitored pipelines.
- Participated in sprint planning.
Healthcare Claims Data Processing
Modernized healthcare claims ETL using Databricks.
Responsibilities
- Created Spark SQL queries.
- Developed PySpark scripts.
- Built reusable notebooks.
- Loaded Delta Lake data.
- Performed reconciliation.
- Optimized pipelines.
- Supported deployments.
- Prepared documentation.
Financial Transaction Reporting
Built reporting pipelines for financial transaction data.
Responsibilities
- Developed ETL workflows.
- Processed large datasets.
- Created Spark SQL transformations.
- Implemented logging.
- Maintained Delta tables.
- Used Git.
- Assisted UAT.
- Collaborated with analysts.
Core competencies
Certifications
- Microsoft Certified: Azure Data Fundamentals
- Databricks Lakehouse Fundamentals
- Microsoft Azure Fundamentals (AZ-900)
Education
Bachelor of Engineering (Computer Engineering)
Gujarat Technological University
Similar engineers
Other Data Engineering engineers with comparable experience.
