Databricks Data Engineer with 6 years of experience delivering cloud-based data engineering solutions using AWS and Databricks Lakehouse. Skilled in PySpark, Spark SQL, Delta Lake, Auto Loader, Unity Catalog, Databricks Workflows, and CI/CD. Experienced in building scalable batch and streaming pipelines for banking and financial services.
Expertise & skills
Databricks
Databricks LakehouseDelta LakeUnity CatalogAuto LoaderDatabricks WorkflowsDatabricks SQL
Big Data
Apache SparkPySparkSpark SQLStructured Streaming
Cloud
AWSAmazon S3AWS GlueIAMCloudWatch
Programming
PythonSQLShell Scripting
Databases
SnowflakeAmazon RedshiftMySQL
DevOps
GitAzure DevOpsJenkins
Visualization
Power BITableau
Selected projects
Responsibilities
- Developed streaming pipelines using Structured Streaming and Auto Loader.
- Designed Bronze, Silver and Gold layers using Delta Lake.
- Optimized Spark jobs and reduced processing time by 40%.
- Implemented Unity Catalog for secure data access.
- Built reusable PySpark frameworks for ingestion.
Environment · AWS, Databricks, Delta Lake, Auto Loader, PySpark, S3, Unity Catalog
Responsibilities
- Integrated customer, loan and transaction datasets.
- Created incremental ETL pipelines with PySpark.
- Improved reporting SLA from 6 hours to 90 minutes.
- Implemented data quality validation framework.
- Supported CI/CD deployment using Jenkins and Git.
Environment · Databricks, Spark SQL, Python, AWS Glue, Redshift, Jenkins
Responsibilities
- Migrated legacy ETL jobs to Databricks.
- Implemented partitioning and Delta optimization.
- Created reusable SQL views for analytics.
- Worked closely with business analysts and QA teams.
Environment · Databricks, Delta Lake, Spark SQL, S3, Tableau
Core competencies
Data Lakehouse ArchitecturePerformance TuningCloud Data EngineeringData GovernanceETL DevelopmentAgile MethodologyProduction Support
Certifications
- Databricks Certified Data Engineer Associate
- AWS Certified Data Analytics – Specialty
Education
Bachelor of Technology (Information Technology)
Visvesvaraya Technological University
Similar engineers
Other Data Engineering engineers with comparable experience.
