Databricks Data Engineer with 6 years of experience building enterprise data platforms for telecom and media organizations. Experienced in Azure Databricks, Delta Lake, PySpark, Spark SQL, Unity Catalog, Delta Live Tables, Kafka integration, Azure Data Factory, and Databricks Workflows. Strong background in performance tuning, streaming, and Lakehouse architecture.
Expertise & skills
Databricks
DatabricksDelta LakeDelta Live TablesUnity CatalogDatabricks WorkflowsDatabricks SQLPhoton Engine
Big Data
Apache SparkPySparkSpark SQLStructured StreamingKafka
Cloud
Microsoft AzureAzure Data FactoryADLS Gen2Azure Event HubsAzure Key Vault
Programming
PythonSQLScala
Databases
Azure SQL DatabaseSnowflakeMongoDB
DevOps
GitAzure DevOpsTerraform
Visualization
Power BI
Selected projects
Responsibilities
- Built scalable ingestion pipelines for network usage data using PySpark.
- Implemented Medallion Architecture with Delta Lake.
- Optimized Spark workloads, reducing execution time by 42%.
- Configured Unity Catalog for secure access management.
- Processed over 15 TB of monthly telecom data.
Environment · Azure Databricks, Delta Lake, PySpark, ADLS Gen2, Unity Catalog
Responsibilities
- Developed streaming pipelines using Structured Streaming and Kafka.
- Integrated Azure Event Hubs for real-time ingestion.
- Created monitoring dashboards using Databricks SQL and Power BI.
- Implemented automated alerts for pipeline failures.
Environment · Azure Databricks, Kafka, Event Hubs, Spark SQL, Power BI
Responsibilities
- Migrated legacy ETL workloads to Databricks Lakehouse.
- Developed reusable ETL framework using PySpark.
- Implemented CI/CD with Azure DevOps.
- Enhanced data quality through automated validation rules.
Environment · Azure Databricks, Azure DevOps, Delta Lake, Python, Snowflake
Core competencies
Lakehouse ArchitectureStreaming Data ProcessingSpark Performance TuningCloud Data EngineeringData GovernanceAgile MethodologyProduction Support
Certifications
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate
Education
Bachelor of Engineering (Electronics & Communication)
Anna University
Similar engineers
Other Data Engineering engineers with comparable experience.
