Azure Data Engineer

Epergne Solutions

Abu Dhabi
Permanent
Full-time

1 month ago

Job Role: - Azure Data EngineerJob Location :- Abu DhabiExperience:- 5+ YearsWe are looking for a highly skilled Azure Data Engineer for an engagement to build scalable, production-ready data solutions.The ideal candidate will have strong hands on experience with Python, PySpark, Azure Databricks, and Azure Data Factory, along with a solid understanding of modern data engineering best practices.Key ResponsibilitiesDesign, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and Spark SQLBuild and optimize workloads on Azure Databricks (notebooks, job clusters, workflows)Implement Delta Lake best practices (schema evolution, partitioning, performance tuning, merge strategies)

Develop robust ingestion frameworks integrating with Azure Data Lake Storage Gen2 and Azure SQL
Implement data validation, transformation logic, and quality control frameworks
Ensure performance optimization and cost-efficient execution
Support production deployments, monitor pipelines, and resolve incidents
Maintain version control using Git and follow CI/CD best practices
Collaborate with architects, analysts, and business stakeholders

Mandatory SkillsStrong hands-on experience in PySpark and Spark SQLProven experience in Spark performance tuning & optimizationSolid expertise in Azure DatabricksStrong knowledge of Delta Lake architecture

Experience working with Azure Data Lake Storage Gen2
Git-based version control experience
Production support and troubleshooting experience

Good to HaveExposure to data governance, data security, and compliance frameworks

Experience integrating data from SAP ERP systems
Knowledge of Azure Data Factory (ADF)
Understanding of DevOps and CI/CD pipelines

Epergne Solutions

Apply Now