Azure Databricks Data Engineer

NovaFlow Ai is seeking an experienced Azure Databricks Data Engineer to design, build, and maintain scalable data pipelines and modern cloud data solutions. You will work closely with software engineers, data scientists, analysts, and technical stakeholders to build reliable, scalable, and high-quality data solutions.

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Python, and SQL
• Build data ingestion and transformation pipelines using Azure Data Factory and Azure Databricks
• Develop and maintain Delta Lake tables and lakehouse data architectures
• Design efficient data models for analytics, reporting, and downstream applications
• Implement data quality, validation, monitoring, and error-handling processes
• Optimize Spark jobs, SQL queries, cluster configurations, partitioning, and data pipelines for performance and cost efficiency
• Implement appropriate security and governance practices using Unity Catalog
• Develop and maintain CI/CD workflows for Databricks notebooks, jobs, and pipelines

• 3+ years of professional experience in Data Engineering
• Strong hands-on experience with Azure Databricks
• Strong experience with Python and PySpark
• Strong SQL skills, including complex queries, joins, CTEs, window functions, and performance optimization
• Experience building production-grade ETL/ELT pipelines
• Experience with Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS Gen2)
• Strong understanding of Apache Spark and distributed data processing
• Experience with Delta Lake and lakehouse architecture