Hiring: Data Engineer

SoftEdge is seeking a Data Engineer to design, develop, validate, optimization, and ongoing ownership of data solutions primarily using AWS and Databricks. The role involves working closely with internal teams, business SMEs, architects, governance teams, analytics teams, and implementation partners.

• Design, develop, and support AWS and Databricks data pipelines using SQL, Python, PySpark, and Spark.
• Develop and maintain Bronze, Silver, and Gold data layers using Databricks and Delta Lake.
• Work with source-system data, business rules, mappings, and integration requirements.
• Review and validate data solutions delivered by implementation partners.
• Perform data profiling, reconciliation, testing, and troubleshooting.
• Support data quality, metadata, lineage, and Unity Catalog activities.
• Optimize Databricks pipelines and queries for performance, scalability, and cost efficiency.
• Participate in code reviews, CI/CD, deployment, and production support.
• Maintain technical documentation and support knowledge transfer to internal teams.
• Collaborate closely with BI and analytics teams on downstream reporting and data requirements.
• Support integration with Tableau and Power BI for reporting and analytics use cases.

• 5+ years of experience in data engineering, ETL/ELT, or data integration
• Strong hands-on experience with Databricks
• Strong experience with AWS data services and cloud-based data platforms
• Strong SQL and Python skills
• Strong experience with PySpark / Apache Spark
• Experience designing and implementing data pipelines and data transformation frameworks
• Strong understanding of data integration, transformation, reconciliation, and data quality