Sr Devops Engineer with Databricks

Senior DevOps Engineer responsible for provisioning and managing data platform resources, operating Databricks and Unity Catalog, orchestrating data pipelines with Airflow, dbt, Fivetran, and Atlan, and ensuring security, monitoring, and cost management.

• Provision and manage platform resources through reusable Terraform modules and configuration as code
• Operate Databricks accounts and workspaces, including identities, groups, service principals, entitlements, compute policies, clusters, SQL warehouses and Jobs/Workflows
• Configure Unity Catalog metastores, catalogs, schemas, external locations, storage credentials, workspace bindings, ownership and least‑privilege access
• Operate dbt projects, environments, connections, jobs, permissions, service tokens, CI quality gates and promotion controls
• Engineer production Astro/Airflow capabilities covering DAG standards, dependencies, retries, timeouts, concurrency, backfills, ingestion and safe recovery
• Orchestrate and integrate Fivetran, dbt, Databricks and Atlan using APIs, webhooks and workload identities
• Implement Fivetran usage monitoring, cost thresholds, non‑production lifecycle guardrails and configuration portability
• Enable governed Atlan and Tableau connectivity, metadata, lineage, access and operational workflows
• Implement ADLC controls for branching, testing, approval, environment promotion, deployment evidence and rollback
• Integrate identity and access workflows with Okta or equivalent identity providers and ServiceNow
• Establish platform and pipeline SLOs; monitor reliability, schema changes, data quality, usage and cost
• Troubleshoot incidents, perform root‑cause analysis and automate recurrence prevention
• Maintain runbooks, configuration inventories, access evidence and recovery procedures

• 6+ years in data platform engineering, DataOps, platform engineering, DevOps or SRE
• 5+ years of hands‑on Terraform and IaC experience
• Strong Python engineering for APIs, automation, testing, and packaging
• Production experience operating Databricks and Unity Catalog
• Strong production Apache Airflow experience
• Experience operating dbt environments, jobs and CI/CD workflows
• Proficiency with AWS IAM, S3, VPC, KMS, Secrets Manager and CloudWatch
• Experience with Git, CI/CD, REST APIs, SQL, Linux, OAuth, SAML/SCIM and service‑principal authentication
• Experience with monitoring, incident response, change control, rollback and cost management
• Relevant Databricks platform training or certification highly preferred