Senior QA Engineer, AI

ICE's AI Center of Excellence is looking for a Senior Software Test Engineer to own QA for our AI-powered applications. This role works with Systems Analysts, Development, and QA to understand business, product, and system requirements, and define test scenarios for AI applications by combining core QA fundamentals, Python scripting, and API validation with specialized AI/LLM evaluation skills.

• Lead test strategy for AI/ML features: model output validation, regression testing across model/prompt versions, and edge case discovery.
• Build evaluation frameworks for testing AI outputs for accuracy, consistency, bias, hallucination, and safety/guardrail compliance.
• Define golden datasets, ground truth labels, and release acceptance criteria with data science and ML engineering.
• Automate prompt-based test suites and model evaluation metrics into CI/CD release gates.
• Lead and mentor a QA team, setting AI testing standards across projects.
• Translate AI feature requirements into test scenarios and quality gates.
• Monitor production of AI behavior and evolve test coverage based on real-world failures.

• 8+ years of QA/test engineering experience, including 3+ years testing AI, ML, or data-driven applications.
• 3+ years of experience with AI/LLM evaluation frameworks: prompt regression, hallucination detection, bias/fairness testing, and model/data drift detection.
• 5+ years of Python experience for test automation, data analysis, and evaluation scripting.
• 3+ years of experience with UI automation frameworks (Selenium, Playwright, or Cypress) for AI-powered features.
• Strong SQL and data validation skills for testing the pipelines that feed AI models.
• Experience testing REST/GraphQL APIs: functional, contract, and schema validation.