Senior QA Engineer - AI Products (6 Months Contract)

About the role

A company is looking for a Senior QA Engineer to own quality across its AI products. You will test LLM-powered features end to end, build automated test and evaluation suites, and help ensure every release is accurate, safe and reliable. This is a 6-month contract.

Key responsibilities

• Define test strategies and plans for AI features across web, API and backend
• Evaluate model outputs for accuracy, consistency, hallucinations, bias and safety
• Build evaluation datasets, including edge cases and adversarial / red-team prompts
• Develop and maintain automated UI, API and regression tests integrated into CI/CD
• Run regression testing on model, prompt and configuration changes
• Test performance, latency, error handling and fallback behavior
• Report defects clearly and give release-readiness sign-off
• Test non-deterministic outputs using scoring rubrics, LLM-as-judge, human evaluation and defined pass/fail thresholds
• Perform AI security testing, including prompt injection, jailbreaks and PII / data leakage across chatbots, RAG and agents
• Work with ML engineers and product managers to define quality metrics and acceptance criteria
• Where applicable, monitor production quality and model drift after release

Required profile

• 7+ years of experience in QA / software testing, with a strong test automation background
• Proven experience testing AI/ML or LLM-based products (chatbots, RAG, agents, etc.)
• Excellent written English and clear bug reporting

Required skills

• Hands-on scripting in Python and/or TypeScript/JavaScript
• At least one UI automation framework (Playwright, Cypress, Selenium) and a test framework such as Pytest
• API testing with Postman, REST Assured or similar
• CI/CD and version control: GitHub Actions, GitLab CI or Jenkins; Git
• Test management: Jira, TestRail or Zephyr
• Nice to have: performance/load testing (k6, JMeter, Locust), prompt engineering, embeddings and vector databases, AI safety and red-teaming, cloud (AWS, GCP, Azure) and Docker