Senior AI Evaluation Engineer - LLMs & Agentic AI
Management Solutions International (MSI) · Doha
Job description
About the role
A company in Qatar is looking for an experienced Senior AI Evaluation Engineer to lead the evaluation, testing, validation and continuous improvement of advanced AI and Large Language Model (LLM) systems. This is not a conventional machine learning or generative AI developer role focused on building and deploying applications; instead, the engineer determines whether AI systems perform correctly, reliably and consistently in real-world production environments.
The role covers designing evaluation frameworks, benchmark datasets, automated testing pipelines and statistical validation methods to measure quality, accuracy, factual reliability and performance of LLM-powered and agentic AI systems.
Key responsibilities
- Design and implement evaluation frameworks for LLMs, generative AI and RAG-based applications, measuring factual accuracy, relevance, hallucinations, contextual understanding, consistency and response quality.
- Implement and validate LLM-as-a-Judge approaches for automated evaluation and scoring, and benchmark different models, prompts and configurations.
- Build and maintain automated evaluation pipelines using tools such as RAGAS, DeepEval, TruLens, OpenAI Evals, LangSmith or Arize Phoenix, and create datasets, test cases, ground-truth benchmarks and scoring rubrics.
- Develop repeatable testing processes and set evaluation metrics and quality thresholds for production readiness.
- Apply statistical methods to assess model performance, run controlled experiments, A/B tests and comparative evaluations, and perform hypothesis testing, significance analysis and confidence interval calculations.
- Analyse results to identify performance gaps, model weaknesses and improvement opportunities.
Required profile
- 7+ years of experience.
- Master's degree or higher in Computer Science, Software Engineering or a related engineering discipline.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in Qatar.
Salaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 48 minutes ago
Expires 1 month from now
2 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Management Solutions International (MSI)
Doha
Related job offers
-
Oracle Primavera Solution Architect
Nexttec Technology Outsourcing Doha -
IT Project Manager – Cybersecurity Initiatives
malomatia Doha -
Oracle NetSuite Functional Consultant
Black & Grey | People Advisory, Executive Search & Employer Branding Doha -
Senior Officer / L3 SOC Analyst
TRIDY HR Services Doha et périphérie -
ICT Technician (Data Center)
QatarEnergy Doha