📢 New: get today's jobs on our WhatsApp Channel
Jobiglo

No results.

Senior AI Evaluation Engineer - LLMs & Agentic AI

Management Solutions International (MSI) · Doha

New
Senior 🇬🇧 English
RAGAS DeepEval TruLens OpenAI Evals LangSmith Arize Phoenix A/B testing Statistical analysis

Job description

About the role

A company in Qatar is looking for an experienced Senior AI Evaluation Engineer to lead the evaluation, testing, validation and continuous improvement of advanced AI and Large Language Model (LLM) systems. This is not a conventional machine learning or generative AI developer role focused on building and deploying applications; instead, the engineer determines whether AI systems perform correctly, reliably and consistently in real-world production environments.

The role covers designing evaluation frameworks, benchmark datasets, automated testing pipelines and statistical validation methods to measure quality, accuracy, factual reliability and performance of LLM-powered and agentic AI systems.

Key responsibilities

  • Design and implement evaluation frameworks for LLMs, generative AI and RAG-based applications, measuring factual accuracy, relevance, hallucinations, contextual understanding, consistency and response quality.
  • Implement and validate LLM-as-a-Judge approaches for automated evaluation and scoring, and benchmark different models, prompts and configurations.
  • Build and maintain automated evaluation pipelines using tools such as RAGAS, DeepEval, TruLens, OpenAI Evals, LangSmith or Arize Phoenix, and create datasets, test cases, ground-truth benchmarks and scoring rubrics.
  • Develop repeatable testing processes and set evaluation metrics and quality thresholds for production readiness.
  • Apply statistical methods to assess model performance, run controlled experiments, A/B tests and comparative evaluations, and perform hypothesis testing, significance analysis and confidence interval calculations.
  • Analyse results to identify performance gaps, model weaknesses and improvement opportunities.

Required profile

  • 7+ years of experience.
  • Master's degree or higher in Computer Science, Software Engineering or a related engineering discipline.

Questions fréquentes

Le salaire n'est pas communiqué publiquement par le recruteur. Vous pouvez postuler et négocier directement avec Management Solutions International (MSI).
Cliquez sur "Postuler maintenant" en haut de la page. Vous pouvez importer votre CV en 1 clic — Jobiglo extrait automatiquement vos informations et postule pour vous.

Why are you reporting this job?

Thank you for your report. We will review this job.

Explore further

Salaries, guides and searches in Qatar.

Apply in 30 seconds

Enter your email to apply. An account will be created automatically.

Apply now →

By continuing, you accept our terms of use.

Already have an account? Login

A question about this job?

Ask it here: you will get the full job summary by e-mail, right away.

Published 48 minutes ago

Expires 1 month from now

2 views · 0 interested

Boost your chances

Upload your CV — we will match you with relevant openings.

Analyzing your CV...

Management Solutions International (MSI)

Doha