Senior AI Evaluation Engineer - LLMs & Agentic AI
Management Solutions International (MSI) · Doha
وصف الوظيفة
About the role
A company in Qatar is looking for an experienced Senior AI Evaluation Engineer to lead the evaluation, testing, validation and continuous improvement of advanced AI and Large Language Model (LLM) systems. This is not a conventional machine learning or generative AI developer role focused on building and deploying applications; instead, the engineer determines whether AI systems perform correctly, reliably and consistently in real-world production environments.
The role covers designing evaluation frameworks, benchmark datasets, automated testing pipelines and statistical validation methods to measure quality, accuracy, factual reliability and performance of LLM-powered and agentic AI systems.
Key responsibilities
- Design and implement evaluation frameworks for LLMs, generative AI and RAG-based applications, measuring factual accuracy, relevance, hallucinations, contextual understanding, consistency and response quality.
- Implement and validate LLM-as-a-Judge approaches for automated evaluation and scoring, and benchmark different models, prompts and configurations.
- Build and maintain automated evaluation pipelines using tools such as RAGAS, DeepEval, TruLens, OpenAI Evals, LangSmith or Arize Phoenix, and create datasets, test cases, ground-truth benchmarks and scoring rubrics.
- Develop repeatable testing processes and set evaluation metrics and quality thresholds for production readiness.
- Apply statistical methods to assess model performance, run controlled experiments, A/B tests and comparative evaluations, and perform hypothesis testing, significance analysis and confidence interval calculations.
- Analyse results to identify performance gaps, model weaknesses and improvement opportunities.
Required profile
- 7+ years of experience.
- Master's degree or higher in Computer Science, Software Engineering or a related engineering discipline.
Questions fréquentes
لماذا تبلغ عن هذا العرض؟
اكتشف المزيد
الرواتب والأدلة وعمليات البحث في قطر.
الرواتب حسب المهنة
قدم طلبك في 30 ثانية
أدخل بريدك الإلكتروني للتقديم. سيتم إنشاء حساب تلقائياً.
بالمتابعة، أنت توافق على شروط الاستخدام.
لديك حساب بالفعل؟ تسجيل الدخول
لديك سؤال حول هذا العرض؟
اطرحه هنا: ستصلك تفاصيل العرض كاملة عبر البريد الإلكتروني، فوراً.
عزز فرصك
حمّل سيرتك الذاتية وسنقترح عليك الوظائف التي تناسب ملفك.
جاري تحليل سيرتك الذاتية...
Management Solutions International (MSI)
Doha
عروض عمل ذات صلة
-
Oracle Primavera Solution Architect
Nexttec Technology Outsourcing Doha -
IT Project Manager – Cybersecurity Initiatives
malomatia Doha -
Oracle NetSuite Functional Consultant
Black & Grey | People Advisory, Executive Search & Employer Branding Doha -
Senior Officer / L3 SOC Analyst
TRIDY HR Services Doha et périphérie -
ICT Technician (Data Center)
QatarEnergy Doha