Senior Manager of AI Safety and Validation Engineering
Discovered View all jobs
- Abu Dhabi
- Permanent
- Full-time
- Define standardised evaluation rubrics and scoring systems for system performance and safety across various AI archetypes including RAG and agentic workflows.
- Design comprehensive test cases covering deterministic logic and adversarial scenarios to stress test AI outputs.
- Implement automated evaluators such as LLM-as-a-judge and targeted small language models Deploy explainability techniques like Shapley Values and Integrated Gradients to ensure model transparency and unfair bias detection in lending and fraud models.
- Partner with platform teams to build telemetry pipelines that capture embeddings and inputs while maintaining strict data privacy standards.
- Establish golden signals such as hallucination rates and semantic similarity to trigger automated circuit breakers when performance thresholds are breached.
- Analyse guardrail violations to identify emerging attack patterns and integrate human-in- the-loop corrections into model refinement cycles.
- Translate technical telemetry into actionable model health reports for second-line risk functions Work with specialist software vendors to bring advanced safety capabilities into the internal AI ecosystem
- 7+ years of experience in Machine Learning Engineering, Data Science, or AI Safety and Testing.
- Bachelor's Degree in Computer Science, Information Technology, or a related engineering discipline.
- Expert proficiency in Python and deep knowledge of statistical testing and XAI libraries.
- Hands-on experience with LLM evaluation frameworks such as Ragas, Giskard, or Arize.
- Proven ability to design observability and monitoring systems at scale.
- Prior experience in the financial sector is highly preferred.
- Strong leadership skills with the ability to act as a technical mentor across cross-functional initiatives.