Key Responsibilities

• Define the overarching evaluation strategy and success metrics for GenAI systems
• Architect and scale frameworks for golden sets, synthetic data, and automated regressions
• Design the end-to-end data flywheel including instrumentation and feedback collection
• Lead cross-functional quality initiatives to align product, ops, and engineering on performance standards
• Productionize pipelines for model monitoring and continuous pre/post-deployment testing

Essential

• Extensive expertise in evaluation methodologies including offline/online alignment and metric design
• Hands-on experience with Generative AI orchestration, including RAG, tool calling, and memory management
• Proven track record of building complex data pipelines and quality systems
• Experience with human-in-the-loop evaluation and labeling workflows
• Ability to lead large-scale technical projects across multiple departments
• Strong background in statistical analysis, A/B testing, and power analysis

Preferred

• Experience with LLM-as-judge frameworks and synthetic data generation
• Background in developing guardrails and bias detection for AI systems
• Familiarity with dataset versioning and governance in a large-scale production environment

Key Skills