M
Senior Data Scientist, AI Evaluation
On-siteFull-timeData ScienceEnterprise Software
Own evaluation methods and infrastructure for production AI agents. Define trusted quality metrics, integrate continuous evaluation into agent workflows, and help teams use evaluation results to make production decisions.
Responsibilities
- Design evaluation metrics, pipelines, and methods used across teams.
- Standardize agent development, regression testing, maintenance, and post-deployment monitoring.
- Translate product expectations into datasets, test suites, and continuous evaluation pipelines.
- Build evaluation tools and convert product requirements into measurable production standards.
- Develop evaluation strategies for emerging agent architectures and failure modes.
- Partner with AI feature teams to improve evaluation practices and adoption.
Requirements
- At least 3 years of non-academic data science experience with complex production AI systems.
- Familiarity with agent frameworks such as LangGraph or LangChain and current agent SDKs.
- Practical understanding of agent models, context, capabilities, and harnesses, with experience in agent evaluation methods.
- Production-grade coding skills and experience building and shipping end-to-end data or evaluation pipelines.
- Ability to inspect execution traces, diagnose failure modes, and create qualitative error taxonomies.
- Strong product judgment and communication skills to turn evaluation data into actionable guidance.
- Experience partnering with engineering and product teams and driving adoption across an organization.
Nice to have
- Experience designing evaluation strategies for production agentic systems.
- Experience on centralized platform or infrastructure teams supporting multiple product areas.
- Familiarity with microservices, GitHub workflows, CI/CD, and agent observability or evaluation tools such as LangSmith or Langfuse.
- Knowledge of TypeScript or modern platform architectures.
- Master’s degree in computer science, data science, statistics, engineering, or a related quantitative field.