# AI Evals

- Page: https://shared.btbytes.com/p/ai-evals
- Date: 2025-07-01

- [AI Eval FAQs](https://files.btbytes.com/pdfs/ai/evals-faq.pdf); June 2025.
- [AI Evals Flashcards](https://files.btbytes.com/ai/Evals-Flashcards.pdf)

## EvalOps

> [EvalOps](https://evalops.dev) is the control plane for AI evaluation and routing. It helps teams compare LLMs, track performance, and automatically send each request to the best LLM.
> EvalOps turns evals into your release contract
> - scorecards,
> - scenarios,
> - monitors, and
> - CI gates that keep quality from drifting and get you out of POC purgatory.

> The Pillar worklow:

> 1. Define -- Scorecards establish quality metrics
> 2. Test -- Scenarios provide comprehensive coverage
> 3. Gate -- CI Gates block regressions
> 4. Monitor -- Continuous production oversight

>  The LLM quality problem -- Traditional testing breaks down with LLMs. Non-deterministic outputs, subjective quality metrics, and emergent behaviors make it impossible to ensure quality with conventional approaches.

> 60% of teams deploy untested changes -- No systematic way to validate LLM quality before production

> Average 4 hours per regression --  Time spent debugging issues that reach production
> Manual evaluation bottlenecks -- Human reviewers can't scale with deployment frequency