Evals

Evals are a fixed set of test cases with known-good answers, run against an AI system to measure whether a change made it better or worse. They're the equivalent of a test suite for a probabilistic system.

Without evals, improving an AI feature is guesswork: someone changes a prompt, tries three examples, and declares it better. Evals replace that with a number — this change moved accuracy from 82% to 87% on 200 cases.

They don't need to be elaborate. Fifty real questions with correct answers, run automatically, catches most regressions. The discipline is building the set from real usage rather than invented examples.

Why it matters

Ask whether a proposed AI build includes an evaluation set. If not, nobody will be able to tell you whether it's getting better or worse after launch.

Related terms

PromptA prompt is the input given to a language model — the instructions, context and data it uses to produce output. In production, prompts are versioned a…Large Language Model (LLM)A large language model is a system trained on very large amounts of text to predict likely continuations. That single ability underlies writing, summa…HallucinationA hallucination is output that is fluent, confident and false — an invented citation, a policy that doesn't exist, a fabricated order number. It's a d…

More in AI Automation & Agents

AI AgentWorkflow AutomationLarge Language Model (LLM)RAG (Retrieval-Augmented Generation)EmbeddingVector DatabaseHallucinationHuman in the LoopGuardrailsTool Use (Function Calling)Model Context Protocol (MCP)PromptTokenIdempotencyWebhookAPI IntegrationAI Voice Agent

Need this built rather than explained?

We publish every price we charge, and you get a quote in writing before anything starts.

See every price
All terms