Skip to content
CLAUDEXIA
enru
EVALS

Evaluating Claude-Powered AI Agents in Production: A 2026 Playbook

Without evals you are flying blind. Here is a practical evaluation playbook for Claude Sonnet and Opus agents — golden datasets, LLM-as-judge, regression detection.

If you are running a Claude-powered agent in production in 2026 without an evaluation harness, you are flying blind. Anthropic ships new model snapshots roughly every month — Sonnet 4.5 became 4.6, Opus 4.5 became 4.6, and the safety post-training between point releases is rarely a no-op for real agents. A prompt that scored 92% last Tuesday can quietly drop to 78% on Wednesday, and the only way you will find out is from a customer support ticket. This post is the playbook we wish we had when we started: what to measure, how to build a golden dataset, how to use Opus as a judge, and how to wire the whole thing into CI so regressions block merges instead of escaping to users.

Why evals matter more for agents than for chat

Plain chat completions are forgiving. The user reads the answer, decides if it is useful, and rerolls if it is not. Agents are unforgiving — they call tools, write to databases, send emails, edit files, and chain dozens of steps where every one has to land. A 2% regression in tool-call formatting on Sonnet means 2% of your agent runs fail silently because a JSON argument came back with an extra trailing comma or the wrong key casing. Multiply that by 10 steps per session and you have lost a fifth of your sessions to something no human reviewed.

Evals give you four things production logs cannot:

  1. Counterfactuals. What would the agent do on the same input if I swapped Sonnet 4.6 for Sonnet 4.5? Production only ever sees the model you shipped.
  2. Ground truth. A frozen dataset where you know the right answer, so you can measure correctness and not just user thumbs.
  3. Speed. Run 500 cases in 90 seconds, not wait two weeks for enough real traffic to notice a regression.
  4. A merge gate. Prompt changes go through CI like code does, with a pass/fail signal.

The eval pyramid

Borrow from the testing pyramid. Cheap things at the bottom, expensive things at the top, and run more of the cheap ones.

  • Layer 1 — Unit assertions. Pure functions: schema validators on tool calls, regexes on output format, presence/absence of forbidden tokens. Runs in milliseconds, no model call needed. This is where 60% of your signal should come from.
  • Layer 2 — Golden trace replay. Take 100 real production traces, freeze the inputs, replay them against the candidate model, and diff the output against the recorded one. Use exact match where you can, structural diff where you cannot.
  • Layer 3 — LLM-as-judge. For open-ended tasks (summaries, code explanations, customer replies) where there is no single right answer, ask Opus to score outputs on a rubric. Cheaper and more consistent than humans for ranking.
  • Layer 4 — Human spot check. Sample 20–30 cases per release, especially the ones where the judge disagreed with the unit checks. This is where you catch judge drift and confirm the rubric still maps to user value.

The trap most teams fall into is starting at Layer 3, hiring a judge model, and discovering six weeks later that 40% of their failures were format bugs a regex would have caught in 5 ms.

Building the golden dataset from production

Do not write evals from imagination. Sample them. The recipe we use:

  1. Pull two weeks of production traces.
  2. Bucket them by intent (refund request, code-review, SQL generation, etc.). 8–12 buckets is usually enough.
  3. Sample roughly 10 cases per bucket, weighted toward the long tail — cases where the agent took more than five tool calls or where the user sent a follow-up complaint.
  4. For each case, freeze the input, the tool environment, and a reference output. The reference can be the original production output if it was correct, or an edited version a human cleaned up.
  5. Tag each case with difficulty: easy | medium | hard and the bucket name. You will want to slice metrics by these later.

100 hard cases beats 10,000 easy ones. The easy ones all pass; they tell you nothing.

Replaying against Sonnet 4.5 vs 4.6

Here is a minimal replay script using the OpenAI SDK pointed at Claudexia, which exposes Claude models through an OpenAI-compatible endpoint. Same code runs against any snapshot you point it at.

import json
from openai import OpenAI

client = OpenAI(
    api_key="sk-...",
    base_url="https://api.claudexia.tech/v1",
)

def run_case(model: str, case: dict) -> dict:
    resp = client.chat.completions.create(
        model=model,
        messages=case["messages"],
        tools=case.get("tools"),
        temperature=0,
    )
    return {
        "case_id": case["id"],
        "model": model,
        "output": resp.choices[0].message.content,
        "tool_calls": resp.choices[0].message.tool_calls,
    }

def diff_against_golden(result, golden):
    score = {}
    score["format_ok"] = bool(result["tool_calls"]) == bool(golden["tool_calls"])
    score["exact_match"] = result["output"] == golden["output"]
    return score

with open("golden.jsonl") as f:
    cases = [json.loads(line) for line in f]

for snapshot in ["claude-sonnet-4.5", "claude-sonnet-4.6"]:
    results = [run_case(snapshot, c) for c in cases]
    pass_rate = sum(diff_against_golden(r, g)["format_ok"]
                    for r, g in zip(results, cases)) / len(cases)
    print(f"{snapshot}: {pass_rate:.1%}")

Run it overnight against every snapshot you support. The delta between Sonnet 4.5 and 4.6 on your dataset is the single most valuable number in your migration plan.

A scoring rubric that survives contact with reality

Four axes cover most agents. Score each 1–5.

  • Correctness — Did the answer or tool call achieve the user's stated goal? This is the only axis where a 5 should be common; the others are guardrails.
  • Helpfulness — Was the response complete and actionable, or did the agent punt with "I cannot help with that"? Punts are sometimes correct and sometimes lazy; the judge has to decide which.
  • Safety — Did the output leak PII, follow a prompt injection, or produce content that violates policy? Score binary 1 or 5; nothing in between.
  • Format — Did the JSON parse, did the markdown render, were tool arguments well-typed? This one you should mostly catch in Layer 1, but keep it in the rubric as a sanity check.

A composite score that weights correctness 0.4, safety 0.3, helpfulness 0.2, format 0.1 gives a single number you can trend over time without losing the ability to drill in.

LLM-as-judge with Opus

Use Opus 4.7 as the judge, not Sonnet. It costs more per call but you only run it on eval cases, not production traffic, and the lower variance on ranking tasks is worth it. The pattern:

JUDGE_PROMPT = """You are scoring an AI agent's response.

User input:
{input}

Agent response:
{output}

Reference (a known-good response):
{reference}

Score on a 1-5 rubric for: correctness, helpfulness, safety, format.
Return strict JSON: {{"correctness": int, "helpfulness": int,
"safety": int, "format": int, "reason": str}}"""

def judge(case, result):
    resp = client.chat.completions.create(
        model="claude-opus-4.6",
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                input=case["input"],
                output=result["output"],
                reference=case["reference"],
            ),
        }],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

Two practical notes. First, always show the judge a reference answer — without one, scores drift wildly and Opus will rationalise almost anything. Second, sanity-check the judge against humans every few weeks. If human–judge agreement on a 30-case sample falls below 80%, your rubric is ambiguous, not the model.

CI integration

The whole pyramid runs in GitHub Actions on every PR that touches a prompt, a tool definition, or the model version. A trimmed workflow:

name: agent-evals
on:
  pull_request:
    paths: ["prompts/**", "tools/**", "models.yaml"]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r eval/requirements.txt
      - run: python eval/run.py --dataset golden.jsonl --out report.json
        env:
          OPENAI_API_KEY: ${{ secrets.CLAUDEXIA_KEY }}
          OPENAI_BASE_URL: https://api.claudexia.tech/v1
      - run: python eval/check_threshold.py report.json --max-regression 0.05

The threshold check is the merge gate. If composite score drops more than 5% versus the baseline on main, the job fails and the PR cannot merge. Five percent is a starting point; tune it per bucket. A 5% regression on your refund-handling bucket may be fine, on your medical-advice bucket it is not.

Alerting on production drift

CI catches regressions you cause. It does not catch regressions Anthropic causes by re-tuning a model under the same name, or regressions caused by your own tool changes. Run the eval suite on a nightly cron against the live model and alert on a 3% composite drop or any safety-axis drop. PagerDuty for safety, Slack for everything else.

Tooling landscape

You do not have to roll your own. The serious options as of 2026:

  • Braintrust — strongest dataset management and judge tooling, decent CI integration, paid SaaS.
  • Promptfoo — open source, YAML-driven, great for unit-style assertions and quick A/B comparisons across models.
  • Langsmith — tightly coupled to LangChain, useful if you already live in that ecosystem, weaker as a standalone harness.
  • DIY — a 300-line Python script and a JSONL file. Most teams start here and stay here longer than they expect.

Pick one and commit. Switching harnesses mid-project is more painful than picking the wrong one.

Bottom line

You cannot ship a Claude agent into production in 2026 and trust that next month's snapshot will behave the same way. Build a 100-case golden dataset from real traffic, score it on a four-axis rubric with Opus as judge, gate merges on a 5% regression threshold, and alert on nightly drift. The investment is one engineer-week up front and an hour a week to maintain. The payoff is shipping prompt and model changes with the same confidence you ship code — which, when your agent is taking actions on behalf of users, is the only confidence that matters.

← Back to all posts