Evaluations

Evaluations are scenarios your agent must handle. Each case has an input and checks. WeHarness runs the cases on a version of your code in a fresh sandbox, or judges them on recorded traces, and compares the results with an earlier run. The cases belong to you and WeHarness, not to the code under test: they live in WeHarness, and the code under test cannot change them or mark one passed.

You work with them in two places:

• Build → Evaluations, per application, without a chat. Its tabs are Sets, Runs, Evaluators, Real sessions and Improve agent. It needs the application’s repository.
• A session’s Evals tab, on the session’s repository and draft.

Both show the same cases: cases belong to the repository, so every session and application on it shares them.

Test sets

A repository has a default test set and can have up to 20. Create one with New set in Sets. A new set starts with the default set’s runner. Evaluators attached to a set apply to every case in it.

How a case runs: the runner

The runner is the command that runs one case:

• It reads the case’s input from the file named by WEHARNESS_EVAL_INPUT and prints the agent’s answer, JSON when it can.
• It can list setup commands (installs) to run first.

WeHarness proposes a runner from your repository, or a coding tool proposes one over MCP (eval_runner_propose). Nothing runs until an owner or admin approves it in the signed-in web app. The set then shows How a case runs with the approved command.

Cases

• Ask in a session, in your own words: “set up evaluations; a 120 EUR refund must be escalated”. Or type /eval. WeHarness proposes a runner and cases.
• Write one yourself in Sets (or the session’s Evals tab) with Add case: a name, an input and checks. A case you write is active at once.
• Make one from a trace with Make a test from this on a trace in Tracing. It is pinned to that trace.
• From a correction: when something went wrong and you say what is right, the case comes with a lesson.
• From an automation: an automation can propose a case from a failing trace.

Cases WeHarness, a coding tool or an automation proposes wait until an owner, admin or member approves each one in the web app. API and MCP clients cannot approve. You can retire a case.

Held-out cases

Owners and admins can hold out a case they write (tick Hold it out). WeHarness’s agent then never sees its input, its checks, its name or the traces it produces. It only learns how many held-out cases passed, so a change is also tested on cases the agent was not tuned on. Improve this agent decides with them.

Held-out cases form the Held-out set (protected): only owners and admins see and change them. Runs count them separately; results elsewhere name none of them. An automation never writes a held-out case.

Checks

Most checks are judged by code:

Check	Passes when
exit_code	The command ended with the given code
output_contains, output_excludes	The output has, or does not have, the given text
output_matches	The output matches a regular expression
json_field	A field of the last JSON value printed has the given value
tool_called, tool_not_called	Your agent’s own traces from this case show the tool called, or not
tool_after	Every call of a tool came after a call of another tool that did not fail, for example a refund only after its approval
no_error_spans	No step in your agent’s traces failed
max_duration_ms	The case finished within the given time
judge	A model judge finds that the answer meets your criterion (see below)

Tool and error checks read the traces your agent sends during the run. Use any OpenTelemetry SDK: the sandbox sets the OTLP endpoint and a key that exists only for the run.

Each check passes, fails or is unknown. Unknown means the evidence it needs was not recorded: no traces from the case, no output, no times on the tool calls. Unknown is never a pass.

tool_after checks the order of what happened, not only the end state. A refund issued before its approval fails the check even if an approval came later and the final answer looks right.

Critical checks

tool_not_called and tool_after are critical: they guard what your agent may do. You can mark any other check critical, or a tool_not_called check not critical. A critical check that fails in any attempt fails its case, even when the other attempts passed, and blocks the run’s release gate.

Judge checks

A judge check grades what code cannot, in your own words, in any language: {"type": "judge", "category": "task", "criterion": "It escalates the refund to a lead and says why"}. The category says what the judge grades:

Category	Grades	Critical
task	The answer does what the criterion says, in substance	No
safety	The agent stayed within what it may do: refusing, escalating or waiting for approval where required, exposing nothing it must not. It judges the recorded steps, not the answer’s own account of them	Yes
grounding	Every fact, result or completion the answer claims is supported by the input or the recorded steps	Yes

The judge sees the case input, the output and the steps your agent’s traces recorded, each with an evidence id. All of it is treated as data: text in the output that asks for a better grade changes nothing. It answers pass, fail or unknown, with its probability and the evidence it relied on. An answer without valid evidence, a malformed answer, or no answer at all is unknown.

Code-judged checks run first. When a critical one failed, no judge is asked.

Each category has a pinned judge profile: a model, a prompt version and thresholds, calibrated on a labelled set. A judge counts only after its profile qualified on that set; until then its verdict is shown as advisory and counts for nothing. All three categories qualified on 26 October, on 48 labelled cases in ten languages, judged twice. A new model or prompt version is a new profile and qualifies again before it counts. Judge calls are made by WeHarness’s model provider for the run; they are not your agent’s calls.

Evaluators

The Evaluators tab is the workspace’s evaluator library. An evaluator is a check you can reuse: attach it to a test set and it applies to every case of the set (up to 10 per set).

Kind	What it does	Counts in a release gate
Code	Typed checks, as in the table above	Always
LLM-as-a-judge	One of the three judges grades a criterion you write	When it is a template marked as qualified, or after you calibrate it
Decision model	Elephant-1 answers a bounded question about the attempt: yes or no, one of 2 to 6 choices, or a score on 3 to 7 levels	After you calibrate it

Make one from scratch, or from a template by category: Security, Safety, Quality, Conversation (thread), Trajectory, or Yours. An evaluator that does not count yet is marked advisory · calibrate: its verdicts are shown and kept, never counted.

Calibrate an evaluator

Calibration checks an advisory evaluator against your own judgement:

1. Attach it to a set and run the set, so it has verdicts.
2. Open Calibrate. For each recorded verdict, choose Should pass or Should fail.
3. When the labels meet the rule, an owner or admin selects Let it count in gates.

The rule: at least 12 labelled results, with at least 3 that should pass and 3 that should fail; no false pass (it passed a result you labelled wrong); at most 1 false fail; at most 2 undecided verdicts. Held-out cases are left out of calibration.

Runs

In Sets, choose what a run uses (Run on):

Run on	What runs
Branch head	The newest commit of a branch of the application’s GitHub repository
Commit or tag	That exact commit
A session’s draft	A session’s draft on the repository, as it is now
Recorded traces	No code: each case pinned to a trace is judged on that trace

A run on code executes every approved case in a fresh sandbox, with the same setup as a clean check and the approved runner. Model calls your agent makes go through WeHarness with the sandbox’s own metered allowance. In a session, /eval runs the cases on the draft.

The approved cases also run on every report of a plan’s task, from your coding tool or the WeHarness cloud, in the sandbox that reproduces the report. The run’s release gate decides the report. See Working on an approved plan.

Result	Meaning
Passed / failed	Every attempt of the case passed, or none did (or a critical check failed in one)
Flaky	Some attempts passed and some failed
Inconclusive	Nothing failed, but some evidence was not recorded
Regressed	Passed in the previous run and fails or is flaky now
Fixed	Failed in the previous run and passes now

The release gate

Every run has one verdict, never an average of scores:

Gate	When
Blocked	A critical check failed in some attempt of some case
Failed	A mandatory case failed or is flaky
Inconclusive	A mandatory case lacks the evidence to decide
Passed	Every mandatory case passed

A run also verifies the lessons its cases show.

Compare runs

In Runs, tick two finished runs to compare them case by case: Regressed, Fixed, Flaky, Inconclusive, New and Unchanged. Held-out cases appear only as counts. Investigate in a session puts a failing case in a session’s message box as a reference; nothing is sent until you send a message.

Replays

A replay judges cases without running any code:

• On recorded traces. A case pinned to a trace is judged on that real run: its steps, and its input and output when the key keeps content.
• On an earlier run. Judge these attempts again re-judges what an earlier run recorded (its output, exit code and traces) with the cases and judges as they are now.

A case with no recording, or a trace that is no longer kept, is inconclusive. Replays never verify lessons: only a run of your code does.

Reviewing real sessions

Real sessions shows traces your application sent in the last 7 days, for you to review:

• every trace that failed;
• 10 in 100 of the others, chosen by the trace id, so the same trace is always in or out;
• never the traces of evaluation runs.

It also shows how many traces there were and how many failed. A trace nobody complained about is not proof that your agent did the right thing.

Mark each trace as looks right, or something’s wrong with what is wrong. Learn from it puts /learn with the trace and your note in a session’s message box. When you send it, WeHarness reads the trace and proposes a lesson with that trace as evidence.

Credits

	Credits
A run on code	Its sandbox and the models your agent calls in it, by what they used, never more than the run reserved
Judges	By what their calls used, at the judge model’s price, within the run’s judges line. A failed run’s judges use none
A replay on recorded traces	Its judges only
Reading runs and results	None

The run’s quote shows the judges line before it starts. See Credits and usage.

Over MCP

A coding tool reads sets, cases and runs (evals_get, eval_run_get, eval_runs_list, eval_review_get), proposes cases and runners (eval_case_propose, eval_runner_propose), starts runs a person’s approvals already cover (eval_run_start), and adds evaluators and sets (evaluator_create, eval_set_create). It sees held-out cases only as counts, and never approves or calibrates. See Tools and prompts reference.