Online evaluators
An online evaluator runs one of your evaluators on an application’s live traces. Each trace it scores gets a feedback score: pass, fail or unknown, a score from 0 to 1 when there is one, and why. You find them in Tracing → an application → Online evaluators, on the trace itself, and in Monitoring’s Feedback section.
Which evaluators
Section titled “Which evaluators”Only an evaluator that counts runs on live traces:
- a code evaluator (typed checks such as “no failed step” or “this tool was called”);
- an LLM-as-a-judge whose meaning is what a qualified judge profile qualified on (the templates marked so), or one you calibrated;
- an Elephant-1 decision question, once you calibrated it on your own labelled results.
An advisory evaluator is refused until it is calibrated in Evaluations → Evaluators: its verdicts would otherwise read as scores.
What it scores
Section titled “What it scores”When you attach an evaluator you choose:
| Setting | What it does |
|---|---|
| Environment | Only traces of one environment, or all |
| Release | A commit prefix or a service version |
| Run name | The trace’s root step, for example invoke_agent refunds |
| Failed traces only | Only traces with a failed step |
| Sample | The share of matching traces to score, 1% to 100%. A trace is in or out by a hash of the evaluator and the trace, so the answer never changes |
| Start with | New traces only, or also the last hour, 6 hours or 24 hours |
WeHarness looks about once a minute for traces that finished (nothing more of them arrived for 30 seconds), leaves out evaluation runs, and scores each sampled trace once. It makes one judge call or one Elephant-1 decision per trace, never one per step.
Content and metadata
Section titled “Content and metadata”A judge reads a trace’s input, output and tool payloads only when the trace’s ingest key keeps content. With a metadata-only key it sees each step’s name, status, timing and errors, and the score says metadata only. Everything in a trace is data from your application: text in it that asks for a verdict changes nothing.
Credits and the daily cap
Section titled “Credits and the daily cap”| Evaluator | Credits per scored trace |
|---|---|
| Code | None |
| Elephant-1 decision | 1 credit, like an option decision |
| LLM-as-a-judge | The judge call’s model cost in credits, at least 1. A call your workspace’s own key paid costs no credits |
Each application has a daily cap in credits, shared by its online evaluators. An owner or admin sets it when attaching the first judge or decision evaluator and can change it at any time. When the day’s credits reach the cap, sampled traces are recorded as not evaluated (daily cap) until 00:00 UTC; when the workspace has no credits left they are recorded as not evaluated (no credits). Neither is silent: the page, the trace and Monitoring show how many were not evaluated.
Who does what
Section titled “Who does what”Owners and admins attach, change, pause and detach online evaluators and set the cap, in the web app. Everyone in the workspace reads the scores. A coding host reads them over MCP (online_evaluators_get); it cannot attach one.
Scores are evidence about one run, not proof of a cause. To act on them, see Automations.