Automations

An automation is a rule on an application’s traces, in Tracing → an application → Automations.

When it acts

Trigger	What starts it
A trace fails	A trace with a failed step
An online evaluator scores a trace low	That online evaluator’s verdict was fail, or its score is under the value you set

You can narrow a rule to an environment, a release (a commit prefix or service version) and a run name (the trace’s root step).

What it does

Choose one or more actions:

• Propose a test case in a test set of the application’s repository. The case is pinned to the trace, with typed checks from its steps (no failed step, and the tools that worked were called). It waits for a person to approve it in Evaluations. An automation never writes a held-out case: only a person does.
• Add the trace to the review list. People mark each item reviewed or dismiss it. The sidebar shows how many wait.
• Raise an alert. It shows in Monitoring’s Alerts, and the rule’s maker gets an email unless they turned it off.
• Prepare a session with the trace as a reference. Nothing starts by itself: opening it puts the trace in a session’s message box, and the session runs only when you send a message.

Nothing an automation does changes your code, your lessons or a run.

Limits and records

A rule acts on a trace at most once, and on at most the number of traces an hour you set (1 to 60). Past it, the trace is recorded as over the hourly limit, with no action. A rule acts only on traces from after it was activated. What the automations did lists every trace a rule acted on, with the outcome of each action.

Who does what

Owners and admins make, activate, pause and delete automations, in the web app. A member, or a coding host over MCP (automation_propose), only proposes one: it does nothing until an owner or admin activates it. Everyone in the workspace sees the rules, the review list and what they did.