Gaps and fixes

Gaps and fixes turns the Run’s failures into a ranked list of what is wrong with your agent and how to fix it. It is on the Gaps stage (#gaps). The fixes stage fills in each gap’s diagnosis and fix on the same stage, so #fixes opens it too. The gaps the simulation found, and the fixes for them. A gap is one confirmed failure of your agent. Several failing conversations can share one gap.

Every gap, worst first

The top of the section is a ledger: one mark per gap, grouped by severity, most severe first. Point at a mark to see the gap’s title. Select it to jump to the gap.

If the Run found nothing, the section reads No gaps and, for example, Every Eval passed on all 7 scenarios in the Run.

Severity

Severity	Meaning
Critical	Breaks a core promise of your agent, or can cause real harm. Fix before you ship.
High	A serious failure a user will notice.
Medium	A real failure with a limited effect.
Low	A minor issue.
Advisory	Worth knowing; not a failure on its own.
Info	For information only.

The gap list

Each row is one gap:

Column	Meaning
Severity	The gap’s severity.
Gap	A one-line title of what the agent does wrong.
Scenarios hit	How many scenarios it hit in the Run, for example `3 of 7`.
Home	Where the fix belongs. See the next table.
Where	Where in the conversation, or in the code, it shows up.
Fix	Where the fix stands. See Fix status.

Home	Meaning
Agent	A real defect in your agent’s own code or prompt.
Test setup	A problem in the harness, the config, a scenario or an eval, not in your agent.
Mock environment	A problem in the mocked world, such as wrong seeded data or a wrong tool response.

A figure next to the list draws the routes through your agent’s flows, colored by the worst gap on each. The scenarios a gap hit are drawn heavier, so you see which flows a gap lives in. Select a gap to expand it, or link to it with #gap-<gap id>.

Inside a gap

Part	What it shows
State	Confirmed, Potential or Undiagnosed, with the diagnosis’s confidence.
Raised	When it was found: found by the Scan, found while building the Suite or caught in the Run.
Where it broke	The scenarios and turns where the gap hit. Select one to open that conversation at that turn.
Why	The eval it breaks, written as Fails when …. No Eval is tied to this gap when none is.
What we saw	What the agent did, quoted from the conversation.
Evidence	The quotes the diagnosis rests on.
Diagnosis	The root cause, and on which Run it was diagnosed.
Where	The file and line in your code the fix touches.
Proposed fix	The smallest change that should close the gap.
Regression	Whether the gap was kept as a regression. See Kept as regressions.
Verification	Whether the fix held on the Run after the fixes, and its Trail: how that was checked.

A gap from a previous Run says Diagnosed on an earlier Run.

Fix status

Status	Meaning
Tracing	The fixes stage is tracing the gap to the code that causes it.
Fix available	A fix is proposed. Nothing is applied until you accept it.
Fixed / Verified fixed	The fix was applied and the gap did not come back on the Run after the fixes.
Still failing	The fix was applied, but the gap came back on the verify Run.
Not measured	The verify Run could not tell, for example because the scenario didn’t run.

The fixes stage never edits your agent’s code on its own. It proposes each fix, and applies one only after you accept that specific fix in chat. See Fix and keep regressions.

A pass on one Run after a fix can be chance. A verification marked weak signal, or the note Run-to-run noise is unknown, so one Run can’t tell a fix from chance, means you should run the suite again before you trust it.

Potential gaps

Suspected while reading the code or building the Suite. None is confirmed; each needs your call. Below the confirmed gaps, the page lists Potential · may be intended gaps. These are behaviors the scan or the suite build flagged that may be on purpose. Each one shows:

Part	Meaning
What we saw	The behavior, in your agent’s code or in a conversation.
What the Suite did	How the suite handled it, for example by not testing that path.
Why it might be intended	The reason it could be by design.
To confirm	What you need to decide.

Tell the agent in chat whether each one is intended.

Kept as regressions

Each replays the turn its gap was confirmed on, so the next build of the Suite catches it if it comes back. When you accept a fix for an agent gap, the conversation that exposed it is saved as a regression before the fix is applied. The next suite build turns it into a scenario that replays that turn. In the suite, it carries the provenance A Regression.

Label	Meaning
Replays	The gap and the turn it replays, for example confirmed on turn 4 of the recorded conversation.
At its turn:	Consistency of the replay at the exact turn the gap was confirmed on, such as Consistent pass.
Anywhere in the replay:	Consistency of the replay across the whole conversation.
Not in this Suite	The regression is saved, but the current suite doesn’t include it yet. The next suite build adds it.
N gaps not kept	Gaps that were not saved as regressions. Expand to see which.

If nothing was kept, the section says No gap was kept as a regression.

Next steps

Fix and keep regressions

Accept a fix, verify it, and keep the regression.

Run results

Open the conversations behind a gap.

Simulate your agent

Go further

Read your report

Every gap, worst first

Severity

The gap list

Inside a gap

Fix status

Potential gaps

Kept as regressions

Next steps

Fix and keep regressions

Run results

​Every gap, worst first

​Severity

​The gap list

​Inside a gap

​Fix status

​Potential gaps

​Kept as regressions

​Next steps

Fix and keep regressions

Run results

Every gap, worst first

Severity

The gap list

Inside a gap

Fix status

Potential gaps

Kept as regressions

Next steps