> ## Documentation Index
> Fetch the complete documentation index at: https://docs.plurai.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Build the suite

> Stand up your agent's mock environment and knowledge corpus, then generate the scenarios and evals that make up your suite.

Building the suite is where Plurai decides what to test your agent on. It:

* **Deploys the digital twin:** stands up your [digital twin](https://docs.plurai.ai/concepts.md#param-digital-twin), and a
  [knowledge corpus](https://docs.plurai.ai/concepts.md#param-knowledge-base) if your agent answers from documents, so every scenario starts from a
  realistic world.
* **Generates the scenarios:** realistic [scenarios](https://docs.plurai.ai/concepts.md#param-scenario) that walk the routes on your
  [agent flow graph](https://docs.plurai.ai/concepts.md#param-agent-flow-graph) where your agent is most likely to break, each with a
  [persona](https://docs.plurai.ai/concepts.md#param-persona) or a task input.
* **Writes the evals:** the [evals](https://docs.plurai.ai/concepts.md#param-eval) that decide what your agent must get right in each scenario.
* **Checks the suite itself:** an independent review makes sure the suite produces sharp signal before your
  agent is tested on it.

The [suite](https://docs.plurai.ai/concepts.md#param-suite) is the scenarios and evals. The digital twin and the corpus are what they play against.

[1 · Scan](https://docs.plurai.ai/guides/scan-your-agent.md)
→
[2 · Ground (optional)](https://docs.plurai.ai/guides/ground-in-production-traces.md)
→
[3 · Build the suite](https://docs.plurai.ai/guides/build-the-suite.md)
→
[4 · Run](https://docs.plurai.ai/guides/run-the-simulation.md)
→
[5 · Fix](https://docs.plurai.ai/guides/fix-and-keep-regressions.md)

**Runs** as part of `/simulate`, or on its own with `/simulation-generate-suite` after a scan.

## What you get

![Your test suite in the report](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/report/suite.png?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=3634b5b3ffcfc88466a5ff8d6da88a8b)

The Suite stage of the report: select a scenario and its route lights up on your agent flow graph.

The **Your test suite** section shows the agent flow graph your scenarios walk, every scenario, and every eval.
If your agent answers from documents, a **Tested knowledge** tab shows exactly which information the suite
uses from your knowledge base: the passages its questions are answered from, and the evals that grade what
your agent says about them.

![Tested knowledge in the report](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/report/tested-knowledge.png?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=467773b8deb015ba96963e1ed1ae1faa)

Tested knowledge: the passages of your knowledge base the suite asks about, in the documents they come from.

The hand-off opens with the suite's status:

| Status | Meaning |
| - | - |
| **Suite: validated** | The quality check came back clean. |
| **Suite: qualified** | The quality check passed with accepted gaps. |
| **Suite: not yet qualified** | The suite is static only, or blockers remain. The hand-off says why. |

For example:

> **Suite: validated** — 7 scenarios · 12 evals · \~80% flow coverage · quality gate ✓. Catches invented refund timelines and skipped identity checks.

Then, when they apply: a **Run integrity** line, regressions, capability gaps, issues the build **already caught** in your agent, and suggestions for a **sharper next round** (production traces, real records, document samples).

Files in `plurai_sim/`:

| File | What it is |
| - | - |
| `artifacts/scenarios.md` | Every scenario: goal, persona, opening line and seeded world. |
| `artifacts/evals.md` | Every eval and when it passes. |
| `artifacts/scenarios/<id>.md` | One breakdown per scenario. |
| `artifacts/charts/*.svg` | Coverage, complexity and environment charts. |
| `artifacts/suite.json` | The suite in machine-readable form. |
| `policies_config.yaml` | A local copy of the evals. The run scores the copy stored on the Plurai backend. |
| `.scenario_set_id`, `.eval_set_id` | The backend IDs of your scenarios and evals. Commit these. |

`scenarios.md`, `evals.md` and `suite.json` are staged in git so anyone who clones the repo can read the suite. See [The suite in the report](https://docs.plurai.ai/report/suite.md).

## Build it

> [!NOTE]
>
> You don't have to run this on its own. `/simulate <path-to-your-agent>` runs the suite build as part of the full
> flow. Use the command below to run or re-run just this stage.

```bash theme={null}
/simulation-generate-suite
```

Or say it in chat: "Build the test suite."

| Argument | What it does |
| - | - |
| `[context]` | Optional. A one-line prompt, or a path to a PRD or operating-policy `.md` file, to steer the scenarios. Without it, scenarios come from what the scan learned about your agent's domain. |
| `--zero-shot` | Build without extra context. Same as leaving `[context]` out. |
| `--no-auto-regressions` | Don't record the agent issues the suite confirms as regressions. |

Before it starts, the plugin checks that your agent's model-provider key is in the gitignored `.env` in your agent's folder, because the suite runs your agent to check itself.

## How long it takes

The suite build runs on its own and is measured in hours, not minutes. The plugin posts progress as it goes, for example "Suite: in progress — about 10-20 minutes; this session picks up again by itself."

| Phase | Typical time |
| - | - |
| Stand up the mock environment, first time | 10–60 minutes, longer for a very large set of tools |
| Stand up the mock environment, later builds | Seconds |
| First sample of 3 scenarios | 10–30 minutes |
| Quality check | 40–70 minutes, usually the longest phase |
| Final 7 scenarios | 20–70 minutes |

## What happens

**1. Knowledge corpus (if your agent needs one)**

If your agent answers from a knowledge base and no corpus exists yet, the plugin asks whether to build one now. Without it, the suite contains no retrieval scenarios and your knowledge base is never tested. On yes, it builds the corpus from your documents. See [Knowledge base](https://docs.plurai.ai/guides/knowledge-base.md).

**2. Mock environment**

The plugin generates and deploys your agent's mock environment: the mocked tools and the world state they serve. On a later build it brings the existing one back up by ID. An agent with nothing to mock skips this step.

**3. A first sample of scenarios and the evals**

It generates a small, diverse sample of 3 scenarios and a mixed set of evals.

**4. Quality check**

It runs your agent on the sample, then an independent reviewer checks the scenarios, evals, mock environment and harness for weak signal. Problems in the suite are repaired and checked again. Problems in your agent are recorded in `agent_findings.md`, not fixed here.

**5. The final suite**

With the checked setup, it generates the final 7 scenarios and links the evals to them. Recorded [regressions](https://docs.plurai.ai/guides/fix-and-keep-regressions.md) join as extra scenarios.

**6. The report**

It renders the scenarios, evals and coverage charts in the report and hands off with a summary.

## Questions you may be asked

The build runs on its own and asks only when it needs something only you hold:

* **Build the knowledge corpus?** Asked when your agent has a knowledge source and no corpus.
* **A missing prerequisite**, such as an unset or expired model-provider key. You can supply it, or choose a static suite that is built and reviewable but not run.
* **Apply a fix to your harness?** See below.
* **Real data needed.** If the scenarios can't be grounded faithfully without real records, the plugin asks for them (see [Real records](https://docs.plurai.ai/guides/real-records.md)) and does not mark the suite as validated.

### Fixes to your harness

`simulate.py`, its helpers, and any runner of your own declared as `harness.entry_wrapper` are your files. When the quality check finds a fix that belongs in one of them, the plugin asks first. It names the file, shows the diff and gives a one-line reason, and waits.

If you decline, nothing is written. The finding stays open and is reported, and the suite is delivered without the **validated** mark.

In CI, where nobody can answer, harness edits are declined unless you approve them in advance with `approve_harness_edits: true` in `plurai.config.yaml`. See [Run unattended](https://docs.plurai.ai/guides/ci.md).

## Next steps

Run `/simulation-run` to score your agent against the suite.

> [card] [Run the simulation](https://docs.plurai.ai/guides/run-the-simulation.md)
>
> Play every scenario and score every eval.

> [card] [The suite in the report](https://docs.plurai.ai/report/suite.md)
>
> Review scenarios, evals and coverage.

> [card] [Shape your simulation](https://docs.plurai.ai/guides/iterate.md)
>
> Change a scenario, an eval or the mock environment.

> [card] [Ground in production traces](https://docs.plurai.ai/guides/ground-in-production-traces.md)
>
> Make the next build reflect real behavior.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.