> ## Documentation Index
> Fetch the complete documentation index at: https://docs.plurai.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run simulations with the SDK

> Use plurai-run-simulation to play a suite against your harness from a script or CI, and read what it writes.

`plurai-run-simulation` is the runner behind `/simulation-run`. It plays each scenario of a scenario set against
your harness: a simulated user talks to your agent over the scenario's seeded world, the mock environment serves
your agent's tool calls, and the backend scores every eval. Run it yourself to score a suite from CI or a script.

Scenarios and evals are referenced by id. The runner needs only your `simulate.py`, not a local copy of the
scenarios.

## Before you start

* The SDK is installed in your agent's environment. See [Install the SDK](https://docs.plurai.ai/sdk/install.md).
* You have built a suite, so your work folder holds `plurai_sim/.scenario_set_id`, `plurai_sim/.eval_set_id`,
  and `plurai_sim/.mock_service_url` for the deployed mock environment.
* You are signed in (`plurai login --status`), or `PLURAI_URL` and `PLURAI_TOKEN` are exported. The Run refuses to start without one.

## Run the suite

```bash theme={null}
plurai-run-simulation \
  --scenario-set-id "$(cat plurai_sim/.scenario_set_id)" \
  --eval-set-id "$(cat plurai_sim/.eval_set_id)" \
  --agent-module-path plurai_sim/simulate.py \
  --out plurai_sim/runs/nightly
```

Omit `--scenario-id` to run every scenario in the set. Repeat it to run only some:

```bash theme={null}
plurai-run-simulation ... --scenario-id scn_01 --scenario-id scn_02
```

The runner reaches the backend you signed in to, or the one `PLURAI_URL` names. There is no URL or token flag.

Without `--eval-set-id` the Run plays every conversation but scores nothing, and the runner says so at the end.

## What it does

1. Checks that the backend is alive and accepts your token, and that the mock environment answers.
2. For each scenario, opens a conversation on the backend and starts your harness in a child process.
3. Seeds the scenario's world and context, then alternates turns: your agent answers through `invoke_agent`, and
   the backend's simulated user replies.
4. When the conversation ends, sends the transcript, tool events and final world state to the backend, which
   scores every eval.
5. Writes the per-scenario artifacts and a `run_summary.json`, and prints a summary line.

## Flags

### Required

| Flag | Meaning |
| - | - |
| `--scenario-set-id ID` | The scenario set to play. |
| `--agent-module-path PATH` | Your `simulate.py`. |
| `--out DIR` | Where the per-scenario artifacts go. |

### What to run

| Flag | Default | Meaning |
| - | - | - |
| `--scenario-id ID` | every scenario | A scenario to run. Repeat for several. |
| `--eval-set-id ID` | none | The eval set to score. Without it, nothing is scored. |
| `--trials N` | `1` | Run the whole set N times (1–10) as independent Runs. Trial 1 writes to `--out`, trial k to `--out/trial_k/`. `run_summary.json` then reports per scenario and per eval whether it passed consistently, failed consistently or was inconsistent, with a pass rate and a 95% interval. Refused with `--run-phase qualification`. A Run recorded to Arize takes one trial. |
| `--run-phase {qualification,full}` | `full` | `full` is the scored Run. `qualification` is a fail-fast check over a small sample, with shorter time budgets. It sets defaults only and never overrides a budget you set. |
| `--replace` | off | Discard a Run already in `--out`. Without it, running the whole set into a folder that holds a `run_summary.json` is refused. A targeted `--scenario-id` re-run merges into the existing summary and needs no flag. |
| `--no-run-pointer` | off | Leave the workspace's latest-Run pointer unchanged, for a side Run into its own folder. |

### Mock environment

| Flag | Default | Meaning |
| - | - | - |
| `--mock-service-url URL` | see below | The deployed mock environment. Without it: `$NETWORK__MOCK_SERVICE_URL`, then `.mock_service_url` beside `simulate.py`, then the URL the server supplies. A pinned URL is health-checked before the Run, and the Run refuses to start when nothing answers. |
| `--mock-mode {connect,local,mockless}` | `connect` | `connect` talks to a deployed mock and fails when it is unreachable. `local` hosts an in-process mock from your `@plurai.mock` handlers, for local development. `mockless` declares that nothing is mocked. Without the flag: `$NETWORK__MOCK_MODE`, then `.run_mode` beside `simulate.py`. |

### Budgets and retries

| Flag | Default | Meaning |
| - | - | - |
| `--max-turns N` | derived | A cap on conversation turns for every scenario. Derived from the longest scenario in the Run, never below 12. The value and its source are recorded in `run_summary.json`. |
| `--concurrency N` | `4` | Scenarios run at once. Each runs your agent in its own process. Pass `1` for serial. |
| `--agent-turn-timeout SECONDS` | 2160 full, 1560 qualification | Wall-clock budget for one agent turn before the child is stopped. Raise it for an agent that makes many model calls per turn. |
| `--scenario-timeout SECONDS` | 2400 full, 1500 qualification | Budget for one whole scenario. When it runs out, the scenario ends between turns, is still graded on the turns it reached, and is recorded `incomplete`. `--experiment-task-timeout` is the old name. |
| `--http-timeout SECONDS` | 1200 | Read timeout for each backend call. Raise it, not `--retries`, when a turn call times out. |
| `--retries N` | 5 | Retry budget per backend request, with exponential back-off. A timed-out turn call is never retried. |
| `--rerun-on-turn-cap` | off | Re-run a scenario once at double `--max-turns` when it ran out of turns before the simulated user's goal was met. Both attempts are kept. |
| `--stability-retry` / `--no-stability-retry` | on | Take one second sample of each failed scenario to tell a reproducible failure from a flaky one. The first attempt stays authoritative; the row is annotated `stable`, `flaky` or `unmeasured`. |
| `--stability-sample {judge_replay,agent_rerun}` | `judge_replay` full, `agent_rerun` qualification | `agent_rerun` re-runs the scenario end to end. `judge_replay` re-grades the first attempt's transcript with the same evals. |

### Regressions

| Flag | Meaning |
| - | - |
| `--replay-turn-only` | Replay each named regression folder's issue turn only: the recorded conversation is seeded, the cut user message is replayed, your agent answers, and the Run stops. Requires `--scenario-id regression_*`. Refused with `--rerun-on-turn-cap` and `--stability-retry`. |

### Recording and identity

| Flag | Meaning |
| - | - |
| `--observability` / `--no-observability` | Record this Run to the platform your `<SINK>__ENABLED` flag selects, or explicitly record nothing. Not recording is the default. See [Trace sources](https://docs.plurai.ai/integrations/trace-sources.md#record-runs). |
| `--upload-datasets` | Before running, upload the scenario set as a dataset to your recording platform. Implied for LangSmith. |
| `--run-name NAME` | The experiment name. Defaults to a unique stamp, or `$PLURAI_EXPERIMENT_RUN_NAME`. |
| `--command-run-id ID` | Labels the Run's requests so the backend can list every job one command started together. 1 to 128 visible characters, no spaces. Also `$PLURAI_COMMAND_RUN_ID`; the flag wins. The plugin sets it for you. |
| `--agent-version TAG` | Your agent's build tag, stamped on every record so Runs compare across versions. Defaults to `git describe --tags --always --dirty`, or `unknown` outside a git repository. |

### Checks

| Flag | Default | Meaning |
| - | - | - |
| `--egress-probe {connect,dns,off}` | `connect` | How the Run checks that your agent cannot reach the production hosts declared in `plurai.config.yaml`. `connect` opens and closes one TCP connection per host and sends nothing. `dns` only resolves names. `off` only reports the declaration. A reachable production host that nothing isolates holds the Run until you overrule its finding. |
| `--scan-contract FILE` | none | A `scan_contract.json` from `plurai-scan-contract check`. At start, the Run compares the recorded source files against the current ones and names a suite built on documents that have since changed. |
| `--scan-corpus DIR` | none | A corpus root to re-hash for that check. Repeat, in the same order as the recording. |
| `--scan-suffixes SUFFIX` | none | An extra file suffix the recording scan admitted. Needed only for a baseline that recorded no scope. |
| `--overrule FILE` | none | Overrules for this Run's gate findings: a YAML or JSON list of `{gate_id, fingerprint, reason}`. An overrule lapses when the evidence changes. Security and invariant gates are never lifted. |
| `--shape {dialog,pipeline}` | from config | Debugging only. Overrides how the bench is driven; normally `plurai.config.yaml` declares `shape:`. |

## Outputs

Each scenario gets a folder `<out>/<scenario_id>/`:

| File | Contents |
| - | - |
| `transcript.json` | The user and agent messages, the turn count, and eval results. |
| `final_world_state.json` | The world as it stood when the conversation ended. |
| `tool_events.json` | Every tool call: the mock's own log, plus calls recovered from your agent's messages (marked `_recovered: true`), so `real`-mode tools are counted too. |
| `verdicts.json` | The verdict per eval for this conversation. |
| `error.json` | Written instead when the scenario crashed, with the failure's kind and message. |
| `error_dialog.json`, `transcript.partial.json`, `tool_events.partial.json` | Beside `error.json`: what the failed attempt reached before it stopped. |

The Run folder also holds `run_summary.json`: the per-scenario status and verdict, scores, the trials block for
`--trials`, the stability annotations, and the turn cap with its source. A stability retry is kept in the
scenario's `stability_retry/` folder; a turn-cap re-run keeps the superseded attempt under `previous_attempt/`.

## Exit codes

`plurai-run-simulation`, `plurai` and `plurai-scan-contract` share one exit code table. A code means the same thing
everywhere, and each command uses only the rows that apply to it. The names are constants in `plurai.exit_codes`, and
`run_summary.json` carries the same number in its `exit_code` field.

| Code | Name | Meaning |
| - | - | - |
| `0` | `OK` | Done; everything checked held. |
| `1` | `POLICIES_FAILED` | The Run completed and at least one scenario failed an eval. |
| `10` | `UPGRADE_REQUIRED` | The backend refuses this SDK version. The command prints the `pip install -U` line to run. |
| `11` | `MOCK_UNANSWERED` | A mocked tool call could not be answered, so the Run stopped. |
| `12` | `ROUTE_MISSING` | The backend is older than the route the command calls. |
| `13` | `JOB_FAILED` | A backend job ended in failure. |
| `14` | `STILL_RUNNING` | `plurai job wait` stopped waiting while the job is still running. Run it again. |
| `15` | `NOT_FOUND` | The job, Run or artifact does not exist. |
| `16` | `NOT_READY` | The artifact is still a draft. |
| `17` | `BACKEND_REFUSED` | The backend refused what was sent; sending it again fails the same way. |
| `18` | `SCAN_HELD` | `plurai-scan-contract` verified the sources and a finding holds the flow. |
| `19` | `REQUIREMENT_GAP` | A host toolchain check found something missing. |
| `20` | `NOT_MEASURED` | The evals could not be measured: no credit, Plurai at capacity or unavailable, or a suite fault. The report names the cause. |
| `64` | `USAGE` | Bad command line. |
| `65` | `REFUSED` | Declined to proceed: a gate holds the Run, or the input or platform is unusable. |
| `66` | `INPUT_UNREADABLE` | A local input file is missing, empty or unreadable. |
| `69` | `TRANSPORT` | The backend or the mock was unreachable or gave an unusable answer. |
| `70` | `INTERNAL_FAULT` | A defect in the SDK; nothing was concluded. |
| `75` | `TEMPFAIL` | Temporary: the sign-in timed out or a server asked to retry later. Run it again later. |
| `77` | `CREDENTIALS` | No usable login, or the backend rejected the token. |
| `130` | `INTERRUPTED` | You pressed Ctrl-C. The scenarios keep what they had written; no `run_summary.json` is written. |

`2` is never returned.

### When a Run stops early

* **A mock that cannot answer (`11`).** The token is rejected, the mock environment is not deployed, or the URL is not
  a mock. Nothing new starts, no score is reported, and stderr names what to change. A dropped connection is retried
  and does not stop the Run.
* **Credit (`20`).** When your organization is out of credit, nothing new starts. Scenarios that already finished keep
  their verdicts; the rest are recorded as **incomplete**, never as failed. stderr says what happened and, when a payment
  fixes it, points to your billing page. Before stopping, a scenario that is only waiting to start can wait for one of
  the Run's own conversations to end.
* **Capacity or an outage (`20`).** If Plurai is in high demand or unavailable, the Run first waits and retries (a
  short backend restart passes unnoticed). If it stays down, the Run stops with one plain sentence, such as
  *"Plurai is unavailable right now. Please try again in a few minutes."*

In each case `run_summary.json` carries an `aborted` block with the reason, and a stopped Run does not update
`.last_run`.

To fail CI on eval results, fail on exit code `1`, or read the verdicts from `run_summary.json`.

## Run from your own Python code

`plurai.orchestrator.Orchestrator` drives a single scenario from Python. It is part of the supported surface, but the
command-line runner is the recommended entry point: it adds the preflight checks, trials, stability retries and
the summary. See the [API reference](https://docs.plurai.ai/sdk/api-reference.md#orchestrator).

## Next steps

> [card] [Run in CI](https://docs.plurai.ai/guides/ci.md)
>
> Run the suite on every change.

> [card] [Write a harness](https://docs.plurai.ai/sdk/write-a-harness.md)
>
> The `simulate.py` contract the runner drives.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.