Run simulations with the SDK

plurai-run-simulation is the runner behind /simulation-run. It plays each scenario of a scenario set against your harness: a simulated user talks to your agent over the scenario’s seeded world, the mock environment serves your agent’s tool calls, and the backend scores every eval. Run it yourself to score a suite from CI or a script. Scenarios and evals are referenced by id. The runner needs only your simulate.py, not a local copy of the scenarios.

Before you start

The SDK is installed in your agent’s environment. See Install the SDK.
You have built a suite, so your work folder holds plurai_sim/.scenario_set_id, plurai_sim/.eval_set_id, and plurai_sim/.mock_service_url for the deployed mock environment.
You are signed in (plurai login --status), or PLURAI_URL and PLURAI_TOKEN are exported. The Run refuses to start without one.

Run the suite

plurai-run-simulation \
  --scenario-set-id "$(cat plurai_sim/.scenario_set_id)" \
  --eval-set-id "$(cat plurai_sim/.eval_set_id)" \
  --agent-module-path plurai_sim/simulate.py \
  --out plurai_sim/runs/nightly

Omit --scenario-id to run every scenario in the set. Repeat it to run only some:

plurai-run-simulation ... --scenario-id scn_01 --scenario-id scn_02

The runner reaches the backend you signed in to, or the one PLURAI_URL names. There is no URL or token flag. Without --eval-set-id the Run plays every conversation but scores nothing, and the runner says so at the end.

What it does

Checks that the backend is alive and accepts your token, and that the mock environment answers.
For each scenario, opens a conversation on the backend and starts your harness in a child process.
Seeds the scenario’s world and context, then alternates turns: your agent answers through invoke_agent, and the backend’s simulated user replies.
When the conversation ends, sends the transcript, tool events and final world state to the backend, which scores every eval.
Writes the per-scenario artifacts and a run_summary.json, and prints a summary line.

Flags

Required

Flag	Meaning
`--scenario-set-id ID`	The scenario set to play.
`--agent-module-path PATH`	Your `simulate.py`.
`--out DIR`	Where the per-scenario artifacts go.

What to run

Flag	Default	Meaning
`--scenario-id ID`	every scenario	A scenario to run. Repeat for several.
`--eval-set-id ID`	none	The eval set to score. Without it, nothing is scored.
`--trials N`	`1`	Run the whole set N times (1–10) as independent Runs. Trial 1 writes to `--out`, trial k to `--out/trial_k/`. `run_summary.json` then reports per scenario and per eval whether it passed consistently, failed consistently or was inconsistent, with a pass rate and a 95% interval. Refused with `--run-phase qualification`. A Run recorded to Arize takes one trial.
`--run-phase {qualification,full}`	`full`	`full` is the scored Run. `qualification` is a fail-fast check over a small sample, with shorter time budgets. It sets defaults only and never overrides a budget you set.
`--replace`	off	Discard a Run already in `--out`. Without it, running the whole set into a folder that holds a `run_summary.json` is refused. A targeted `--scenario-id` re-run merges into the existing summary and needs no flag.
`--no-run-pointer`	off	Leave the workspace’s latest-Run pointer unchanged, for a side Run into its own folder.

Mock environment

Flag	Default	Meaning
`--mock-service-url URL`	see below	The deployed mock environment. Without it: `$NETWORK__MOCK_SERVICE_URL`, then `.mock_service_url` beside `simulate.py`, then the URL the server supplies. A pinned URL is health-checked before the Run, and the Run refuses to start when nothing answers.
`--mock-mode {connect,local,mockless}`	`connect`	`connect` talks to a deployed mock and fails when it is unreachable. `local` hosts an in-process mock from your `@plurai.mock` handlers, for local development. `mockless` declares that nothing is mocked. Without the flag: `$NETWORK__MOCK_MODE`, then `.run_mode` beside `simulate.py`.

Budgets and retries

Flag	Default	Meaning
`--max-turns N`	derived	A cap on conversation turns for every scenario. Derived from the longest scenario in the Run, never below 12. The value and its source are recorded in `run_summary.json`.
`--concurrency N`	`4`	Scenarios run at once. Each runs your agent in its own process. Pass `1` for serial.
`--agent-turn-timeout SECONDS`	2160 full, 1560 qualification	Wall-clock budget for one agent turn before the child is stopped. Raise it for an agent that makes many model calls per turn.
`--scenario-timeout SECONDS`	2400 full, 1500 qualification	Budget for one whole scenario. When it runs out, the scenario ends between turns, is still graded on the turns it reached, and is recorded `incomplete`. `--experiment-task-timeout` is the old name.
`--http-timeout SECONDS`	1200	Read timeout for each backend call. Raise it, not `--retries`, when a turn call times out.
`--retries N`	5	Retry budget per backend request, with exponential back-off. A timed-out turn call is never retried.
`--rerun-on-turn-cap`	off	Re-run a scenario once at double `--max-turns` when it ran out of turns before the simulated user’s goal was met. Both attempts are kept.
`--stability-retry` / `--no-stability-retry`	on	Take one second sample of each failed scenario to tell a reproducible failure from a flaky one. The first attempt stays authoritative; the row is annotated `stable`, `flaky` or `unmeasured`.
`--stability-sample {judge_replay,agent_rerun}`	`judge_replay` full, `agent_rerun` qualification	`agent_rerun` re-runs the scenario end to end. `judge_replay` re-grades the first attempt’s transcript with the same evals.

Regressions

Flag	Meaning
`--replay-turn-only`	Replay each named regression folder’s issue turn only: the recorded conversation is seeded, the cut user message is replayed, your agent answers, and the Run stops. Requires `--scenario-id regression_*`. Refused with `--rerun-on-turn-cap` and `--stability-retry`.

Recording and identity

Flag	Meaning
`--observability` / `--no-observability`	Record this Run to the platform your `<SINK>__ENABLED` flag selects, or explicitly record nothing. Not recording is the default. See Trace sources.
`--upload-datasets`	Before running, upload the scenario set as a dataset to your recording platform. Implied for LangSmith.
`--run-name NAME`	The experiment name. Defaults to a unique stamp, or `$PLURAI_EXPERIMENT_RUN_NAME`.
`--command-run-id ID`	Labels the Run’s requests so the backend can list every job one command started together. 1 to 128 visible characters, no spaces. Also `$PLURAI_COMMAND_RUN_ID`; the flag wins. The plugin sets it for you.
`--agent-version TAG`	Your agent’s build tag, stamped on every record so Runs compare across versions. Defaults to `git describe --tags --always --dirty`, or `unknown` outside a git repository.

Checks

Flag	Default	Meaning
`--egress-probe {connect,dns,off}`	`connect`	How the Run checks that your agent cannot reach the production hosts declared in `plurai.config.yaml`. `connect` opens and closes one TCP connection per host and sends nothing. `dns` only resolves names. `off` only reports the declaration. A reachable production host that nothing isolates holds the Run until you overrule its finding.
`--scan-contract FILE`	none	A `scan_contract.json` from `plurai-scan-contract check`. At start, the Run compares the recorded source files against the current ones and names a suite built on documents that have since changed.
`--scan-corpus DIR`	none	A corpus root to re-hash for that check. Repeat, in the same order as the recording.
`--scan-suffixes SUFFIX`	none	An extra file suffix the recording scan admitted. Needed only for a baseline that recorded no scope.
`--overrule FILE`	none	Overrules for this Run’s gate findings: a YAML or JSON list of `{gate_id, fingerprint, reason}`. An overrule lapses when the evidence changes. Security and invariant gates are never lifted.
`--shape {dialog,pipeline}`	from config	Debugging only. Overrides how the bench is driven; normally `plurai.config.yaml` declares `shape:`.

Outputs

Each scenario gets a folder <out>/<scenario_id>/:

File	Contents
`transcript.json`	The user and agent messages, the turn count, and eval results.
`final_world_state.json`	The world as it stood when the conversation ended.
`tool_events.json`	Every tool call: the mock’s own log, plus calls recovered from your agent’s messages (marked `_recovered: true`), so `real`-mode tools are counted too.
`verdicts.json`	The verdict per eval for this conversation.
`error.json`	Written instead when the scenario crashed, with the failure’s kind and message.
`error_dialog.json`, `transcript.partial.json`, `tool_events.partial.json`	Beside `error.json`: what the failed attempt reached before it stopped.

The Run folder also holds run_summary.json: the per-scenario status and verdict, scores, the trials block for --trials, the stability annotations, and the turn cap with its source. A stability retry is kept in the scenario’s stability_retry/ folder; a turn-cap re-run keeps the superseded attempt under previous_attempt/.

Exit codes

plurai-run-simulation, plurai and plurai-scan-contract share one exit code table. A code means the same thing everywhere, and each command uses only the rows that apply to it. The names are constants in plurai.exit_codes, and run_summary.json carries the same number in its exit_code field.

Code	Name	Meaning
`0`	`OK`	Done; everything checked held.
`1`	`POLICIES_FAILED`	The Run completed and at least one scenario failed an eval.
`10`	`UPGRADE_REQUIRED`	The backend refuses this SDK version. The command prints the `pip install -U` line to run.
`11`	`MOCK_UNANSWERED`	A mocked tool call could not be answered, so the Run stopped.
`12`	`ROUTE_MISSING`	The backend is older than the route the command calls.
`13`	`JOB_FAILED`	A backend job ended in failure.
`14`	`STILL_RUNNING`	`plurai job wait` stopped waiting while the job is still running. Run it again.
`15`	`NOT_FOUND`	The job, Run or artifact does not exist.
`16`	`NOT_READY`	The artifact is still a draft.
`17`	`BACKEND_REFUSED`	The backend refused what was sent; sending it again fails the same way.
`18`	`SCAN_HELD`	`plurai-scan-contract` verified the sources and a finding holds the flow.
`19`	`REQUIREMENT_GAP`	A host toolchain check found something missing.
`20`	`NOT_MEASURED`	The evals could not be measured: no credit, Plurai at capacity or unavailable, or a suite fault. The report names the cause.
`64`	`USAGE`	Bad command line.
`65`	`REFUSED`	Declined to proceed: a gate holds the Run, or the input or platform is unusable.
`66`	`INPUT_UNREADABLE`	A local input file is missing, empty or unreadable.
`69`	`TRANSPORT`	The backend or the mock was unreachable or gave an unusable answer.
`70`	`INTERNAL_FAULT`	A defect in the SDK; nothing was concluded.
`75`	`TEMPFAIL`	Temporary: the sign-in timed out or a server asked to retry later. Run it again later.
`77`	`CREDENTIALS`	No usable login, or the backend rejected the token.
`130`	`INTERRUPTED`	You pressed Ctrl-C. The scenarios keep what they had written; no `run_summary.json` is written.

2 is never returned.

When a Run stops early

A mock that cannot answer (11). The token is rejected, the mock environment is not deployed, or the URL is not a mock. Nothing new starts, no score is reported, and stderr names what to change. A dropped connection is retried and does not stop the Run.
Credit (20). When your organization is out of credit, nothing new starts. Scenarios that already finished keep their verdicts; the rest are recorded as incomplete, never as failed. stderr says what happened and, when a payment fixes it, points to your billing page. Before stopping, a scenario that is only waiting to start can wait for one of the Run’s own conversations to end.
Capacity or an outage (20). If Plurai is in high demand or unavailable, the Run first waits and retries (a short backend restart passes unnoticed). If it stays down, the Run stops with one plain sentence, such as “Plurai is unavailable right now. Please try again in a few minutes.”

In each case run_summary.json carries an aborted block with the reason, and a stopped Run does not update .last_run. To fail CI on eval results, fail on exit code 1, or read the verdicts from run_summary.json.

Run from your own Python code

plurai.orchestrator.Orchestrator drives a single scenario from Python. It is part of the supported surface, but the command-line runner is the recommended entry point: it adds the preflight checks, trials, stability retries and the summary. See the API reference.

Next steps

Run in CI

Run the suite on every change.

Write a harness

The simulate.py contract the runner drives.

Use the SDK

Reference

Before you start

Run the suite

What it does

Flags

Required

What to run

Mock environment

Budgets and retries

Regressions

Recording and identity

Checks

Outputs

Exit codes

When a Run stops early

Run from your own Python code

Next steps

Run in CI

Write a harness

​Before you start

​Run the suite

​What it does

​Flags

​Required

​What to run

​Mock environment

​Budgets and retries

​Regressions

​Recording and identity

​Checks

​Outputs

​Exit codes

​When a Run stops early

​Run from your own Python code

​Next steps

Run in CI

Write a harness

Before you start

Run the suite

What it does

Flags

Required

What to run

Mock environment

Budgets and retries

Regressions

Recording and identity

Checks

Outputs

Exit codes

When a Run stops early

Run from your own Python code

Next steps