> ## Documentation Index > Fetch the complete documentation index at: https://docs.plurai.ai/llms.txt > Use this file to discover all available pages before exploring further. # Run the simulation > Have a simulated user play every scenario against your agent, score every eval, and read the results. The [run](https://docs.plurai.ai/concepts.md#param-run) is where you find out how your agent really behaves. It: * **Records into your observability tool:** every scenario lands as a trace, with its eval scores. * **Plays every scenario** on your [digital twin](https://docs.plurai.ai/concepts.md#param-digital-twin): a simulated user plays each [persona](https://docs.plurai.ai/concepts.md#param-persona) in a conversation, and a task or workflow agent gets the scenario's input. * **Repeats each one three times:** to separate consistent failures from flaky ones. * **Scores every [eval](https://docs.plurai.ai/concepts.md#param-eval):** on every conversation or result. * **Shows the evidence:** the conversation, your agent's steps and every tool call. ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/langsmith.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=038c05ac100fccf3f2b6c29e78571cc1)LangSmith ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/langfuse.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=296da0ea32ddeb4ff4427f0cdf841b1e)Langfuse ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/arize.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=db8a516fdd0ac42ab39c6ea1cdb4375f)Arize ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/braintrust.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=e96f38394bb8141429b53f0a99919f01)Braintrust The run scores your agent. Finding out why it failed is the next stage, Fix. [1 · Scan](https://docs.plurai.ai/guides/scan-your-agent.md) → [2 · Ground (optional)](https://docs.plurai.ai/guides/ground-in-production-traces.md) → [3 · Build the suite](https://docs.plurai.ai/guides/build-the-suite.md) → [4 · Run](https://docs.plurai.ai/guides/run-the-simulation.md) → [5 · Fix](https://docs.plurai.ai/guides/fix-and-keep-regressions.md) **Runs** as part of `/simulate`, or on its own with `/simulation-run`. Re-run it after every change to your agent. ## What you get ![Simulation run results in the report](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/report/run.png?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=49c4f85cfc5dd575d58723e1a188813f) The Run stage of the report: every scenario played three times, with the evals that failed. Every scenario comes with the full evidence of what happened: | You see | For | | - | - | | **The conversation**, turn by turn, between the simulated user and your agent | Conversational agents | | **Your agent's actions**: each step it took, in order | Every agent | | **The tools it used**: every call, with its inputs and what the tool returned | Every agent | | **The verdicts**: which evals passed or failed, and why | Every agent | You see all of it in the report, and in your own observability tool if you connect one (see [below](#record-it-in-your-observability-platform)). The chat summary looks like this: > **Run complete** — 7 scenarios (3 trials each): ✅ 15 of 21 trials passed · ❌ 4 of 21 failed a critical check · ⚠ 2 of 21 failed a high-severity check. It continues with what failed (in your domain's language, with scenario IDs), the run integrity line if any, and where the results are. | Where | What | | - | - | | The report | The run's results. See [Run results](https://docs.plurai.ai/report/run-results.md). | | `plurai_sim/artifacts/runs//` | One folder per scenario with `transcript.json`, `tool_events.json`, `verdicts.json`, and `error.json` on failure, plus `run_summary.json`. | | `analytics.md` and `charts/` | A results summary, written when the run was not recorded as a LangSmith experiment. | | Your observability platform | The dataset, experiment and per-scenario trace links, printed at the end. | Run folders are gitignored. The run's saved artifacts are also uploaded to the Plurai backend so failures can be attributed. See [Data and privacy](https://docs.plurai.ai/security/data-and-privacy.md). ## Start a run > [!NOTE] > > You don't have to run this on its own. `/simulate ` runs the run as part of the full > flow. Use the command below to run or re-run just this stage. ```bash theme={null} /simulation-run ``` Or say it in chat: "Run the simulation." | Argument | What it does | | - | - | | `--agent-module-path ` | The harness to run. Defaults to `simulate.py` in your `plurai_sim/` folder. | | `--experiment-name ` | A name for this run. A timestamp is added. Without it, the run gets a unique name. | | `--max-concurrent ` | How many scenarios run at once. Defaults to 4. | | `--agent-version ` | A version label for the agent under test. | ## Before it starts The run checks its prerequisites before it spends anything: * A finished suite. If `/simulation-generate-suite` did not complete, run it first. * A live mock environment. If it has lapsed, the plugin redeploys it by ID. Agents with nothing to mock skip this. * **Your agent's model-provider key** (for example `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`) in the gitignored `.env` file in your agent's folder. An exported key does not reach the run. The run scores the eval set stored on the Plurai backend. Editing `policies_config.yaml` by hand changes nothing; to change an eval, say so in chat. See [Shape your simulation](https://docs.plurai.ai/guides/iterate.md). ## Record it in your observability platform Plurai works with the observability tool you already use: **LangSmith, Langfuse, Arize or Braintrust**. Connect one, and each simulated scenario is recorded there just like a production trace: the conversation, every agent step and tool call, and the eval scores attached as feedback. You can debug a simulated failure with the same tools and views you use for real ones, and compare runs over time. ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/langsmith.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=038c05ac100fccf3f2b6c29e78571cc1)LangSmith ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/langfuse.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=296da0ea32ddeb4ff4427f0cdf841b1e)Langfuse ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/arize.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=db8a516fdd0ac42ab39c6ea1cdb4375f)Arize ![](https://mintcdn.com/plurai/V3XeGRFO5iyJejs5/images/logos/braintrust.svg?fit=max\&auto=format\&n=V3XeGRFO5iyJejs5\&q=85\&s=e96f38394bb8141429b53f0a99919f01)Braintrust When you run `/simulation-run` yourself, the plugin asks: > Record this run in your observability platform? > > 1. **As a tracked experiment** *(recommended)* — one record per scenario, eval scores attached as typed feedback, comparable across runs. > 2. **Just traces** — emit the per-scenario traces without the dataset-backed experiment grouping. > 3. **No observability** — run and score locally only. LangSmith, Langfuse, Arize and Braintrust are supported, one at a time. If a key is missing, the plugin asks for it. Put it in your agent folder's `.env`: ```bash LangSmith theme={null} LANGSMITH__API_KEY=... LANGSMITH__PROJECT=... # optional ``` ```bash Langfuse theme={null} LANGFUSE__SECRET_KEY=... LANGFUSE__PUBLIC_KEY=... LANGFUSE__HOST=... # optional, for self-hosted Langfuse ``` ```bash Arize theme={null} ARIZE__API_KEY=... ARIZE__SPACE_ID=... ``` ```bash Braintrust theme={null} BRAINTRUST__API_KEY=... ``` If you can't provide a key, the run continues without observability. In LangSmith, every run of the same scenario set lands under one dataset, with each run as its own experiment. Recording is best effort: a failed upload never changes the result. See [Trace sources](https://docs.plurai.ai/integrations/trace-sources.md). ## What happens in a run **1. Each scenario starts in your sandbox** Your agent runs on your machine, with its real components. Every call it makes to a mocked system goes to your private sandbox in Plurai's cloud, where the scenario starts from its own seeded world. **2. The simulated user plays each scenario** For a conversational agent, the simulated user is a user agent we provide. It plays the scenario's persona, opens with its first line and pursues its goal over several turns. It sees only what a real user would see, and can give up when the goal is unreachable or push back on the agent's closing turn. A task or workflow agent instead receives the scenario's input and returns a result. **3. Each scenario runs three times** A run plays each scenario in **3 independent attempts (trials)**, so you can tell a reliable failure from an intermittent one. Recorded regressions are replayed 5 times each. **4. Every eval is scored** Each conversation is scored against every eval in the suite. **5. Results are written and the report updates** You get a summary in chat and the report updates with the run's results. ### Why a conversation ended early When a conversation ends before its plan finishes, the report shows the reason: | Reason | What it means | | - | - | | `user_gave_up` | The simulated user decided the goal was unreachable. | | `agent_ended` | Your agent ended the conversation. | | `handed_off` | Your agent handed off, for example to a human. | | `agent_raised` | Your agent raised an error. | | `agent_recursion_budget` | Your agent hit its recursion limit. | | `policy_terminated` | An eval ended the conversation. | | `turn_cap_exhausted` | The conversation hit its turn limit. | | `time_budget_exhausted` | The scenario ran out of time. | | `turn_budget_exhausted` | A single agent turn ran out of time. | | `provider_infra` | Your agent's model provider failed (rate limit or outage). Check the key and quota, then run again. | | `plan_exhausted` | The scenario delivered everything it planned. This is not a truncation. | You may see other reasons. They are shown as reported. ### Run integrity If the run could not measure everything it set out to, the results carry a **Run integrity** line. It appears only when something applies, for example: > Run integrity — 1 scenario landed incomplete on the 40-minute scenario time budget; it was graded to the turn it reached and counts as neither a pass nor a fail. Other cases are trials that could not run on Plurai's side, trials stopped by your model provider, and trials where your agent's process could not start. Each tells you whether to run them again. A green run with an integrity line is not a full measurement. An agent that raises an error inside its turn counts as a failure, not as "could not run." ## Next steps If anything failed, run `/simulation-fixes`. It diagnoses each failure in your agent and hands you a ranked fix plan. > [card] [Fix and keep regressions](https://docs.plurai.ai/guides/fix-and-keep-regressions.md) > > Turn failures into fixes and replayable regressions. > [card] [Run results](https://docs.plurai.ai/report/run-results.md) > > Read the run in the report. > [card] [Run unattended](https://docs.plurai.ai/guides/ci.md) > > Run the suite where nobody can answer questions. > [card] [Run from the SDK](https://docs.plurai.ai/sdk/run-simulations.md) > > Run `plurai-run-simulation` directly. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.