Your agent’s side
The layer the scan generates to drive your agent in a simulation. It stubs production dependencies, swaps
your tools for mock-aware versions, and runs the agent with in-memory state. Your agent’s source code is
never modified. The harness is stored as a script in your project,
simulate.py. It is the only thing the
simulation edits, and it asks you first.plurai.config.yaml
Lists your agent’s tools with their mode (mocked or real) and description, plus project settings such as
the anonymization posture. See the config reference.
report section
The map of your agent the scan builds: every flow it can take, and the tools, skills and sub-agents it uses
along the way. Scenarios are built to walk the routes where an agent is most likely to break, so the suite
tests smarter, not more. The report lights up the route each scenario walks.
The simulated world
mocked | live
Replacing a system your agent calls with a stand-in that answers the same way, without touching the real
thing. When a tool is mocked, your agent calls it exactly as usual, but the call is answered from the
digital twin instead of your production system. Your tool’s own code doesn’t run.The scan decides, per tool, what is mocked and what runs live (for real):
- Writes are mocked by default, so a refund, cancellation or update never reaches a real customer. A live write needs your explicit opt-in.
- Reads can run live against the simulated world, since they change nothing.
- Safe: your agent can act freely with no side effects on real systems or data.
- Controlled: each scenario starts from a world built for it, including cases that are rare or hard to set up in production.
- Repeatable: the same scenario starts from the same world every time, so a failure reproduces and a fix can be checked.
- No infrastructure: you don’t need staging systems or test accounts to run a simulation.
lookup_invoice for real”. See
Scan your agent.mock environment
A mocked copy of the systems your agent calls, such as your CRM, database or payment service. It runs in a
private sandbox in Plurai’s cloud, dedicated to you, and holds a seeded world: the records and state
each scenario starts from, which change as your agent acts on them. Every mocked call your agent makes is answered here. The docs
also call it the mock environment.
optional
For agents that retrieve from documents or databases: a corpus built from your material, with questions
planted in scenarios to check that the agent answers from it.
Production grounding
optional
Records of your agent’s real production conversations, from your observability platform (LangSmith,
Langfuse, Arize or Braintrust) or a saved export. The SDK reads them on your machine with your own
credentials.
optional stage
Basing the simulation on your traces instead of only on your code. Plurai learns your agent’s tool traffic,
the anomalies it meets, how your users talk and what fails in production, so the digital twin, scenarios
and evals match reality. Real failures can also become scenarios that replay from the turn where the
conversation broke. See Ground in production traces.
The suite
scenarios + evals
The tests for one agent: the scenarios and the evals that score them. They play against the mock
environment, which is not part of the suite.
dialog | pipeline
How the run drives your agent. A dialog agent holds a conversation, so a simulated user plays it. A
pipeline agent, such as a classifier, an extractor or a workflow step, takes one structured input
and returns one structured output. The scan picks the shape; it is the
shape field in
plurai.config.yaml.One test case. For a dialog agent it is a simulated conversation with a goal (what the user wants), a
persona (who the simulated user is and how they behave), an opening line and a seeded world. For a
pipeline agent it is a realistic task input, such as a ticket, a document or a record. Either way it
has a condition it passes when. A scenario comes from one of these sources:
your requirements, a production failure, a regression, or the suite itself.
Dialog agents only. The simulated user who plays a scenario. It reacts to your agent turn by turn, and can give up, push back
or ask for a human the way real users do.
A check scored on every conversation. Each eval says when it fails, where it comes from, and how it
is judged, and has a severity. An eval is judged in one of three ways:
- LLM as a judge: a model reads the conversation and decides, for checks that need judgment, such as tone or whether an answer is grounded.
- Code: a deterministic check, for facts that are simply true or false, such as whether a tool was called or a record changed.
- Hybrid: a model judge broken into code checks and small, focused model calls, so the result is as deterministic as possible and cheaper to run.
Running and results
One execution of the suite. Each scenario is played in one or more attempts (trials), so a result can
be read as a consistent pass, a consistent fail, or inconsistent.
A confirmed failure of your agent found by a run, ranked by severity, with the conversation and turn
where it broke as evidence. A potential issue is one suspected while reading code or building the
suite, which may be intended and needs your call.
A proposed change to your agent that addresses a gap, traced to the code that causes it. After a rerun,
a gap can be marked verified fixed.
A gap kept as a scenario that replays the turn it was confirmed on, so every later suite catches it if it
comes back.
Organizing your work
A named, committable snapshot of a mock environment, its scenarios and evals, for example one per
component under test. You can switch between environments without regenerating anything.
See Environments.
The single live page that every stage updates. It keeps the same link for the whole flow.
See The simulation report.
Whether data is scrubbed of personal information on your machine before upload. It is set once per
environment and cannot be switched afterwards. See Data and privacy.