Scenarios¶
Scenarios define the multi-turn, multi-agent workloads that PawBench runs against your endpoint. Each scenario simulates a realistic coding task with tool calls, injected tool results, and quality expectations.
How Scenarios Work¶
A scenario is a JSON file describing:
- Agents -- One or more parallel agents, each with a role and conversation turns
- Turns -- Sequential conversation steps with user prompts, tool results, and expected outcomes
- Tools -- The tool schemas available to the model (e.g.,
write_file,read_file,run_command) - Expectations -- What the model should produce at each turn (tool calls, keywords, format)
PawBench dispatches all agents in a scenario in parallel at each concurrency level, measures throughput and latency, then scores quality against the expectations.
Built-in Scenarios: PawStyle by Lola¶
All built-in scenarios revolve around building PawStyle by Lola -- a boutique dog apparel e-commerce store inspired by Lola (@_justlolathings).
Two parallel agents (frontend and backend) collaborate to build the store. Products include Lola's Signature Bandana, Cozy Knit Sweater, Rainy Day Raincoat, Adventure Booties, Dapper Bow Tie, and Walk-in-Style Harness -- with "Lola's Pick" badges on her personal favorites.
pawstyle-independent¶
Purpose: Pure parallel throughput + quality baseline with no cross-agent communication.
Frontend and backend work independently on Lola's shop. No steering events. This scenario establishes the quality baseline -- how well the model performs when each agent works in isolation.
- Agents: Frontend (HTML/CSS/JS product page, cart, checkout) + Backend (Python REST API with products, orders, health endpoints)
- Turns per agent: 3
- Steering events: None
- Measures: Throughput, quality, efficiency (baseline)
pawstyle¶
Purpose: Tests adaptability via a mid-task steering event.
Same as independent, but the backend agent receives a steering event at turn 3: "Frontend added a Size Guide button -- implement Lola's breed-specific sizing endpoint." The model must adapt its plan and incorporate the new requirement.
- Agents: Frontend (TypeScript fullstack) + Backend (receives steering event)
- Turns per agent: 3
- Steering events: 1 (backend turn 3)
- Measures: Throughput, quality, efficiency, steering response
pawstyle-nudge¶
Purpose: Tests cross-agent coordination via nudge events.
Frontend adds Lola's Favorites (wishlist) and Compare features that require backend changes. Backend receives nudge events simulating cross-agent communication. This tests the model's ability to handle context injection and adapt to requirements driven by another agent.
- Agents: Frontend (driver, adds wishlist + compare) + Backend (receives nudges to add supporting endpoints)
- Turns per agent: 3
- Steering events: Multiple nudges
- Measures: All four dimensions, with emphasis on adaptability
JSON Format Specification¶
Scenarios follow this JSON schema:
{
"id": "my-scenario",
"name": "Human-readable scenario name",
"description": "What this scenario tests",
"agents": [
{
"id": "agent-1",
"name": "Agent Display Name",
"turns": [
{
"turn": 1,
"role": "user",
"content": "The user prompt for this turn...",
"tools": ["write_file", "read_file"],
"expect": {
"tool_calls_min": 1,
"tool_name_any": ["write_file"],
"output_mentions": ["keyword1", "keyword2"],
"steering_followed": true,
"steering_keywords": ["size-guide", "size_guide"]
}
},
{
"turn": 2,
"role": "tool_result",
"inject_tool_results": [
{
"tool_call_id": "auto",
"name": "write_file",
"content": "{\"status\": \"ok\", \"path\": \"file.py\"}"
}
],
"content": "Next instruction after tool result...",
"tools": ["write_file"],
"expect": {
"tool_calls_min": 1,
"output_mentions": ["expected_keyword"]
}
}
]
}
],
"tools_schema": [
{
"type": "function",
"function": {
"name": "write_file",
"description": "Write content to a file",
"parameters": {
"type": "object",
"properties": {
"path": {"type": "string", "description": "File path"},
"content": {"type": "string", "description": "File content"}
},
"required": ["path", "content"]
}
}
}
]
}
Fields¶
| Field | Required | Description |
|---|---|---|
id |
Yes | Unique scenario identifier (used in filenames and reports) |
name |
Yes | Human-readable name displayed in output |
description |
No | Longer description of what the scenario tests |
agents |
Yes | Array of agent definitions |
agents[].id |
Yes | Unique agent identifier within the scenario |
agents[].name |
Yes | Display name for the agent |
agents[].turns |
Yes | Array of conversation turns |
tools_schema |
Yes | OpenAI-format tool definitions available to all agents |
Turn Fields¶
| Field | Required | Description |
|---|---|---|
turn |
Yes | Turn number (1-indexed) |
role |
Yes | "user" for prompts, "tool_result" for injected tool results |
content |
Yes | The prompt text or follow-up instruction |
tools |
No | Which tools from tools_schema are available this turn |
inject_tool_results |
No | Simulated tool results to inject before this turn's prompt |
expect |
No | Quality expectations for scoring |
Expect Fields¶
| Field | Description |
|---|---|
tool_calls_min |
Minimum number of tool calls expected |
tool_name_any |
At least one tool call must use one of these function names |
output_mentions |
Keywords that should appear in output text or tool call arguments |
steering_followed |
Whether this turn tests a steering/nudge response |
steering_keywords |
Keywords indicating the model followed the steering event |
Writing Custom Scenarios¶
- Create a JSON file following the format above
- Include at least one agent with 2+ turns
- Add tool calls and
expectblocks for quality measurement - For steering tests, use
inject_tool_resultsto simulate cross-agent events
Run your custom scenario:
You can combine custom and built-in scenarios:
Scenario Design Tips
- Use 3+ turns per agent for meaningful multi-turn measurement
- Include tool calls in every turn (this is what differentiates PawBench from chat benchmarks)
- Add
inject_tool_resultsbetween turns to simulate realistic tool-use workflows - Include at least one steering variant to test adaptability
- Keep prompts detailed and specific -- vague prompts produce noisy quality scores