ScenarioForge — User Guide

Marketing / product name is ScenarioForge. The on-disk identifier (Python package, CLI, env vars) stays scenariogen — every command in this guide uses that name. See docs/BRAND.md for the naming rule.

A practical walkthrough of installing ScenarioForge, generating test scenarios from CSV or Jira, reviewing them in the browser-based review console, and pushing the approved ones back to Jira. End-to-end examples + the trouble spots most users hit on first run.

TL;DR for a five-minute demo: ```bash git clone https://github.com/luigipascal/Scenarios.git cd Scenarios python -m venv .venv && .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1 pip install -e ".[dev]" scenariogen demo --output-dir runs # no LLM key needed scenariogen serve --runs-dir runs --port 8000

open http://127.0.0.1:8000

```

Table of contents

1. What it does
2. Install
3. Quick demo without an LLM key
4. The review console — a tour
5. Generating real scenarios with an LLM key
6. Feeding the AI your guides and manuals
7. Controlling what gets generated
8. Reviewing, editing, regenerating
9. Pushing approved scenarios back to Jira
10. Executable test cases (Phase 3)
11. White-label for consultancies
12. Switching LLM providers
13. Troubleshooting
14. Reference: every CLI command

1. What it does

ScenarioForge turns a backlog of work items (CSV export or Jira read) plus optional reference documents into a structured, traceable set of test scenarios — positive, negative, boundary, regulatory, security, integration, and more — with full audit trail and the workflow to approve them and push them back to Jira as Test issues, links, or comments.

The AI agent is the engine: it researches the system under test (web search + any docs you provide), grounds itself in your acceptance criteria, and produces scenarios with categories, priorities, expected results, and traceability. You review them in a browser, approve the ones that fit, and push to Jira.

 ┌──────────┐    ┌──────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────┐
 │ Jira/CSV │ ─▶ │ AI agent │ ─▶ │ Review/edit  │ ─▶ │ Test cases   │ ─▶ │  Jira    │
 │  tickets │    │ Treasury │    │ scenarios    │    │ review + CSV │    │ hierarchy│
 └──────────┘    └──────────┘    └──────────────┘    └──────────────┘    └──────────┘
   ingest          scenarios         approve            generate / import    sub-task +
                   + research                                              attachment

2. Install

Prerequisites

• Python 3.11 or 3.12
• Git
• (Windows) PowerShell — examples below use the PowerShell form

Steps

git clone https://github.com/luigipascal/Scenarios.git
cd Scenarios

# create + activate a virtualenv
python -m venv .venv
.venv\Scripts\Activate.ps1                       # Windows
# source .venv/bin/activate                      # Mac/Linux

# install scenariogen (the [dev] extra also includes the optional LLM SDKs)
pip install -e ".[dev]"

# copy the env template
copy .env.example .env                           # Windows
# cp .env.example .env                           # Mac/Linux

After installation you should have the scenariogen command on your PATH:

scenariogen --help

You'll see six sub-commands: ingest, research, run, report, edit, approve, apply, serve, demo.

3. Quick demo without an LLM key

If you just want to see the UI populated, the demo command seeds a complete run from bundled stub fixtures. No LLM key, no network, no cost.

scenariogen demo --output-dir runs
scenariogen serve --runs-dir runs --port 8000

Open http://127.0.0.1:8000 in a browser. You'll see one run with five Kondor-style tickets and ~16 scenarios across positive / negative / boundary / regulatory categories. Click through to see the review console working.

The demo command is also useful when you want to demo the tool to a stakeholder without burning LLM credits.

4. The review console — a tour

Layout

┌─────────────────────────────────────────────────────────────────────────────┐
│ scenariogen review   [tenant]                AI: openai/gpt-4.1   runs: ./runs│  ← header
├──────────────┬──────────────────────────────────────────────────────────────┤
│ Runs         │ Runs / 20260515T120000 / KON-101                              │  ← breadcrumbs
│ • 2026-05…   │                                                              │
│ • 2026-05…   │ ┌──────────────────────────────────────────────────────────┐ │
│              │ │ 🟣 AI assistant                                          │ │
│ Tickets      │ │ Active: openai / gpt-4.1                                 │ │
│ • KON-101 ✓  │ │ [Type an instruction…]               [+ Add with AI]     │ │  ← AI panel
│ • KON-102    │ └──────────────────────────────────────────────────────────┘ │
│ • KON-103    │                                                              │
│ • KON-104 ✓  │ KON-101 — Add FX forward booking for CNH    [Approve all]    │
│ • KON-105    │                                                              │
│              │ ┌─────────────────────────────────────────────────────────┐  │
│              │ │ KON-101-S01 — Booking happy path                        │  │
│              │ │ positive · high · draft · confidence 0.90 · req: AC-1   │  │
│              │ │ Given a logged-in trader                                │  │
│              │ │ When they submit a valid trade                          │  │
│              │ │ Then the trade is persisted                             │  │
│              │ │ Expected: Trade saved with state CONFIRMED              │  │
│              │ │ [Status: draft ▾] [Approve] [Reject]                    │  │  ← scenario card
│              │ │ [Regenerate with AI…] [Remove]                          │  │
│              │ └─────────────────────────────────────────────────────────┘  │
└──────────────┴──────────────────────────────────────────────────────────────┘

Header

• Brand title — <product_name> review. Shows your branding if a branding.yaml is configured (see section 11).
• Tenant chip — visible only when consultancy_id / end_client_id are set, useful for consultancies tracking which client a session belongs to.
• AI pill — shows the active LLM provider + model, purple when credentials are configured, red when missing. Hover for tooltip with full details.
• Runs root — the directory the server is watching.

Left sidebar

Three collapsible sections (click the header to expand/collapse):

• Runs — every run folder under the runs root. The active run is highlighted in blue. The number on the right is total scenarios in that run.
• Scenarios (tickets) — appears when a run is selected. Each row is a work item (e.g. KAN-1532). Numbers show approved / total scenarios (3/4 ✓ = 3 of 4 scenarios approved).
• Cases — executable test-case packs for the active run. Each row is a scenario id with approved_cases / total_cases. A green dot and Jira key (e.g. KAN-4128) appear after a successful publish. Click a row to open that ticket and scroll to the scenario card.

While test cases are generating, a progress bar appears under Cases (with Cancel). While publishing to Jira, a status line appears there too.

Main area

The main area changes based on what's selected in the sidebar:

• Nothing selected → welcome screen with hints if there are no runs.
• Run selected, no ticket → table of tickets plus toolbar: Push to Jira…, Generate test cases…, Import test cases CSV, Publish test cases to Jira.
• Ticket selected → AI assistant panel + scenario cards (one per scenario), filterable by status; ticket toolbar includes the same test-case actions scoped to that work item.

Scenario card

Each scenario shows:

• Title + ID
• Metadata row: category, priority, current status, AI confidence, requirement reference (the AC it's grounded in)
• Steps in Given/When/Then style
• Expected result
• Source text (the exact snippet from the AC the AI is grounding the scenario in)
• Rationale (why the AI generated this case)
• Action row: status dropdown, Approve, Reject, Regenerate with AI…, Remove
• Executable test cases panel: recommended count, confidence per case, approve/delete, download CSV, link to Jira sub-task after publish

A regenerate-in-progress card shows a purple spinner in the top-right and greys out until the AI responds.

5. Generating real scenarios with an LLM key

1. Set credentials. Edit .env:

ini SCENARIOGEN_LLM_PROVIDER=openai SCENARIOGEN_MODEL=gpt-4.1 OPENAI_API_KEY=sk-...

For other providers, see section 12.

2. Prepare your input. Either: - A CSV with columns including key, summary, description, acceptance_criteria (newline-, semicolon-, or pipe-separated), plus optional labels, components, priority. Other common headers like "Issue Key" or "Title" are auto-mapped. - A Jira JQL — set JIRA_URL, JIRA_USER, JIRA_TOKEN in .env, then pass --source jira --input "project = KON AND sprint in openSprints()".

3. Run:

powershell scenariogen run ` --source csv ` --input my_tickets.csv ` --system "Kondor Fusion" --sector "Banking/Treasury" ` --output-dir runs ` --limit 5 --confirm-cost

The default tick limit is 10. Above 5 you must pass --confirm-cost — this is the budget guard. Add --max-spend 5.00 to cap dollar cost.

4. Refresh the browser. The new run appears in the left sidebar.

6. Feeding the AI your guides and manuals

The AI agent's research is much better when you give it your internal documentation — proprietary system specs, runbooks, API references that aren't on the public internet.

scenariogen run `
  --source csv --input tickets.csv `
  --system "Internal trading platform" --sector "Banking" `
  --docs "docs/architecture.pdf,docs/api-spec.html,docs/runbook.md" `
  --output-dir runs --limit 5 --confirm-cost

Supported formats: PDF, Markdown, plain text, Word (.docx), HTML, XML. Pass either individual files or directories (directories are scanned recursively). OneNote .one binaries can't be parsed — export them to PDF or DOCX first.

Doc content is recorded in brief.sources as doc:<path> and the cache key includes a document fingerprint, so changing the docs forces a fresh research call.

7. Controlling what gets generated

Four CLI flags shape what the AI produces:

scenariogen run ... `
  --categories positive,negative,boundary,regulatory `
  --exclude-categories performance,accessibility `
  --focus "payments, settlement, intraday reconciliation" `
  --max-scenarios 8 `
  --extra-instructions "Emphasise EOD batch failure paths."

Same options also live in profiles/<name>.yaml so you don't have to type them every time:

name: kondor-banking
categories: [positive, negative, boundary, data_validation, integration, regulatory, role_permission]
focus: ["payments", "settlement"]
max_scenarios_per_ticket: 8

Then: scenariogen run --profile profiles/kondor-banking.yaml ...

8. Reviewing, editing, regenerating

From the UI

Inside a ticket page:

• Approve / Reject — single scenario via the buttons in the action row, or the whole ticket via the "Approve all in ticket" toolbar.
• Change status to anything else — use the Status: dropdown (draft → review → approved → ready → executed → passed / failed / blocked / deferred / rejected / needs_clarification).
• Regenerate with AI… — opens a dialog where you type an instruction: e.g. "make this a boundary case at notional = 10,000,001 USD". The card shows a spinner; when the AI returns, the scenario is replaced in place.
• Add scenarios with AI — the purple AI panel at the top of the ticket page accepts free-form instructions: e.g. "Add three negative cases for failed FX rate lookups". The AI appends new scenarios to the bundle, deduped by idempotency key.
• Remove — drops the scenario from the bundle (manifest records the deletion for audit).

The AI panel also doubles as a status indicator: green pill = provider configured, red pill = no key set yet.

From the CLI

Equivalent operations are available without the browser:

scenariogen edit --run runs\20260515T120000 --remove KON-101-S03,KON-104-S05
scenariogen edit --run runs\20260515T120000 --regenerate KON-101-S02 `
                  --instruction "Boundary at 10M+1"
scenariogen edit --run runs\20260515T120000 --add-to KON-101 --count 3 `
                  --instruction "Add three negative cases for failed FX rate lookups"
scenariogen edit --run runs\20260515T120000 --import edited_scenarios.csv

scenariogen approve --run runs\20260515T120000 --ids KON-101-S01,KON-101-S02
scenariogen approve --run runs\20260515T120000 --all KON-101
scenariogen approve --run runs\20260515T120000 --all all

Every operation is appended to runs/<ts>/manifest.jsonl for audit.

9. Pushing approved scenarios back to Jira

The default writer is hierarchy (set SCENARIOGEN_DEFAULT_JIRA_WRITER=hierarchy in .env). It creates one Jira issue per approved scenario (Task, Sub-task, or Story), links it to the source story/task/bug with Relates, and can parent scenario tasks under a Test Scenario epic.

Other writers are still available:

Hierarchy dialog (recommended)

On the run page, click Push to Jira… and set:

Run Dry-run first, then Push now. Idempotency is per writer — a prior comment push does not block hierarchy.

From the UI

On the run page (click a run in the sidebar but no ticket), click Push to Jira…. On a ticket page, use the same flow scoped to that work item if needed.

From the CLI

# Dry-run first — see what would happen
scenariogen apply --run runs\20260515T120000 --writer hierarchy

# Push with Test Scenario epic (existing)
scenariogen apply --run runs\20260515T120000 --writer hierarchy --apply `
  --epic-mode existing --epic-key KAN-1605

# Push a subset
scenariogen apply --run runs\20260515T120000 --writer hierarchy --apply `
  --only-ids KAN-1324-S01,KAN-1324-S03

# Other writers
scenariogen apply --run runs\20260515T120000 --writer xray --apply

Idempotency: every push is recorded in runs/<ts>/write_log.jsonl. A re-run skips scenarios that have already been pushed (matched by idempotency_key = sha1(work_item + title + category)). Pass --force to re-push anyway.

Gating: only scenarios in status approved, ready, or executed are pushed by default. Use --include-status to widen the gate if you need to (e.g. --include-status approved,review for in-progress reviews).

10. Executable test cases (Phase 3)

After scenarios are approved and (typically) pushed to Jira via hierarchy, Phase 3 adds detailed executable test cases under each scenario — realistic positive and negative steps for Treasury (FX, nostro, settlements, approvals, calendars, etc.). The AI recommends how many cases per scenario; you review confidence on screen, then export CSV and publish to Jira.

Workflow

1. Approve test scenarios (Phase 2).
2. Push scenarios with hierarchy + Test Scenario epic (Phase 2 Jira).
3. Generate test cases — run or ticket toolbar, or per-scenario Review test cases.
4. Review each case: confidence % (green / amber / red), approve, change status, or delete.
5. Approve executable test cases in each pack (scenario approval alone is not enough for publish).
6. Publish to Jira — creates or updates a Sub-task under each scenario Jira issue and attaches the CSV.

You do not use Jira directly for this step — everything goes through ScenarioForge (review console or API). In Jira, open the scenario issue created by hierarchy push (e.g. KAN-1660 for KAN-1532-S04), not only the parent story.

From the UI

Optional Treasury notes in the generate dialog (e.g. CNH calendar, four-eyes approval).

Pilot first: generate for one ticket before running thousands of scenarios (LLM cost/time).

What you see when publishing

After you confirm publish:

1. Cases sidebar — “Publishing to Jira…” (working state).
2. Top of the main panel — blue info message while the API runs.
3. Dismissible report — lists Created (new sub-tasks), Updated (existing sub-tasks refreshed), Skipped/warnings, and Errors with scenario ids and Jira keys.
4. Cases list + scenario card — green Jira badge and issue key when published.

If nothing is published, the report explains why (usually no approved test cases yet).

Regenerate vs publish (Jira behaviour)

Regenerating test cases does not auto-update Jira — run Publish again after you approve the new cases.

Artifacts

Under runs/<timestamp>/:

Import CSV columns

Minimum useful headers: Scenario ID, Test Case Title, Category, Steps, Expected Result. Optional: Linked Jira Key, Scenario Title (for matching), Confidence, Status, Preconditions, Test Data, Rationale.

Prerequisites for Jira publish

• Push the scenario with the hierarchy writer first (write_log.jsonl must have writer: hierarchy, status: pushed, and a target_key per scenario).
• Executable test cases to publish must be status approved, ready, or executed (not draft).
• Approving the scenario in Phase 2 does not approve the test cases inside the pack.

CLI

# One ticket, regenerate even if packs exist
scenariogen testcases-generate `
  --run runs\20260518T145057 `
  --work-item KAN-1532 `
  --replace-existing

# First N scenarios (skips packs that already exist unless --replace-existing)
scenariogen testcases-generate --run runs\<id> --limit 10

# Named scenarios only
scenariogen testcases-generate --run runs\<id> --only-ids KAN-1532-S04,KAN-1532-S05

Publish to Jira is UI or API only today (POST …/testcases/publish-jira).

HTTP API (same behaviour as UI)

See docs/SCENARIOGEN_GUIDE.html for technical detail.

11. White-label for consultancies

Consultancies reselling ScenarioForge can rebrand the CLI and UI with a single YAML file. Create branding.yaml:

product_name: "ConsultCo QA Studio"
tagline: "AI-assisted test scenario generation"
report_header: "ConsultCo — Confidential. Generated for client review only."
report_footer: "Generated by ConsultCo QA Studio. Contact: qa@consultco.example"
consultancy_id: "consultco"
end_client_id: "acmebank-treasury"
contact: "support@consultco.example"

Then point at it via env:

$env:SCENARIOGEN_BRANDING_FILE = "$pwd\branding.yaml"
scenariogen serve --runs-dir runs

Branded outputs:

• CLI banner — uses your product name
• Review console title — "ConsultCo QA Studio review"
• Tenant chip in the header — shows consultco → acmebank-treasury
• CSV — adds Tenant and End Client columns (only when branding is set; default CSV is unchanged)
• Markdown reports — header + footer with your strings
• Gherkin .feature files — # Generated by ConsultCo QA Studio comment
• Audit manifest — every run.inputs event tagged with consultancy_id + end_client_id for usage attribution

See branding.example.yaml in the repo for the full schema.

12. Switching LLM providers

ScenarioForge ships with adapters for six providers. Pick the one that matches your stack and set the corresponding env vars in .env:

OpenAI (default)

SCENARIOGEN_LLM_PROVIDER=openai
SCENARIOGEN_MODEL=gpt-4.1
OPENAI_API_KEY=sk-...

Anthropic Claude

SCENARIOGEN_LLM_PROVIDER=anthropic
SCENARIOGEN_MODEL=claude-sonnet-4-6
ANTHROPIC_API_KEY=sk-ant-...

Native web_search_20250305 tool, forced emit_output tool for structured output.

Azure OpenAI

The default for regulated buyers — private endpoints, no-training clauses, EU/UK regions.

SCENARIOGEN_LLM_PROVIDER=azure_openai
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AZURE_OPENAI_DEPLOYMENT=gpt-4.1
AZURE_OPENAI_API_VERSION=2024-10-21
TAVILY_API_KEY=tvly-...   # for web search (Azure has no native web search)

Google Gemini

SCENARIOGEN_LLM_PROVIDER=gemini
SCENARIOGEN_MODEL=gemini-2.5-pro
SCENARIOGEN_EMBEDDING_MODEL=text-embedding-004
GOOGLE_API_KEY=...

Native Google Search grounding.

AWS Bedrock

SCENARIOGEN_LLM_PROVIDER=bedrock
SCENARIOGEN_MODEL=anthropic.claude-sonnet-4-6-20250929-v1:0
AWS_BEDROCK_REGION=us-east-1
AWS_BEDROCK_EMBEDDING_MODEL_ID=amazon.titan-embed-text-v2:0
TAVILY_API_KEY=tvly-...   # for web search

Standard boto3 AWS credential resolution (env, ~/.aws, instance role).

Self-hosted / OpenAI-compatible (vLLM, Ollama, LiteLLM, OpenRouter)

SCENARIOGEN_LLM_PROVIDER=openai_compatible
SCENARIOGEN_MODEL=llama-3.1-70b-instruct
OPENAI_COMPATIBLE_BASE_URL=http://localhost:8000/v1
OPENAI_COMPATIBLE_API_KEY=not-required
TAVILY_API_KEY=tvly-...   # for web search if needed

This is the path for air-gapped deployments — point at your own vLLM or Ollama box, no public APIs called.

Verify which one is active

Hit http://127.0.0.1:8000/api/health or look at the AI pill in the review console header — it shows AI: <provider> / <model> in purple when credentials are configured, and AI: <provider> (no key) in red when they're missing.

13. Troubleshooting

"No such command 'demo'" / 'apply' / etc.

You're on an older install. Pull the latest, then:

Get-Process scenariogen -ErrorAction SilentlyContinue | Stop-Process
git pull origin main
pip install -e ".[dev]"

"OSError: [WinError 32] The process cannot access the file… 'scenariogen.exe'"

A previous scenariogen serve process is still holding the binary.

Get-Process scenariogen -ErrorAction SilentlyContinue | Stop-Process
pip install -e ".[dev]"

If that doesn't find anything, also close any Python REPL or Jupyter that has imported scenariogen.

Run "succeeded" but the UI shows zero scenarios

Two possible causes:

1. No LLM key configured. The CLI now refuses to start without credentials — but if you're on an old install, an old run may have silently produced zero bundles. Check runs/<ts>/bundles.jsonl — if it's empty, that's the symptom. Set your key in .env and re-run, or use scenariogen demo to seed a no-key run.

2. Wrong runs root in the server. The UI shows runs from the directory you passed to scenariogen serve --runs-dir <path>. Make sure the scenariogen run command wrote to the same directory.

/favicon.ico 404 in the browser console

Already fixed by the inline SVG favicon. Hard-refresh your browser (Ctrl+F5) — browsers cache favicons aggressively.

"AI: openai (no key)" in red

The provider is configured but no credential is set. Edit .env and restart the server. The AI pill polls on page load.

Snapshot tests are failing after a model upgrade

Delete tests/fixtures/snapshot_prompts/*.txt — they'll be regenerated on the next test run and the new content becomes the new snapshot.

"ImportError: anthropic" / "google.genai" / "boto3"

You're using a non-OpenAI provider but the optional SDK isn't installed:

pip install -e ".[anthropic]"            # or .[gemini] or .[bedrock]
pip install -e ".[all-llms]"             # all three at once

Jira write-back returns 401 / 403

Your JIRA_TOKEN is wrong or expired. Generate a new API token in Atlassian → Account Settings → Security → API tokens, paste it into .env, and restart.

For Xray, regenerate XRAY_CLIENT_ID / XRAY_CLIENT_SECRET from the Xray Global Settings page.

Test case publish fails (“no Jira scenario issue”)

Push the scenario with the hierarchy writer first. Publish looks up the scenario issue key in write_log.jsonl. Re-push hierarchy for that scenario if the log entry is missing.

Generate test cases does nothing (instant finish, 0 generated)

The job finished with warnings like pack exists (skipped) — packs are already on disk. Enable Replace existing packs in the dialog, or use CLI --replace-existing.

Also check Only approved scenarios if some tickets were not approved yet.

Publish to Jira — nothing in Jira / empty report

1. Approve test cases in each pack (Review test cases → Approve), not only the scenario status.
2. Look under each scenario issue in Jira (hierarchy child of the story), not the story itself.
3. Read the publish report at the top of the page (warnings list skipped packs).
4. Ensure hierarchy push ran first (write_log.jsonl has target_key for that scenario).

Publish shows Created/Updated but I cannot find issues

Sub-tasks are children of KAN-xxxx-Snn scenario issues (e.g. KAN-1660 for KAN-1532-S04), linked from hierarchy push. Search Jira for the key shown in the report or the Cases sidebar.

Browser keeps calling the API (Network tab)

Hard-refresh the review console (Ctrl+Shift+R) after upgrading ScenarioForge — old UI versions could poll job status in a loop. Idle tabs should not hammer /testcases or /api/jobs/….

14. Reference: every CLI command

Every command has --help for full flag documentation.

Next steps

• Phase 3b (delivered): Executable test cases — Treasury expert generation, CSV, Jira publish with update-in-place (UI + API + testcases-generate CLI).
• Phase 4 (next major): RBAC + SSO + audit DB + risk-based test packs + dashboards.
• Phase 5 (after): Air-gapped + self-hosted LLM, TestRail/qTest/Azure DevOps writers, automation skeletons.

Maintainers: When you change product behaviour (CLI, UI, API, Jira writers), update docs/USER_GUIDE.md, docs/SCENARIOGEN_GUIDE.html, and README.md in the same change.