# Windup — full documentation

> Natural-language E2E tests with deterministic replay — the LLM plans once, replays run without it. ~1s, $0 per replay.

This file concatenates the entire Windup documentation as clean markdown for LLM consumption. Individual pages are also available at https://windup.run/docs/<page>.md and mapped in https://windup.run/llms.txt.

---

# Getting started

**Natural-language E2E tests with deterministic replay — the LLM plans once, replays run without it.**

Describe a test in plain language — *"log in as the test account, add product X to the cart, check out and verify the order confirmation"* — and Windup turns it into a deterministic JSON plan of browser actions. From the second run on, the test replays **with zero LLM calls**: ~1 second, $0, stable results.

```bash
npm i -D windupjs        # Chromium is provisioned automatically (one-time, machine-wide cache)
npx windup init          # 3 questions → windup.config.ts + example scenario
npx windup scan          # index your app's routes & elements from source code
npx windup new "log in as admin and create an invoice"   # LLM-assisted scenario authoring
npx windup run checkout  # 1st run: the LLM plans · every run after: ~1s replay, $0
```

## Requirements

- **Node ≥ 20.**
- An **API key** for your planner LLM in `.env.local` or `.env` (`.env.local` wins — use it when your `.env` is committed): `GOOGLE_GENERATIVE_AI_API_KEY` for Google (default) or `OPENAI_API_KEY` for OpenAI.

Keys are only used for planning; cached replays never call an LLM. To use an existing Chrome instead of the auto-downloaded Chromium, set `CHROME_PATH`; to skip the download entirely, set `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1`.

## A five-minute tour

The full workflow on a fresh project, with what you should expect to see:

```bash
# 1. Install — Chromium is provisioned automatically
npm i -D windupjs

# 2. Initialize — 3 questions (base URL, model, scenarios dir)
npx windup init
#    → windup.config.ts + e2e/scenarios/ + .windup/ (gitignored)

# 3. Index your app from source — before anything ever runs
npx windup scan
#    scan complete (full): framework=react-router routes=106 elements=1125
#    The site map now knows your real routes and selectors; the planner
#    will use them instead of guessing. Re-run after big changes
#    (windup scan --update re-indexes only files changed since, via git).

# 4. Register test credentials once — values never touch git
npx windup secret set admin        # hidden prompts → .env.local + mapping

# 5. Author a scenario from a rough instruction
npx windup new "log in with the admin account and create an invoice for ACME"
#    → e2e/scenarios/create-invoice-acme.json — precise task grounded in
#      your real screens, account referenced by name, final verification

# 6. First run — the LLM plans once (~3s, ~$0.002)
npx windup run create-invoice-acme
#    PASS  create-invoice-acme  cache=miss llm_calls=1 ... cost=$0.0024

# 7. Every run after — deterministic replay, zero LLM
npx windup run create-invoice-acme
#    PASS  create-invoice-acme  cache=hit llm_calls=0 total=600ms cost=$0

# 8. Read results like a human, ship reports to CI
npx windup run --all --summary --reporter html
npx windup costs                   # AI spend: totals, per provider/model
```

If a run fails after an app change, the cached plan is invalidated and re-planned automatically on the next run — **you edit scenarios, not selectors.**

---

# How it works

```
natural-language task ──▶ planner (LLM, 1 call) ──▶ JSON action plan
                                                        │
       trajectory cache ◀── cheap verification ◀── deterministic executor
             │
             └──▶ subsequent runs: zero LLM, ~1s, $0
```

The expensive part — figuring out the browser actions — happens a single time and is turned into cached, verifiable data.

- **Plans are data, not code** — schema-validated JSON; no generated scripts, no conditionals.
- **Cheap verification** — DOM/URL postconditions after every action. A failed verification invalidates the cached plan and triggers an automatic re-plan.
- **Site map** — every execution feeds a graph of pages and transitions; `windup scan` seeds that graph straight from your source code before the first run, so the planner uses your app's *real* selectors instead of guessing.
- **Fragments** — proven action blocks (e.g. login) that the planner composes via `{ "type": "use" }` instead of regenerating.
- **Zero hardcoded site knowledge** — the engine knows frameworks and the web, never *your* site. All site knowledge arrives as input (scenarios, config, manifest) or is discovered at runtime.

## Why Windup

Hand-written scripts are cheap to run but expensive to maintain. Per-run AI agents are easy to write but slow and non-deterministic. Windup takes the good half of each.

|  | Hand-written scripts | AI agent per run | **Windup** |
|---|---|---|---|
| Authoring | code + selectors by hand | plain language | plain language |
| Run cost | $0 | LLM on **every** run | LLM on **first** run only |
| Run speed | fast | slow (model in the loop) | ~1s replay |
| Determinism | high | low — improvises each time | high — same plan every replay |
| App changed | you fix the script | may silently do something else | verification fails → auto re-plan |

For the deeper mechanics — module boundaries, data formats, cost and security posture — see [Architecture &amp; spec](https://windup.run/docs/architecture.md).

---

# Scenarios

A scenario is a JSON file in your scenarios directory (default `e2e/scenarios/`):

```json
{
  "scenario_id": "checkout",
  "start_url": "/",
  "task": "Log in as the qa account, add 'Backpack' to the cart, check out and verify the order confirmation message appears.",
  "hints": ["Optional site-specific tips for the planner. Delete if not needed."]
}
```

- `start_url` is **optional** (defaults to `/`) and should stay environment-free: a path, resolved against the effective base URL.
- End the task with **what to verify** — that becomes the plan's final postcondition.
- Never put secrets in tasks. Reference accounts from the project manifest (see [Test credentials](https://windup.run/docs/credentials.md)); the plan will use `value_ref: "ENV:VAR"` and the real value is resolved only at runtime, never cached.

## Scenario dependencies (`depends_on`)

Flows rarely start from zero — creating a bank account requires being logged in. Declare prerequisites and each scenario stays small, focused and individually cacheable:

```json
{
  "scenario_id": "create-bank-account",
  "depends_on": ["login"],
  "task": "Already on the dashboard, open Settings > Bank accounts, create an account named 'Inter' and verify it appears in the list."
}
```

- Dependencies run **in the same browser session**, in order, each with its own cache — a warm suite replays the whole chain with zero LLM calls.
- Without a `start_url`, the dependent scenario **continues from where the last dependency ended** — and on first planning the LLM sees that real page (the post-login dashboard), instead of planning blind.
- Chains work (`login` → `select-company` → `create-account`), cycles are rejected, and a failing dependency fails the run with kind `dependency` before the scenario itself starts.
- Each dependency keeps its own self-healing: if its cached plan breaks, it re-plans and re-caches — dependents benefit automatically.
- Editing a scenario's `task` invalidates its cached plan (a rewritten test is a different test).

`windup new` handles dependencies both ways: `--depends-on login` declares them explicitly, and **the author LLM also suggests them on its own** — it sees every existing scenario (id + task) and, when the instruction presupposes a state one of them produces ("already logged in…"), emits `depends_on` automatically (mechanically filtered against real scenario ids — never invented).

## Authoring with `windup new`

You don't have to write detailed tasks by hand. Give `windup new` a rough instruction and the LLM acts as a test author — it rewrites it into a precise, verifiable scenario using the **site map** (real screens, menus and elements from `windup scan` and past runs) and the **project manifest** (accounts referenced by name, never literal credentials):

```bash
npx windup new "log in with the qa user, add the backpack to the cart and check out"
# → e2e/scenarios/purchase-backpack-qa.json — real screen names, concrete fake
#   form data, account referenced as "the qa account", explicit final verification
```

It generates the `scenario_id`, picks the `start_url` from known routes (falling back to `/` — it never invents paths), and adds selector hints from the map when they help. Add **`--validate`** to have it run the generated scenario and, if it fails, refine it from the failure and retry (up to 3 attempts) — you get back a scenario that *already passed once*, with a warm cache:

```bash
npx windup new "log in and create a cost center named Marketing" --validate
#   attempt 1: FAIL — element button:has-text('Save') not visible
#   attempt 2: PASSED
#   ✓ validated in 2 attempts — the plan is cached
```

**Credentials in the instruction never land in the scenario file**: they are auto-registered as a named account (values in `.env.local`, mapping in `windup.credentials.json`) and the task references the account — see [Test credentials](https://windup.run/docs/credentials.md).

Flags: `--id <id>`, `--force` (overwrite), `--depends-on <ids>`, `--llm <provider[:model]>`. The output is a file for **you to review, edit and commit** — authoring is assisted, the test remains yours. One LLM call (~$0.001), recorded in the `windup costs` ledger under `authoring`.

---

# Test credentials

Credentials never live in scenario files, plans, the cache or git — only **references**. Values stay in `.env.local` (gitignored) or CI secrets; the account → ENV-name mapping lives in `windup.credentials.json` (committed — it contains no values) and is merged into the project manifest automatically.

```bash
npx windup secret set admin        # hidden interactive prompts → .env.local + mapping
npx windup secret list             # accounts + whether each ENV is set (never prints values)
```

Tasks then reference the account by name — *"log in with the admin account"* — and plans use `value_ref: "ENV:WINDUP_ADMIN_PASSWORD"`, resolved only at execution time.

`windup new` does this automatically: credentials typed in the instruction are detected, registered, and scrubbed — the generated scenario mentions the account, never the values. In CI, define the same variable names as pipeline secrets.

---

# Environments (dev / staging / CI)

The start URL origin comes from, in precedence order:

1. `--base-url` flag
2. `WINDUP_BASE_URL` env
3. `baseUrl` in `windup.config.ts`
4. an absolute `start_url` in the scenario

An explicit override rebases even absolute scenario URLs (path and query are preserved).

The plan cache is **environment-portable**: cache identity uses the start URL *path*, not host/port. A plan generated against `localhost:8080` replays on staging or CI with zero LLM calls.

```bash
npx windup run checkout --base-url https://staging.example.com
WINDUP_BASE_URL=http://localhost:8080 npx windup run --all
```

---

# LLM providers

The planner is provider-agnostic. Google Gemini and OpenAI are supported; configure several at once and pick one per run:

```ts
// windup.config.ts
llm: {
  provider: "google",                       // default for runs without --llm
  model: "gemini-3.1-flash-lite",
  providers: {
    openai: { model: "gpt-5-mini" },        // default model when --llm openai is used
    // openai: { apiKeyEnv: "MY_OPENAI_KEY", baseUrl: "https://my-proxy/v1" },
  },
},
```

```bash
npx windup run checkout                         # config default (google)
npx windup run checkout --llm openai            # provider default model (gpt-5-mini)
npx windup run checkout --llm openai:gpt-5-nano # explicit provider:model
WINDUP_LLM=openai:gpt-5-mini npx windup run --all   # same thing via env (CI)
```

- `--llm` works on `run`, `bench` (compare providers on the same scenario) and `scan` (LLM-assist layer).
- API keys: `GOOGLE_GENERATIVE_AI_API_KEY` / `OPENAI_API_KEY` by default; override the env-var name with `apiKeyEnv`.
- `baseUrl` (OpenAI only) points at any OpenAI-compatible endpoint — Azure, a proxy, or a local model server.
- Switching providers never invalidates the plan cache: plans are data, replays are LLM-free regardless of who planned them.
- `windup costs` breaks spend down **by provider and by model**, so alternating between LLMs keeps per-vendor spend visible.

---

# CI / CD

```bash
npx windup run --all --reporter junit --report-file reports/windup.xml
```

- `--all` runs every scenario in the directory (one warm browser for the whole suite).
- Exit code is non-zero when any scenario fails.
- `--concurrency <n>` runs scenarios in parallel over one shared warm browser (~2× faster on a mixed suite); `--browser firefox|webkit` runs the suite cross-browser.
- `--reporter junit` emits JUnit XML (GitHub Actions, GitLab and Jenkins consume it natively); `--reporter json` emits a machine-readable summary; `--reporter html` emits a self-contained human-friendly page (zero JS/deps — upload it as a CI artifact or open locally). Default output: `.windup/reports/`.
- `windup costs --json` reports AI spend for pipeline tracking.

## Example: GitHub Actions

```yaml
- run: npm ci && npx playwright install chromium
- run: npx windup run --all --base-url http://localhost:8080 --reporter junit --report-file reports/windup.xml
  env:
    GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GEMINI_KEY }}
- uses: dorny/test-reporter@v1
  if: always()
  with: { name: windup, path: reports/windup.xml, reporter: java-junit }
```

---

# Configuration (`windup.config.ts`)

```ts
import { defineConfig } from "windupjs";

export default defineConfig({
  baseUrl: "http://localhost:3000",
  llm: {
    provider: "google",
    model: "gemini-3.1-flash-lite",
    // Several providers at once — pick per run with --llm (see "LLM providers"):
    providers: { openai: { model: "gpt-5-mini" } },
  },
  scenarios: "e2e/scenarios",
  framework: "react-router",          // detected by init; used by scan
  // browser: "chromium",             // or "firefox" / "webkit" (need: npx playwright install <name>)
  scan: {
    llmAssist: { enabled: true, maxCalls: 20 },   // hard cost cap per scan
  },
  // Project manifest: team-provided knowledge injected into the planner prompt.
  context: {
    conventions: ["every interactive element has a data-testid"],
    credentials: {
      qa: { user: "ENV:QA_USER", password: "ENV:QA_PASSWORD" },
    },
    vocabulary: { "order": "the Order entity, screen /orders" },
  },
});
```

- **`context.credentials`** maps account names to ENV references. When a task mentions the account, the plan uses `value_ref` — manifest credentials take precedence even if the page displays values, and the planner is forbidden from inventing ENV names.
- **LLM-assist** (scan layer 3) reads files the static layers couldn't resolve (dynamically built routes, indirect components), capped by `maxCalls`. Results are remembered per file hash — unchanged files never cost again. Costs are recorded in the ledger and shown by `windup costs`.

## What lives where

| Path | Contents | Commit? |
|---|---|---|
| `windup.config.ts` | Configuration | ✅ |
| `e2e/scenarios/*.json` | Your tests, in natural language | ✅ |
| `e2e/fragments/*.json` | Curated reusable blocks | ✅ |
| `windup.credentials.json` | Account → ENV-name mapping (no values) | ✅ |
| `.env.local` | Credential values | ❌ (auto-gitignored; CI uses secrets with the same names) |
| `.windup/` | Derived state: plan cache, run ledger, site map, reports | ❌ (init adds it to `.gitignore`) |

---

# Programmatic API & test runners

Run a scenario from code and get back run metrics:

```ts
import { run } from "windupjs";
const result = await run("checkout");   // RunMetrics: result, llm_calls, cost, per-action timing
```

Plug into **vitest** (jest-compatible contract) — one native test per scenario, sharing the warm engine:

```ts
// e2e/windup.test.ts — vitest
import { windupSuite } from "windupjs/vitest";
await windupSuite();                    // one native test per scenario
```

---

# Commands

| Command | Description |
|---|---|
| `windup init` | Create `windup.config.ts`, `.windup/` (gitignored) and an example scenario |
| `windup new "<instruction>" [--id x] [--force] [--depends-on ids] [--validate]` | Generate a scenario from a rough instruction; `--validate` runs and refines it until it passes (≤3 attempts) |
| `windup run [scenario]` | Run one scenario (replay when cached, plan on miss) |
| `windup run --all` | Run every scenario — CI mode |
| `windup scan [--update] [--no-assist]` | Statically index routes and interactive elements into the site map; `--update` re-indexes only files changed since the last scan (git diff); `--no-assist` skips the LLM layer (zero cost) |
| `windup costs [--last n] [--days n] [--json]` | AI usage report from the run ledger: totals, free replays, per-provider, per-model and per-scenario breakdown, scan and authoring spend |
| `windup status` | Site-map pages by source, staleness, cached scenarios, fragments |
| `windup fragment extract <scenario> <a1..aN> --id <id> --description <text>` | Promote a slice of a cached plan to a reusable fragment |
| `windup secret set <account> [--user u] [--password p]` | Register test credentials: values → `.env.local`, mapping → `windup.credentials.json` |
| `windup secret list` | Accounts + whether each ENV is set (never prints values) |
| `windup sig <url> [--repeat n]` | Structural page signature (diagnostics) |
| `windup bench <scenario>` | Full validation protocol (generation, replay determinism, failure recovery) |
| `windup cache clear` | Drop the trajectory cache (next runs re-plan) |

### `run` flags

| Flag | What it does |
|---|---|
| `--all` | Run every scenario in the directory — CI mode, one warm browser for the whole suite. Non-zero exit code if any scenario fails. |
| `--concurrency <n>` | Run up to `n` scenarios in parallel over one shared warm browser with isolated contexts — ~2× faster on a mixed suite. Sequential by default. |
| `--no-cache` | Ignore the cached plan and re-plan from scratch (forces one LLM call), even when a valid trajectory exists. Use to regenerate a plan on purpose. |
| `--no-map` | Plan without the site-map graph — skip the indexed routes and selectors. Useful for debugging the planner or a brand-new environment. |
| `--repeat <n>` | Run the scenario `n` times back-to-back over the same warm browser — stability and flake checks. |
| `--headed` | Show the browser window instead of running headless. |
| `--slowmo <ms>` | Add a delay between actions so you can watch each step — demo and debugging pace. |
| `--base-url <url>` | Override the start-URL origin for this run (dev / staging / CI). Rebases even absolute scenario URLs, preserving path and query. |
| `--browser chromium\|firefox\|webkit` | Run on the chosen engine (default Chromium). The same plan replays across all three — author once, run everywhere. |
| `--llm <provider[:model]>` | Pick the planner LLM for this run (e.g. `openai:gpt-5-mini`). Only affects planning; cached replays never call an LLM. |
| `--summary` | After the run, one extra LLM call writes a human-readable debrief quoting real values observed on the final page. Off by default so replays stay $0. |
| `--suggest` | On a **failed** run, one extra LLM call proposes a concrete fix to the scenario. Fires only on failure. |
| `--reporter junit\|json\|html` | Emit a CI report — JUnit XML, a machine-readable JSON summary, or a self-contained HTML page. |
| `--report-file <path>` | Write the report to a specific path (default `.windup/reports/`). |

## AI debrief (`--summary`)

For humans reading results (not CI), `--summary` adds one LLM call after each run that writes a short debrief: what the test did, the outcome, **concrete values observed on the final page** (prices, messages, product names — quoted literally from the page), and any difficulties (slow steps, re-planning, failures). It prints in the terminal, lands in the run ledger, and shows as a highlighted block in the HTML/JSON reports.

```bash
npx windup run checkout --summary --reporter html
# summary: "The test logged in and completed checkout for 3 items; the
#  confirmation page showed 'Thank you for your order'. Prices observed: ..."
```

Off by default on purpose — cached replays stay at zero LLM calls and $0. The debrief cost (~$0.0005 on the default model) is tracked separately in the run metrics and included in `estimated_cost_usd`.

## Fix suggestions on failure (`--suggest`)

When a run **fails**, `--suggest` adds one LLM call that acts as a senior QA engineer debugging it: it compares the executed plan and the failing step against the **real final page** and the site map's known selectors, then proposes a concrete fix to the scenario — the wrong selector and the real one, a targeted screen that doesn't hold what the task expects, a missing step, or a timeout too short for a slow page.

```bash
npx windup run create-invoice --suggest
# FAIL  create-invoice  ... element button:has-text('Save') not visible
#   suggested fix: The 'Save' button does not exist; the dialog's real button
#   is labeled 'Create'. Change the hint to button:has-text('Create').
```

It turns a red run into a specific edit — instead of reverse-engineering the app by hand. Only fires on failure (green runs cost nothing), never edits the scenario itself, and shows as a highlighted block in the HTML/JSON reports. Pairs naturally with `--summary`.

---

# Architecture &amp; spec

A condensed view of the [living specification](https://github.com/windupjs/windup/blob/main/docs/specs/SPEC.md). It describes the system as shipped.

## Thesis

Browser E2E tests can use an LLM **only for planning** (and re-planning on failure), with deterministic execution and cheap verification — so repeated runs make **zero LLM calls**. Validated end-to-end: first run plans in seconds for fractions of a cent; replays take ~0.5–1s and cost $0.

## Principles

1. **The LLM is the exception, not the rule.** One call per cache miss; replays never call it.
2. **Zero hardcoded site knowledge.** The engine may know *frameworks* (Next.js, react-router, design-system naming) and the web platform — never a specific site. All site knowledge arrives as input or is discovered at runtime.
3. **Every execution is also collection.** The executor already visits every page of a flow; persisting what it sees costs ~one `evaluate` per action.
4. **Knowledge is cache, not truth.** Anything the site map or static scan asserts can be stale; it degrades to runtime discovery. Precedence: `execution > static > llm`.
5. **Cost never surprises.** Every LLM touchpoint has an explicit cap; every call is recorded in a ledger; repeated static/assist work is memoized.
6. **Plans are data, not programs.** No conditionals, no loops, no free-form code. If a flow needs branching, split the scenario.

## Architecture

```
scenario (natural language, versioned)
    │
    ▼
┌─ runner ─────────────────────────┐
│ cache lookup (path-keyed)         │
│   miss → planner (LLM, 1 call)    │
│   hit  → cached plan (rebased)    │
│   → executor (deterministic)      │
│   → verifier (postconditions)     │─▶ fail → invalidate → re-plan
│   → cache save + run ledger       │
└───────────────────────────────────┘

site map (graph) ◀── windup scan (static + LLM-assist)
                 ◀── passive collection (every run)
```

Module boundaries (all in `packages/windup/src/`):

| Module | Responsibility |
|---|---|
| `browser.ts` | Single engine boundary (Playwright). Fresh `BrowserContext` per session on a lazy Chromium singleton; trusted clicks with native actionability; `ariaSnapshot()` feeds the planner. Targets the first *visible* match, not the first DOM match. |
| `llm.ts` | Multi-provider LLM boundary. One `LlmClient` interface; Google Gemini (SDK) and OpenAI (plain REST) implementations. Several providers configurable at once; selected per run. Switching providers never touches the plan cache. |
| `planner.ts` | Planning logic (provider-agnostic). Prompt = task + page a11y tree + real elements + site-map slice + fragment catalog + manifest + hints. Structured output; two retry levels (semantic + transient); output sanitized then validated (Ajv is the authority). |
| `executor.ts` | Deterministic loop: goto → per action (gate on visibility → act → verify). Emits passive site-map observations. |
| `verifier.ts` | Postconditions, all LLM-free: element visible, URL glob, input value. Polling with frame-safe waits. |
| `cache.ts` | Trajectory cache keyed by `scenario_id` + start-URL **path** (environment-portable). Saved only after full verified execution; a failed replay invalidates and re-plans, keeping the stale entry as evidence. |
| `signature.ts` | Structural page identity: SHA-256 of normalized interactive elements — no text, no data — so environment noise doesn't split identities. |
| `sitemap.ts` | Page/transition graph. Nodes carry `source: execution\|static\|llm` with staleness and provenance. Prompt slice: BFS (depth ≤ 3), term-scored, within a char budget. |
| `scan/` | Project indexing. Layer 1: routes by convention (Next.js, react-router). Layer 2: interactive elements via brace-aware JSX parsing (raw tags + design-system components). Layer 3: capped LLM-assist, memoized per file hash. |
| `fragments.ts` | Reusable tested sub-trajectories, committed in `e2e/fragments/`. Plans reference them by id; the cache stores the reference (updates propagate). |
| `authoring.ts` | `windup new` — the LLM rewrites a rough instruction into a precise scenario grounded in the site map and manifest; credentials auto-registered and scrubbed; output is a committed file for review. |
| `metrics.ts` / `costs.ts` | Every run writes a ledger record (tokens, calls, model, timing, failure class). Prices are a dated per-model table; `windup costs` recomputes so history stays correct. |
| `summary.ts` / `suggest.ts` | Opt-in post-run AI debrief and post-failure fix suggestion — one extra LLM call each, tracked separately, never affecting the run result. |
| `reporters.ts` | JUnit XML, JSON and self-contained HTML reports; non-zero exit on failure. |
| `secrets.ts` | Credentials without committed secrets: values in `.env.local`, account → ENV-name mapping committed; plans carry `value_ref` only, resolved at run time. |

## Data formats

**Action plan** (`plan_version: "0.1"`): `{ plan_version, scenario_id, task, start_url, generated_by, actions[] }`. An action is `{ id, type: goto|click|fill|wait_for|use, target?, value? | value_ref?, url?, use?, expect?, timeout_ms }`. Semantic rules: click/fill/wait_for require `target.selector`; fill requires exactly one of value/value_ref; `value_ref` must be mentioned by task/hints/manifest (no invented ENV names); the final action must carry `expect` (or be a `use`).

**Cache entry** (`cache_version: "0.2"`): `{ key: {scenario_id, start_url(path), start_sig?}, plan, status: active|stale, stats }`.

**Site map** (`map_version: "0.1"`): `{ last_scan_sha, pages, transitions, assist_seen }`.

## Model &amp; cost posture

Default planner model: **`gemini-3.1-flash-lite`** (measured: 1 clean call/generation, ~3–4s, ≈ $0.0025/generation). Provider-agnostic: Google Gemini and OpenAI ship in-box behind the `llm.ts` boundary. Every ledger record carries provider+model; prices are a flat per-model table. Adding a vendor = one client implementation.

## Security posture

Page content captured from the app under test is fed to the LLM **delimited and marked as untrusted data**, with an explicit instruction to treat it as data, never as instructions. Because plans are schema-validated data executed deterministically, a page cannot make Windup run arbitrary code. Credential values never enter scenarios, plans, the cache or LLM prompts. Full threat model: [SECURITY.md](https://github.com/windupjs/windup/blob/main/SECURITY.md).

## Known limitations

- Nested react-router relative paths are collected best-effort (corrected by execution observations).
- Scan is single-package aware; monorepos are detected and warned, per-app indexes are future work.
- No daemon between CLI invocations (deliberate); the warm pool is per-process.
- Fragment auto-detection is not built; extraction is manual.

## Verification

- `npm test` (packages/windup): ~97 unit/integration tests, LLM-hermetic. CI runs them with real Chromium on Ubuntu.
- `windup bench <scenario>`: the validation protocol — 5 generations (≥ 4/5 valid), 10 replays (10/10 with `llm_calls=0`), replay ≥ 5× faster, $0 replay, break-a-selector recovery.
- Real-project dogfood: a react-router app with 106 routes — plans replay at ~0.5s/$0, cache portable across environments.

---

# Engineering notes — the techniques behind Windup

A summary of the approaches that make natural-language tests deterministic and cheap:

- **Plan once, replay free.** The LLM is used exactly once per scenario (plus automatic re-planning when the app changes). Its output is a schema-validated **JSON action plan — data, not code**: no generated scripts, no conditionals, no runtime improvisation. Replays execute the cached plan with zero model calls.
- **Deterministic execution.** Plans run on Playwright with native actionability checks and trusted input events. Every action carries an explicit postcondition (`expect`: element visible / URL glob / input value) verified **LLM-free** — verification costs a DOM query, not tokens.
- **Self-healing cache.** Trajectories are cached keyed by scenario + start-URL *path* (portable across dev/staging/CI hosts). A failed verification invalidates the plan, preserves the stale entry as evidence, and triggers a re-plan with the failure as context.
- **Structural page signatures.** Pages are identified by a SHA-256 of their normalized interactive elements — no text, no data — so environment noise doesn't split identities, and start-page drift is detected (leniently) on replay.
- **Layered site knowledge.** A site-map graph feeds the planner real routes and selectors, built from three sources with strict precedence — runtime observation > static source scan > capped LLM-assist. Knowledge is cache, not truth: anything stale degrades to runtime discovery.
- **Prompt budget discipline.** The planning prompt stays ≈ constant size (~32k chars): page tree, map slice, fragments catalog, and manifest each have hard char budgets. Long prompts measurably degrade small models — budgets are a correctness feature, not an optimization.
- **Mechanical normalization over prompt hope.** Model output is sanitized deterministically: empty fields dropped, ids renumbered, `wait_for`⇄`expect` normalized, fragment-echo actions deduped, credentials scrubbed from authored scenarios. Cross-provider A/B testing showed prompt instructions alone don't hold across models — code has the final word.
- **Two-tier retry.** Semantic failures (invalid plan) get one short retry carrying the validation errors; transient API pathologies (token-loop degeneration, network) get re-calls with varied seeds. Full-prompt retries are avoided — they reliably re-trigger degeneration.
- **Composable building blocks.** Fragments are curated, committed sub-trajectories (e.g. login) that plans reference by id — updated once, propagated everywhere, expanded at run time. The project manifest injects team knowledge (conventions, accounts, vocabulary) into every plan.
- **Secrets by reference.** Credential values live in `.env.local`/CI secrets; committed files carry only account → ENV-name mappings. Plans use `value_ref`, resolved at execution time — secrets never reach the LLM, the cache, or git.
- **Provider-agnostic LLM boundary.** One interface, Google and OpenAI implementations (the OpenAI client is plain REST — no SDK weight), selectable per run. Swapping the browser engine and adding a provider were each a one-file change — the boundaries are the architecture.
- **Cost you can audit.** Every LLM touchpoint has an explicit cap and lands in a per-run ledger with tokens, model and provider; `windup costs` recomputes from a dated price table, so history stays accurate as prices move.