How they test

When you work alone, a test suite is mostly a safety net for you. In a repo where hundreds of people and their agents push changes, the suite is also how strangers prove their change works, and how the maintainers decide what to trust. T3 Code’s testing rules are written for that situation. They say what deserves a test, at which level, what you may fake, and how a test is allowed to wait.

This chapter goes level by level: unit tests and their rules, the provider replay integration tests that carry most of the weight, real-app scenarios that prove a bug fix, mobile behavior runs, and evals for model output. Then it compares all of that with fud, antre, and vibe-idx.

Decide what deserves a test

The rules live in docs/internals/testing.md, and AGENTS.md carries the same text so agents see it too. The first rule is about what to leave out:

- Test behavior that can break, such as orchestration and event-sourcing, provider protocol translation, auth and permission guards, persistence and migrations, external-format parsing, race fixes, and the logic a bug fix changed. Leave out tests for what types already guarantee, trivial logic, copy text, smoke runs, mock-call mirrors, callback wiring, and props rendered to static markup.
- Extend before you add. When an existing test or table covers the behavior, add a row or an assertion there. Write a new `it` only when no case fits.
- Test at one layer: the lowest one that fails on `main`. Do not repeat the case in a wrapper test, at each call site, or in web and mobile copies of the same logic.
- Write a browser test only for behavior that needs a real browser: layout and scrolling, focus and pointer input, real workers, or a component wired to its hooks. Logic a component calls gets a unit test beside it (`*.logic.test.ts`). A browser test file spends seconds in CI loading its module graph before its first test runs; a unit test takes milliseconds.

Two ideas here are worth slowing down on.

“The lowest one that fails on main” is a precise instruction. Before fixing a bug, you write a test that fails on the current code. Then you pick the cheapest level where that is true. If a pure function is wrong, the test belongs beside that function, not in a browser test that mounts the whole screen.

“Mock-call mirrors” means tests that assert expect(fn).toHaveBeenCalledWith(...) for every line of the code under test. They repeat the implementation, so they break on every refactor and catch nothing when behavior breaks.

The repo backs these rules with tooling. The pull request size workflow counts the test cases a PR adds. When the PR adds 20 or more, or adds a new browser test file, it comments with the count, and a new browser file gets a note that “each browser test file costs CI seconds of module loading before its first test runs; a unit test costs milliseconds” (test-delta.cjs). There is even an eval, evals/test-signal, that checks whether a cheap model can flag low-signal tests in a diff (README).

Proof also depends on the kind of bug. The same doc says logic and state bugs get a failing test first. Purely visual bugs such as spacing, colour, or a layout shift get before-and-after screenshots “and no test.” The team does not want UI tests for nitpicks.

Never wait on the clock

A flaky test passes or fails depending on timing. It usually waits a fixed time, such as “sleep 50 ms then check,” and fails on a slow machine. The T3 Code server is event-sourced, meaning every change is stored as an event and side effects run after the event commits. So the rule is to wait for the event:

- The server is event-sourced, and side effects run after the command commits. In tests, drain the effect worker (`OrchestrationEffectWorkerV2.drain`) or await the specific persisted event or `Deferred` that marks the milestone. Never wait on sleeps or polling. A test that needs a timeout to pass is wrong. Concretely:
  - `it.effect` runs on a `TestClock`; `Effect.sleep` only advances through `TestClock.adjust`. `it.live` is wall-clock and only for subprocess or network fixtures.
  - No `yieldNow` or `sleep` polling loops. Await the `Deferred`, take from the `Queue`, or drain the worker.
  - No per-test timeout argument or `retry`. The project timeout is the ceiling, and flaky tests are fixed, not retried.
  - No `performance.now()` or `Date.now()` deltas. Assert on events and counts, not elapsed time.
  - Browser tests assert through `expect.element` or `vi.waitFor`, never `requestAnimationFrame` or `transitionend`.
  - `vi.mock` factories must not spread a real module that pulls in Shiki or the diff workers.
  - `it.skip` and friends need a tracking issue on the line above: `// quarantine: #1234 <why>`.
  - Lint enforces this: `t3code/no-test-timeout-or-unreasoned-skip`, `t3code/no-frame-waits-in-browser-tests`, `t3code/no-wall-clock-in-tests`, `t3code/no-real-time-wait-in-tests`, `t3code/no-polling-loop-in-tests`.

A TestClock is a fake clock from Effect that only moves when the test tells it to. A Deferred is a value that one part of the code completes and another waits on. Both let a test wait for a fact instead of a duration.

The last bullet is the important one. These rules are not advice. They are lint rules in the repo’s own oxlint plugin. Julius Marminge’s PR #22561 added no-frame-waits-in-browser-tests and gives the motivation with numbers. It says “main was red in ~600 of ~2,900 runs since September; the genuinely flaky tests all waited on real time, scheduler budgets, or renderer frames. These rules make those tests fail lint instead of CI.” The rule’s error message names the fix:

const MESSAGE =
  "Frame and transition waits race the renderer. Assert with `expect.element`/`vi.waitFor` (auto-retry), or install `fakeTimers()` (full set) and advance the timer the component sets.";

A sibling rule, no-dom-test-environment, came from #21937. A sidebar test had used a @vitest-environment jsdom comment to run in a fake DOM instead of the real browser project. The fix moved it into a real Chromium test and banned the pattern:

const DOM_ENVIRONMENT_PACKAGE = /^(?:jsdom|happy-dom)(?:\/|$)/u;
const MESSAGE =
  "Select test environments in Vitest project configuration instead of a per-file pragma or a DOM emulator import. Web DOM tests belong in *.browser.test.{ts,tsx}.";

The reasoning is that jsdom and happy-dom imitate a browser. They do not lay out pages, so focus order, scrolling, and sizes can pass in the imitation and fail in a real browser. T3 Code keeps two kinds of tests. Logic tests run in Node, and DOM tests run in a real browser through Vitest’s browser mode. Nothing sits in between.

Replace only the provider, run everything else for real

The core of T3 Code is orchestration, which turns messages from a provider such as Codex or Claude into runs, turns, and checkpoints. This is where the hardest bugs live, and docs/orchestration-v2/testing-strategy.md describes the shape every test there follows:

command dispatch
  -> real Orchestrator
  -> real ProviderAdapter
  -> replayed ProviderRuntime transport
  -> real adapter normalizer
  -> real V2 event store/sink using production persistence semantics
  -> real V2 projection/projector
  -> real Checkpoint policy
  -> assertions

Only one line says “replayed.” Everything else says “real.” The doc then lists what is not allowed: a mocked orchestrator, a mocked provider adapter, a mocked projection reducer, and “pre-normalized domain events used as the input for adapter tests.” Allowed fakes are limited to true boundaries: the provider’s transport, the clock, random IDs, a temporary filesystem, and a temporary database.

What a fixture looks like

A fixture is a folder under testkit/fixtures. The repo has about a hundred. The simple one holds an input.ts (send one message), one recorded transcript per provider, and an output check per provider. The transcript is newline-delimited JSON recorded from a real run. Each line is either something the app must send (expect_outbound) or something the provider sent back (emit_inbound). The first lines of the Claude transcript record the model and SDK version and the exact options the app sent when it opened the session (claude_transcript.ndjson).

The check reads the final projection, which is the state the UI would display:

export function assertSimpleClaudeOutput(
  result: OrchestratorV2ScenarioResult,
  transcript: ProviderReplayTranscript,
) {
  assertBaseProjection({ result, transcript, runCount: 1, runStatuses: ["completed"] });

  const projection = projectionFor(result, transcript.scenario);
  assertSemanticProjectionIntegrity(projection);
  assertVisibleTurnItemsMirrorLocalTurnItems(projection);
  assertRunOrdinals(projection, [1]);
  assertExecutionNodeKinds(projection, ["root_turn", "assistant_message"]);
  assertConversationMessageRoles(projection, ["user", "assistant"]);
  assertTurnItemTypes(projection, ["user_message", "assistant_message"]);
  assertUserMessagesInclude(projection, [SIMPLE_PROMPT]);
  assertAssistantTextIncludes(projection, "fixture simple ok");
}

None of these assertions mention a function call. They check one completed run, a root node with one assistant message, and the user and assistant text. The strategy doc calls “exact internal function call counts” a weak assertion. You could rewrite the adapter completely and this test would still pass, as long as the user sees the same result. That is the point.

The doc also says which level matters most: “The fourth level is the most important one. It is the test that catches lifecycle mismatches such as child turns closing parent runs or checkpoints being captured too early.” Those are bugs that only show up when the adapter, the event store, and the projection run together.

Fakes that fail loudly

When a fake is needed, the repo puts it in a small package with a strict default. packages/source-control-testing gives tests a fake Git host. Every Git operation the test did not supply dies with a message:

const unexpectedGit = (operation: string) => () =>
  Effect.die(`Unexpected git ${operation} in a test that supplied none.`);

const failingGit: Git = {
  execute: unexpectedGit("execute"),

A forgiving fake that returns an empty success for everything lets a test pass while the code calls Git in ways nobody checked. This one forces each test to state the Git calls it expects. packages/provider-testing does the same job for providers. It has a TestProviderHost layer that creates a temporary state directory per test, plus helpers to read replay transcripts.

Prove a fix in the real app

A scenario is an end-to-end test of one behavior in the running app. docs/operations/scenarios.md describes each one as “a real server on its own T3 home, real HTTP and WebSocket, and a real browser.” A dev-only replay agent stands in for the model, “so a run is free and repeatable.” Here is a whole scenario:

test(
  "A replayed turn streams a reply, shows its tool calls, and finishes",
  WEB,
  async ({ t3, app, screen }) => {
    await t3.project("demo", { "src/math.ts": "export const sum = 1 + 1;\n" });
    await t3.open();

    await screen.getByRole("textbox").fill("Give me a short tour");
    await screen.getByRole("textbox").press("Enter");
    const stop = screen.getByRole("button", "Stop generation");
    await expect(
      screen.getByText("The project has a single math module", { exact: false }),
    ).toBeVisible();
    await expect(stop).toBeVisible();
    await app.screenshot("streaming");

    await expect(screen.getByText("This turn is a replay.", { exact: false })).toBeVisible();
    await expect(stop).toBeHidden();
    await expect(screen.getByText("Give me a short tour").first()).toBeVisible();
    await app.screenshot("finished");
  },
);

t3.project creates a Git repository and adds it through the app’s own CLI, so the state is real events, not rows written by hand. The labeled screenshots exist for the next step.

scripts/prove-scenario.ts runs one scenario on a base commit and a head commit, each in a temporary worktree, and compares the results. Per the doc, it “exits 0 only when the scenario fails on base and passes on head, 1 when it ran and did not prove the fix, and 2 when it could not run.” It writes a Markdown table of base and head screenshots, ready to paste into a pull request. A scenario that passes on base “does not reproduce the bug, and the verdict says so.”

The doc is also clear about limits. No CI job runs scenarios. They cannot cover desktop menus, Android, or provider-specific behavior, and “Replay streaming is simulated, so a scenario cannot show a performance regression.” Knowing what a tool cannot prove is part of using it.

For hand testing, the repo has an agent skill, .agents/skills/test-t3-app, that tells agents to drive the app in T3 Code’s own Browser panel against isolated dev state, never the developer’s real database. Its SQLite fixtures reference allows writing database rows directly for visual states such as long titles or long timelines, but warns that direct writes “do not create a coherent event history” and must not be used “to claim backend business behavior works.”

Measure behavior that varies

Some behavior is not pass or fail on a single run. T3 Code has two tools for that, and they share a shape.

vp run mobile:behavior leases an iPhone and an iPad simulator, runs a scripted fake provider, and reads the app’s own state through a debug probe: scroll positions, dropped frames, composer drafts, status labels. Each case is data with a source issue and a kind:

    id: "scroll-up-never-teleports",
    area: "scroll",
    kind: "guard",
    source: "#12072",
    experience: "Dragging up a little after scroll-to-end, the viewport jumps far away.",
    scenario: "scrollUpLongThread",
    key: "largestStepPx",
    checks: [
      { metric: "movedPx", min: 300 },
      { metric: "teleports", equals: 0 },
    ],
  },

A bad case reproduces a known bug and fails on main until it is fixed. A guard must keep passing. A control is ordinary use, there to catch a fix that overcorrects. The README says to add a case before its fix and “flip it from bad to guard when the fix lands.” At this commit the file has 17 guards, 5 controls, and 3 bad cases.

Evals apply the same idea to model output. The evals/README.md rules say each case reproduces a real bad experience and names its source, must fail on the old text and pass on the new, and runs several times because “model output varies.” A case fails when half or more of its samples fail. Evals import prompts and tool descriptions from source instead of copying them, so the eval always tests the current text. They call paid models, so CI does not run them. CI does run the graders’ unit tests.

Keeping the suite fast: test:cached

A big suite needs a way to skip work. scripts/test-cached.ts turns each test file into its own vite-plus task. The task’s cache input is “every file it read” (cached-tests.ts), so a test whose inputs did not change is replayed from cache. CI’s server shards run it with --verify-hits 1, which picks one cached hit at random and reruns it without the cache. That checks the cache is not hiding a failure. The transfer budget test is marked --uncached because it writes a report that must be fresh.

How this compares with fud, antre, and vibe-idx

Your repos already share T3 Code’s most important instinct, which is to run the real thing at the boundary.

fud tests its Worker against Miniflare, the local Cloudflare runtime, with a real D1 database. tests/migration.test.ts spins one up per test, and its title reads like a T3 Code assertion list. It is “Convex import preserves scoped links and monthly totals on retry without reading bodies or sending Alerts.” The tests/fixtures folder holds dozens of real BCA, Grab, and Apple email layouts. That is the same idea as T3 Code’s recorded provider transcripts, where real external input goes through the real parser.

antre uses convex-test, which runs your real Convex functions against an in-memory database. ownerTest builds it from your real schema and modules with a WorkOS-shaped identity. The scheduling tests drive time with vi.setSystemTime and finish scheduled work with t.finishAllScheduledFunctions(vi.runAllTimers). That is the TestClock idea in Vitest form. The test moves the clock on purpose and never sleeps. Its vite.config.ts also splits tests into convex and server projects by environment, which is what T3 Code’s no-dom-test-environment rule asks for.

vibe-idx has a GitHub Actions ci.yml that runs vp check and each app’s tests, and a Playwright hillclimb harness that answers a frozen list of trader questions and counts the steps each one takes. That harness is closer to a T3 Code eval than to a unit test, and its merge commit for PR #106 reports cutting the steps “from 181 to 85.”

Now the differences. T3 Code’s lint rules would reject three patterns in your code.

First, per-file DOM pragmas. fud has // @vitest-environment happy-dom at the top of five test files, including snapshot.test.ts, and vibe-idx has it in App.test.tsx. For logic and rendering checks that is fine. For scroll and focus it can pass in the imitation and fail in Safari.

Second, waits on real time. tests/nota.test.ts waits 20 ms with setTimeout and then waits one requestAnimationFrame. Those are the exact patterns #22561 found behind T3 Code’s red main.

Third, a raised timeout. Your fud PR #30 fixed a test that timed out on Workers Builds by giving it 30 seconds:

  const payee = await new Ledger(limited, "alice").payee("GRAB", "2026-10-24");
  expect(payee.count).toBe(61);
  expect(payee.transactions.every(({ trip }) => trip?.from === fixture.receipt.from)).toBe(true);
  // 65 sequential local D1 imports run past Vitest's 5s default on shared build machines.
}, 30_000);

The PR description is good. It says the test “checks a query count, not speed,” which is the same reasoning T3 Code uses. T3 Code’s rule would still reject the per-test timeout, because “a test that needs a timeout to pass is wrong.” Their fix would shrink the fixture until it still proves the query budget, or move the 65 imports into one batch. For you the timeout was a reasonable unblock. Just know that every raised timeout makes the next one easier to add.

Some of T3 Code’s machinery would be overkill for you. You have one provider per project, so a generic replay framework is unnecessary. A recorded fixture per external API is enough. Mobile behavior sampling and paid evals make sense at 500,000 users. A solo app is better served by one scenario that proves a bug fix.

Try it

1. Remove a real-time wait from fud. In fud/tests/nota.test.ts, find the setTimeout(resolve, 20) and requestAnimationFrame waits near line 903. Replace them with vi.waitFor(() => expect(...)) on the URL or element the next line checks, or with fake timers you advance on purpose. To check it worked, run for i in $(seq 10); do taskset -c 0 pnpm vitest run tests/nota.test.ts || break; done. Pinning the run to one CPU core with taskset slows it down, and every run should still pass.
2. Ban the pattern in lint. In fud/eslint.config.mjs, add a no-restricted-syntax entry for test files that rejects new Promise((resolve) => setTimeout(resolve, ...)), with a message that names the fix, as T3 Code’s no-frame-waits-in-browser-tests does. To check it worked, put one of the old waits back in a scratch copy and confirm pnpm lint reports it.
3. Prove your next antre fix with a failing test first. Pick the next scheduling bug you hit in antre. Before touching convex/posts.ts, add a case to convex/scheduling.test.ts that uses ownerTest() and vi.setSystemTime to reproduce it. Commit the failing test, then the fix, and paste both test runs into the pull request description. To check it worked, git checkout HEAD~1 -- convex/posts.ts && npm test must fail.

For how these tests show up as required checks on a pull request, see CI and pull request hygiene. For how T3 Code turns repeated review notes into rules like these, see Turning review comments into lint rules.