Releases and observability

When you work alone, “release” usually means “push to the main branch and let the host build it”. That is a fine default. It stops being enough when other people install your software, keep it for months, and upgrade it on their own schedule. T3 Code ships a desktop app for three operating systems, a command-line server, a hosted web app, and phone apps, and all of them have to agree on versions.

This chapter reads the release workflow and the observability docs closely. It picks four mechanisms. A nightly build doubles as the release candidate. The release job waits for every check and a real launch. Updates can roll back. Telemetry stays useful without collecting what users type. Then it compares them with how you ship fud, monosodiums, and antre.

Nightly is the release candidate

T3 Code has three release channels. A channel is a separate stream of builds that users opt into. The release checklist lists them:

Each channel also has its own app icon. The assets/ folder holds a dev, a nightly, and a prod icon project, so you can tell from the dock which build you are running.

Why stable builds an old commit

The interesting rule is that a stable release does not build the newest commit on main. Theo opened PR #10410 to change this, and its description explains why. “We merge hundreds of PRs a day. By the time a nightly is verified, main has moved on, so cutting a stable from main HEAD ships code nobody has run.” The fix is a first job, resolve_commit, that picks the commit every later job checks out:

const releaseChannel = context.eventName === 'schedule'
? 'nightly'
: context.eventName === 'workflow_dispatch'
? dispatchChannel
: 'stable';

await assertReleaseSource({ github, context, releaseChannel });

if (context.eventName === 'schedule') {
core.setOutput('has_changes', await shouldReleaseNightly({ github, context, core }));
core.setOutput('ref', context.sha);
} else if (context.eventName === 'workflow_dispatch' && dispatchChannel === 'stable') {
const { tag, sha, version } = await resolveLatestNightlyCommit({ github, context, core });
await assertCommitOnDefaultBranch({ github, context, sha });
core.notice(`Stable release builds ${sha}, the commit shipped by ${tag}.`);
core.setOutput('ref', sha);
core.setOutput('nightly_version', version);
} else {
core.setOutput('ref', context.sha);
}

A scheduled run builds the current commit. A manual stable run looks up the newest published nightly and builds its commit instead. You can confirm this in the tags. v0.1.9 and v0.1.9-nightly.20261111.4515 point at the same commit, 24e7747, and so do v0.1.8 and its nightly.

$ git tag --points-at $(git rev-list -n1 v0.1.9)
v0.1.9-nightly.20261111.4515
v0.1.9

How often a nightly ships

The schedule fires every 30 minutes, but most runs build nothing. shouldReleaseNightly skips the run unless six hours have passed since the last nightly and main has new commits:

const tag = lastNightly.tagName;
if (isInGap(lastNightly)) {
core.info(`Nightly ${tag} was published less than six hours ago. Skipping.`);
return false;
}

const { data: comparison } = await github.rest.repos.compareCommitsWithBasehead({
...context.repo,
basehead: `${tag}...${context.sha}`,
per_page: 1,
});
if (comparison.status !== "ahead") {
core.info(`Candidate commit is ${comparison.status} relative to ${tag}. Skipping.`);
return false;
}

core.info(`New commits since ${tag}, and the six-hour gap has passed.`);
return true;

The repo has 692 nightly tags and 51 plain stable tags at the pinned commit. The last six stable releases, from Oct 28 to Nov 11, came one to six days apart.

The preview channel exists so the team can run the whole release, including signing and publishing, on a branch that has not merged. The checklist is careful that no normal user can receive it. Preview builds “carry no update feed”, and the npm package goes to a preview tag “which nothing resolves unless asked for by name” (release.md).

Nothing publishes until it has launched

The job that publishes the GitHub Release lists every check and every platform build it depends on. GitHub Actions skips a job when a job it needs fails, and the if: line spells out that each one must have succeeded:

release:
name: Publish GitHub Release
needs:
[
preflight,
relay_public_config,
quality,
test,
test_web,
test_server,
desktop_mac_arm64,
desktop_mac_x64,
desktop_linux_x64,
desktop_linux_arm64,
desktop_win_x64,
desktop_win_arm64,
]

A launch check with old data

Passing tests do not prove that the packaged app starts. Packaging can drop a native library, and a database migration can fail on real data. So the release opens the built app, the same way a user would. scripts/smoke-desktop-app.ts launches the packaged Electron app with Playwright, opens a thread, and checks that its messages render. The database it starts from is old on purpose:

// A snapshot from an older build on purpose: the packaged server must migrate it forward, as it
// does for every upgrading user. It holds one project and two threads, nothing else.
const FIXTURE_DB = NodePath.join(import.meta.dirname, "fixtures/desktop-launch/statev2.sqlite");

The checklist tells the next person who touches that file, “Do not regenerate it on schema changes” (release.md). A fresh fixture would only prove that new code can read new data. The old fixture proves that the migration path still works, which is the thing upgrading users depend on.

Publish in the order users will notice

After the release exists, two more jobs follow, and the order is a rule the docs call the server self-update release invariant. A connected server updates itself to the exact version of the client that talks to it. If the hosted web app moved to a new version before that version’s server archive existed, the Update server button would point at nothing. So:

1. release publishes the GitHub Release, with the server archives and their checksums.
2. publish_cli publishes the t3 npm launcher, which downloads from that release.
3. deploy_web moves the hosted web app to the new version. Its needs line is [preflight, build_web, release, publish_cli].

The web app is built early, while the desktop builds run, with vercel deploy --prod --skip-domain. That flag uploads the build without pointing any public domain at it. The last step only runs vercel alias set to move the domain. The slow work happens in parallel, and the change users see happens last, in one quick step.

The same caution shows up on the desktop. The desktop auto-update notes say there is “No automatic download or install”. The app shows an update button, and the user decides when to restart.

Updates that can undo themselves

T3 Code can run as a background service on a machine you reach from your phone. That server updates itself, and nobody is watching when it does. The server updates doc describes a two-step commit. A small, stable launcher process owns the update. It records the pending update to disk, stops the old server, and starts the new one as a trial. The trial must finish its migrations and bind its HTTP port, then report prepared. Only then does the launcher record the new version as committed.

Before the trial starts, the launcher copies the SQLite files:

Before stopping the old child, the launcher refuses an update whose snapshot will not fit on disk, so a full disk does not cause an outage. After the old child exits, the launcher snapshots SQLite’s main file, WAL, and shared-memory file. This makes trial migrations reversible without down migrations.

A “down migration” is a hand-written script that reverses a schema change. T3 Code does not write them. It keeps a copy of the whole database from before the change and restores it if the trial fails. The idea transfers to almost any app with a database. Take a snapshot you can restore before you migrate, and you need far less cleverness in the migration itself.

Watching software on someone else’s computer

A web service runs on your servers, so its logs are yours. T3 Code’s server runs on the user’s laptop. The observability doc describes one model built around that fact. Logs go to the terminal for humans. Every finished span goes to a local file, server.trace.ndjson. A span is one timed unit of work, such as one WebSocket request or one git command, with a name, a duration, and attributes. Exporting to a real tracing backend is optional.

Because the record lives on the user’s disk, the server ships its own reader. t3 trace summary --since 30m prints counts and latency percentiles per span name, and it works “while the server is stalled or stopped” (observability.md).

Where detail belongs

The doc gives rules for adding tracing to new code. Two of them apply to any project with logs:

• Put span boundaries at RPC methods, provider calls, external processes, and persistence writes. Most helpers should not create their own span (observability.md).
• Put IDs, paths, and other per-request detail on spans. Keep metric labels to a small fixed set such as “operation kind” or “outcome”, never raw thread IDs, file paths, or prompts (observability.md).

“Cardinality” is the number of distinct values a field can take. A metric labelled by thread ID creates a new time series for every thread, which gets expensive and unreadable fast. A span can carry the thread ID because it is a single record.

The T3 Connect relay, a service the team does run, sends its traces to Axiom, a hosted log and trace store. The relay observability doc gives the Worker, the phone app, and other relay clients each their own ingest token. The same doc calls those tokens “write-only credentials”, so a token copied out of the phone app cannot read traces back.

Product events and their boundary

Product analytics is a separate stream. The server, not the browser, sends events to the team’s own endpoint, which archives them and forwards some to PostHog. The product analytics doc draws the line in one sentence. “Do not send prompts, authentication material, raw provider payloads, user-assigned device names, or conversation identifiers.” The user-facing telemetry page repeats that list for users and tells them how to turn it off with T3CODE_TELEMETRY_ENABLED=false.

Sending events from a laptop means the network will fail. The sender in AnalyticsService.ts retries with backoff, but it also knows when to stop:

// A failed batch is tried again until it has failed for this long. Ingest answers 503 when it
// cannot store a batch, and a broken Cloudflare location can take about 5 minutes to fail over.
// The waiting batch holds newer events back; `maxBufferedEvents` bounds them meanwhile.
const MAX_BATCH_RETRY_MS = 30 * 60_000;
const RETRY_BASE_DELAY_MS = 2_000;
// ...
const RETRY_MAX_DELAY_MS = 300_000;
// These statuses reject the batch itself; unlike 408/429, retrying cannot help.
const REJECTED_BATCH_STATUSES = new Set([400, 401, 413]);

/** `random` is in [0, 1). */
function retryDelayMs(failures: number, random: number): number {
const ceiling = Math.min(RETRY_MAX_DELAY_MS, RETRY_BASE_DELAY_MS * 2 ** (failures - 1));
return Math.round(ceiling / 2 + (ceiling / 2) * random);
}

Those lines make three decisions. The wait doubles from about 2 seconds up to 5 minutes, with some randomness, likely so that many laptops do not retry at the same instant. A batch the endpoint rejects as malformed is dropped at once, because sending it again cannot succeed. Any other failure is retried for 30 minutes and then dropped, so a dead endpoint cannot fill memory. The 30-minute number came from an incident. Theo’s PR #25474 says the old limit gave up after about 1.5 minutes, and that during a 5-minute Cloudflare failure on Oct 25, 2026, “about 160 events were lost”.

How your projects ship, and what they watch

fud

fud is the closest to T3 Code. Its Workers Builds build command is pnpm build:cloudflare, which installs the frozen lockfile and runs pnpm verify. verify runs lint, both type checks, the tests, the build, two Wrangler dry runs, and a check of the built Worker. A failing test blocks the deploy, which is the same promise as T3 Code’s release job. The deployment guide adds branch previews behind Cloudflare Access and a public demo host with no secrets. Those play the role of T3 Code’s preview channel. The guide’s cutover record also notes the Vercel move and how to roll it back, which is the kind of note T3 Code keeps in docs/operations/.

Logging is careful in the same direction as T3 Code’s collection boundary. Both wrangler configs turn on Workers observability with redact_query_string, and the request handler logs a fixed string:

// Do not put provider errors, OAuth URLs, email content or credentials in logs/responses.
console.error("fud_request_failed");

That line keeps credentials out of logs. It also throws away every clue. When this fires you will know that something failed, but not which route, which method, or what kind of error. T3 Code’s answer is to log a small set of safe, fixed fields, such as a span name and a failure_code, and never the content.

The other gap is migrations. The README applies production migrations by hand with wrangler d1 migrations apply fud --env production --remote, with no snapshot step before it.

monosodiums

monosodiums deploys through Cloudflare Workers Builds on every push to master, as the README says. The manual path is vite build && wrangler deploy. Neither path runs npm run lint or scripts/check-world.mjs, even though check-world.mjs is already a launch check. It serves the built dist/, opens it in Chromium, waits for frames, and fails on page errors. That is the same idea as smoke-desktop-app.ts, and it only needs to be wired in.

For watching production, the site loads Cloudflare Web Analytics:

<script
type="module"
src="https://static.cloudflareinsights.com/beacon.min.js"
data-cf-beacon='{"token": "8121c077dc97411dabf699e85376081a"}'
></script>

That counts visits and page timing without cookies, which is the right size for a portfolio. It will not tell you when a visitor’s browser fails to load a building model. buildings.js logs that to the visitor’s own console, where you will never see it.

antre

antre deploys on Vercel. Its vercel.json runs npx convex deploy and the build for production and only the build otherwise. The project has a test suite (npm test) and a type check (npm run check), but the build command runs neither. A failing test does not stop a production deploy. The public workspaces rollout doc does something T3 Code also does well. It writes down which checks ran, which did not, and the commands to run in order.

What would be overkill

Release channels, six signed desktop builds, a staged web alias, and a self-updating launcher all exist because T3 Code is installed software with many users on many versions. Your apps are web apps. Visitors load your latest deploy when they open the site, so you have few old versions in the wild to stay compatible with. Branch previews, which fud has, already give you most of what a nightly gives T3 Code. A product analytics pipeline with its own ingest endpoint is also more than a solo project needs. The host’s built-in analytics and logs are enough until you have a question they cannot answer.

Try it

1. Gate Antre’s deploy on its tests. In antre/vercel.json, put npm run check && npm test && in front of the existing buildCommand, so the tests run before npx convex deploy or the build. Push a branch with one test changed to fail. The Vercel preview build should fail at the test step. Revert the test and the build should pass.
2. Run Monosodiums’ launch check before it goes live. Add a verify script to monosodiums/package.json that runs npm run lint && npm run build && npm run check:world. Add a GitHub Actions workflow that runs npm ci, npx playwright install --with-deps chromium, and npm run verify on every pull request. Open a PR that throws an error in src/main.js. The check should fail on the page error. If it fails on a clean branch because WebGL cannot start on the hosted runner, that is a limit of the runner, not your site. In that case keep check:world as a local step before each merge. Then set the Workers Builds build command to at least npm ci && npm run lint && npm run build, so a direct push to master cannot skip lint.
3. Make Fud’s failure log useful. In fud/worker/index.ts, replace console.error("fud_request_failed") with console.error({ event: "fud_request_failed", method: request.method, route, error: error instanceof Error ? error.name : typeof error }). Set route to the path only when it is one of the fixed route strings the handler checks, and to "other" otherwise. Keep the existing comment and add nothing else. Trigger a failure in wrangler dev and confirm the log shows the route and error name but no query string, email text, or token.