Chapter 25. 2026-07-16: The Silent Fallback That Took Production Down

On 2026-07-16 the Suede Agent Studio at agents.suedeai.ai was backed by a non-durable SQLite file inside a serverless function. Vercel’s Production env vars for the datastore, DB_DRIVER and the SUPABASE_* group, were pulling as empty strings; the repository layer read the empty driver value, raised nothing, and took its fallback path, holding production state for a product that exists to keep what members build.

The multi-lane audit that surfaced it came back with real P0 findings: a false save/recovery conflict that auto-persisted a draft on a mere page view, a disabled Run button, live /api/v2/* endpoints 503ing, and mobile and WCAG failures stacked on top. A list like that invites four separate investigations. The visual and UX items turned out to be what they looked like, and they got their own fix. The rest sat on a single structural fault that was not in the application code at all.

Why SQLite in a serverless function cannot hold production state

SQLite in a serverless function cannot hold production state, because the database is a file on the function instance’s local disk. Instances get recycled without notice, a fresh instance starts with nothing, and two concurrent instances each hold a private copy of the truth. A write returns success, and the success means “written to a disk that may not exist in an hour.” No exception fires anywhere in that arrangement. The endpoints answer, the canvas renders, and the store underneath is scratch paper.

Agent Studio is the builder. Members assemble node-graph flows on a canvas and launch them as x402 pay-per-call endpoints that settle USDC on Base through the Coinbase CDP facilitator. The product’s job is to hold a flow today and run it tomorrow, with money attached, and that is the promise a file on a recycled disk cannot make.

My notes do not map each P0 finding to the fallback one for one, and the mapping matters less than the mechanism. A store whose contents differ per instance and vanish on recycle is a machine for manufacturing this class of bug: state that a member saved and cannot recover, state that recovers when nobody saved it, endpoints that answer from an instance holding tables and 503 from an instance holding none. From the outside, that presents as several intermittent bugs in unrelated features. From the inside, it is one line of config resolving to the wrong branch. The worst property of the failure is that it degrades the product into something that mostly works, and “mostly works” generates bug reports instead of alarms.

The repo’s own split made the trap hard to see from a laptop. Dev runs sqlite by design. Prod is supposed to run Supabase, selected by DB_DRIVER=supabase, and there is no local .env for prod, so nothing on my machine can testify about what production is doing. The one instrument that reaches the real answer is curl against production itself. That rule sits in my operating notes now, and this is the day that wrote it.

Why an empty env var passes the check a missing one fails

An empty string is present. It passes the existence check and fails the meaning check, and a config reader with a default bolted on treats "" as a request for the default. A missing variable fails presence checks; code that requires it tends to crash early and name it in the stack trace. The distinction carried the whole failure. DB_DRIVER equal to "" is not equal to "supabase", so the driver switch chose sqlite, the same branch it chooses in dev, where that choice is correct. The behavior that makes local development pleasant masked a production outage, and masked it without one log line of complaint. My notes do not record why the vars pulled empty; the record starts at what the emptiness did.

The audit found a second layer under the empty vars, and the second layer was worse. Agent Studio’s Supabase runtime had never been implemented. Phase 9 of the master plan assumed DB_DRIVER=supabase in prod; the assumption lived in the plan and never became code. Populating the env vars alone would not have saved anything, because production had no Supabase path to take.

The June “verified live” entry, and why it proved nothing

My project notes from 2026-06-11 state that migrations were applied to prod that evening and verified live: the gateway llm endpoint returning 200 with metering, topup issuing 402 challenges with a real payTo. The same file, five weeks later, states the Supabase runtime was never implemented. Both entries sit there today.

Two readings survive contact with the evidence. Either the June verification exercised endpoints that succeeded against the SQLite fallback, since a 200 and a 402 challenge come back byte-identical no matter which database produced them, or the production env regressed somewhere between 2026-06-11 and 2026-07-16. The notes do not say which, and I decline to resolve it by picking the reading that flatters the June entry. The same file carries a second internal wrinkle: one line marks the relay migration SQL in docs/migrations/phase-8-relay.sql PENDING on prod Supabase, while a later line calls the PENDING block historical. I flag both rather than harmonize them, because a note that smooths over its own contradictions lies with confidence.

The June entry holds the sharpest lesson in the incident. On 2026-06-11 the repo also showed all 10 master-plan phases complete, 355 of 355 vitest passing, and a clean build. Green tests prove the code agrees with the tests. A green endpoint check proves the route works. Neither one identifies which backend served the request. “Verified live” was true and useless at the same time: the check measured HTTP behavior while the question was durability. A durability claim needs a durability test. Write a record, redeploy, read it back. The redeploy is the step that does the work, because it recycles the function instance and takes the scratch paper with it. Ephemeral SQLite passes a write-then-read check within one instance’s lifetime, and it passes with a straight face.

The PRs that landed on 2026-07-16

Two PRs carried the repair on 2026-07-16. PR #71 took the visual and UX findings. PR #72 took the database: proper Supabase config restored, RLS hardened, the service-role key removed from Vercel, atomic RPCs added. Agent Studio’s tables landed in the same shared Supabase project that serves the rest of the estate. That placement is deliberate and temporary, my call, and I want it on the record as a choice so no future audit reads it as an accident. An end-to-end canary passed: create, bind, version, run. PR #73 closed the day with a small fix to the persistent top nav. PR #69 landed the same day too, converting 12 Business Skills templates from decorative shells into action-capable workflows, all verified 200 end-to-end on the live site.

The terminal handoff for the day states “no release blocker remains.” My notes carry a caveat beneath it that I wrote on purpose: given how much churn happened in one day, the work deserves an independent spot-check before anything release-critical relies on it. Both lines are honest, and they belong together. A day that merges four PRs, rewires the production datastore, and tightens security posture produces claims that should be re-verified by someone who was not sprinting through the day.

Three empty-env incidents in five weeks

Agent Studio was the third empty-env incident in this estate inside five weeks.

The three incidents share one shape across three unrelated stacks. In each case a required dependency was absent, and in each case the code held an opinion about what to do instead of an error to raise. Strumly trusted forged payment proofs. The voice web app rendered a sign-in that could not sign anyone in, and Agent Studio kept production state on a disk with the lifespan of a function instance.

Strumly also carries the repair I treat as the standard now. A guard in lib/api/payment.ts makes production throw at boot unless STRUMLY_PAYMENT_MODE is exactly “live” or “stub”. An ambiguous value refuses to boot. That guard converts a live revenue hole into a failed deploy, and a failed deploy costs minutes and embarrasses no one but me.

The same Agent Studio repo had already taught the fail-open lesson on the money path. On 2026-06-11, Phase 9 shipped the settlement toggle opt-in with a default of false. Production had no settlement_live column, so each agent mapped to dry-run, and dry-run skips the x402 402 challenge. Priced agents were free to call while settlement was live. An orchestrator spot-curl caught it, and hotfix commit 12d841b inverted the default: missing or NULL means LIVE, an explicit owner toggle-off is the one way to disable settlement, and env X402_SKIP_SETTLEMENT is the sole kill-switch. The datastore fallback is the same species. Code met an absent value and chose the permissive branch. The settlement fix chose the right posture: when a required signal is missing, fail toward the behavior that protects the promise. On a money path that means charge. On a datastore path that means refuse to boot.

Fail the boot when required config is empty

A fallback that hides a missing dependency is a bug with a delay on it. The delay is the expensive part. The deploy goes green, the demo works, the first sessions look fine because a warm instance keeps its file for a while, and the failure arrives later wearing a disguise: a save/recovery conflict, a 503, a Run button that will not enable. By then the incident costs a multi-lane audit instead of a failed boot.

Four rules came out of this day.

Required config gets validated at boot, and "" counts as missing. The check belongs where the process starts, not where the value gets used, and the error should name the variable and show the value it received. The shape I want in any repo with a driver switch:

// Boot guard. JSON.stringify distinguishes "" from undefined in the message.
const driver = process.env.DB_DRIVER;
if (process.env.VERCEL_ENV === "production" && driver !== "supabase") {
  throw new Error(
    `DB_DRIVER must be "supabase" in production, got ${JSON.stringify(driver)}`
  );
}

Dev fallbacks require an explicit opt-in. sqlite selected by DB_DRIVER=sqlite is a feature. sqlite reached by falling through a condition that nothing satisfied is a trap, and the trap arms itself the moment an env var reads as "".

The datastore gets verified from production, not from config intent. For Agent Studio that means curl against agents.suedeai.ai, because dev is sqlite, prod claims Supabase, and no local file can testify about the difference.

A datastore change earns a durability test. Write, redeploy, read back. A 200 is a statement about the route. Survival across a redeploy is a statement about the database.

I keep the June note in the file above the July one, both uncorrected. The pair says more than any rule I could append under them: a verification is a claim about the check you ran, not about the system you ran it against. In June I asked production whether it could answer HTTP, and it could. The question that mattered was where production kept its memory, and the honest answers to that one are a boot-time throw or a record that outlives a redeploy.