Chapter 33. Deploy Discipline: Pins, Promotions, and Silent No-ops

When Vercel reports Ready and vercel promote returns a 409 saying already current, the per-domain alias records are what is stale; the project’s production pointer is the only thing that 409 describes. Those are two separate claims, and on 2026-06-10 they had been out of sync on suede-ai-app for a week. Every trap collected here has one shape: a green status somewhere in the pipeline describes a different object than the one I care about. A build that fails is cheap. The expensive failure is the deploy that reports Ready and serves nothing new.

Why a 409 from vercel promote is not a health check

Symptom: a week of merged PRs on suede-ai-app produced deployments that all went Ready, while the live domains kept serving old content. The complaint, in my own head before I had any diagnosis, was “merged but prod looks old.”

I ran vercel promote and got a 409, “already current.” For about a minute I read that as good news.

Cause: every suede-ai-app domain had been aliased by hand to a Jun 4 deployment. The project’s production pointer was healthy, which is the only thing the 409 was reporting. The per-domain alias records were stale, and the promote path does not touch them. A correct project pointer coexists with stale domain aliases for as long as nobody checks, and neither one complains about the other.

Countermeasure, run per domain:

npx vercel alias set <new-deployment-url> <domain> --scope suede-ai-64d39175

And a diagnostic that answers the real question instead of the adjacent one. vercel inspect https://<domain> reports which deployment serves that host. Compare it against vercel ls <project>, which reports what the project has built. When those two disagree, the domain is pinned.

I hit this on 2026-06-10 and hit it again on 2026-07-10. The second time is the interesting one, because I had already written the fix down. The fix was correct and the diagnosis was correct, and the underlying condition kept regenerating, which meant I had been treating a symptom.

Why the bare domain goes stale while www. stays fresh

The condition underneath the repeat: bare studio.suedeai.ai, guitarhub.org, and distro.suedeai.ai kept going stale after production deploys while their www. variants rode along fine.

The www. hosts were attached to the suede-ai-app project. The bare hosts existed only as alias records, and an alias record never auto-follows a production deploy. That split is the whole tell. Two of them were also held elsewhere: guitarhub.org still belonged to an orphan project named guitarhub, created 2026-06-11, and distro.suedeai.ai plus distro.suedeai.xyz belonged to an orphan project named suede-distro, created 2026-04-25.

On 2026-07-10 I attached all three bare domains to suede-ai-app through the Vercel API:

POST /v10/projects/{id}/domains

guitarhub.org came off guitarhub first; the distro pair came off suede-distro. Both orphan projects still exist with no domains on them. I have not deleted either one.

The audit that would have caught this years earlier takes one call:

GET /v9/projects/{id}/domains

Read it for attached versus alias-only. Anything alias-only is a future stale domain wearing a working costume.

A related version of the same confusion cost me a payment path. On 2026-05-17 all app.suedeai.xyz traffic 308-redirected to the marketing site and landed on a 404. The domain had been attached to the suede-home project with redirect='suedeai.ai' and redirectStatusCode=308, so requests never reached the app project. Vercel requires a domain-level redirect target to live in the same project as the domain, which is why the repair took two moves rather than one:

DELETE /v9/projects/suede-home/domains/app.suedeai.xyz
POST /v10/projects/suede-ai-app/domains {"name":"app.suedeai.xyz"}
PATCH /v9/projects/suede-ai-app/domains/app.suedeai.xyz
{"redirect":"app.suedeai.ai","redirectStatusCode":308}

End state: app.suedeai.xyz/* 308s to app.suedeai.ai/* with the path preserved. The live Stripe webhook had been configured against that same broken host, and Stripe does not follow redirects on webhook delivery, so deliveries died on the 404. I repointed both endpoints, live we_1TSTRVRdcsaZ58FLjeCn43SY and test we_1SHSMORdcsaZ58FLMnypb8AS, to https://app.suedeai.ai/api/payments/stripe/webhook. A webhook URL is a destination host, never an alias and never a redirecting host.

vercel alias set needs --scope or it looks at the wrong account

On 2026-07-13, shipping PR #53, vercel alias set errored with “can’t find deployment under context jason-colapietro.” The command had resolved against my personal context instead of the team that owns the deployment, so the deployment URL was invisible to it. The deployment existed. The lookup was pointed at the wrong account.

vercel alias set <dpl-url> agents.suedeai.ai --scope suede-ai-64d39175

--scope suede-ai-64d39175 goes on every alias operation across this estate. The default context is the personal account, and the deployments belong to the team.

The same class of pin bit the email work. On 2026-06-10, during the Resend activation, ip.suedeai.ai was serving an ancient deployment, so the email-sending routes I had just deployed were not live on the domain. vercel alias set fixed it. Worth naming the disguise this failure wears: it does not present as a deploy failure. It presents as “my new route isn’t there,” or as an endpoint behaving like an older version of itself. When a route I just deployed 404s, the alias is the first thing I check, before I go reading my own routing code.

Vercel projects that do not build on push

Merge does not mean deployed. Two projects in this estate taught me that in the same week.

On 2026-07-09 I merged PR #32 to main on suede-agent-studio and no production deploy appeared. The latest deployment was still 12 hours old. The Vercel project behind it, named agentix, is not wired to build on push or merge to main. Nothing errors. The merge succeeds, GitHub is happy, and production keeps serving whatever it served yesterday.

The manual ship for that project is four steps and then a verification:

git worktree add../suede-agent-studio.deploy origin/main
cp -R.vercel../suede-agent-studio.deploy/.vercel
cd../suede-agent-studio.deploy && vercel --prod --yes # vercel whoami must be suede-ai
vercel alias set <dpl-url> agents.suedeai.ai --scope suede-ai-64d39175

The alias step is not optional cleanup. A fresh production deploy creates a deployment; it does not move an alias-only domain onto it. Skipping step four produces a build that is Ready and a domain that is unchanged, which is the 409 story again from a different direction.

Step five is a curl against the public surface: curl https://agents.suedeai.ai/robots.txt and the /.well-known/x402 endpoint. Ready describes a build. It says nothing about what a host serves.

suede-muse was the worse case, because the project lied in writing. Its own CLAUDE.md claimed “push to main → production.” On 2026-07-09 a push sat on origin/main for more than 30 minutes with no new Vercel deployment. The project had link: null, no GitHub connection at all, and gh api repos/JasonColapietro/suede-muse/hooks returned zero webhooks.

The structural reason it stayed broken is the part worth carrying forward. The Vercel team account’s working GitHub identity had the Vercel GitHub App installed only on the Suede-AI GitHub namespace, never on JasonColapietro, where the real suede-muse repo lives. Every vercel git connect attempt and every dashboard connect flow could see only Suede-AI repos. No amount of CLI retrying reaches a repo the installation cannot see. I installed the app on JasonColapietro scoped to that single repo, installation 145762179, through github.com/apps/vercel/installations/new under “Only select repositories.” That alone was not enough: GET /v1/integrations/git-namespaces?provider=github still returned only Suede-AI. Visiting Vercel’s OAuth App connection page at github.com/settings/connections/applications/Iv1.9d7d662ea00b8481 appears to have triggered the namespace refresh, and the namespace switcher, not the CLI, is what finished the job.

The project now reports link.org: "JasonColapietro" and link.repo: "suede-muse" on GET /v9/projects/prj_4LIbMgjOxj5XjNO1Teu19onHmbXg, which is auto-deploy on push where there was none before. The link was confirmed working on 2026-07-10, after a GitHub App install and a namespace re-sync finished what the CLI could not. A repo’s documentation claiming push-to-deploy is not evidence of push-to-deploy. Read link.org and link.repo off the API.

One more suede-muse trap, undated but permanent: a manual vercel --prod fails when I invoke it from inside website/. Root Directory website is a project setting that Vercel applies on its own, so running the CLI from inside that directory applies the prefix twice. Deploy from the repo root, ~/code/suede-muse.

The wrong git link fails checks on another repo’s PRs

At 04:52:21 EDT on 2026-07-10, a dashboard click connected the suede-muse project to the only namespace visible to it, Suede-AI/Suede-AI-App, instead of the intended suede-muse repo. The click went through a Chrome tab a session had navigated to while that session’s own click and read tools were erroring, which is its own lesson about acting through a surface you cannot observe.

The blast radius landed on a different repository. A failing “Vercel, suede-muse” status check appeared on every Suede-AI-App PR from #543 onward and never cleared, one example failed deployment being dpl_CnUKBRUn7XLyE6KyT7PVKxwQFijP.

cd ~/code/suede-muse && vercel git disconnect --yes

I verified the repair two ways, because one way was not enough to trust: the API’s link field reading null, and a throwaway Suede-AI-App PR, #548, whose check list I could count. It collected its 6 legitimate Vercel checks and no muse check.

There is a coda I keep in the record on purpose. At about 10:57 UTC that same day, a sibling agent session deleted the written account of this incident as fabricated prompt injection. It had gh pr list output showing stale PR numbers, concluded the referenced PRs did not exist, and threw out the whole record. The incident was real, confirmed against Vercel’s API and event log, and the record was restored and superseded. Stale tool output can make a true record look invented. Before deleting a prior record as false, confirm against the authoritative system rather than a single CLI listing that may be serving cache.

vercel redeploy returns cancelled with an empty git diff

Symptom, 2026-06-10: vercel redeploy on Suede-AI-App subdirectory projects came back cancelled. No build ran. The reason given was an empty git diff.

Cause: those subdirectory projects carry a path-scoped ignoreCommand, which evaluates against the git diff. Redeploying an existing commit presents no diff inside the scoped path, so the ignore rule does its job and cancels the build. The mechanism is working. It is answering “did anything change in my path” with a correct no.

Countermeasure: stop using redeploy for this. Run a fresh vercel deploy --prod from an origin/main worktree. CLI production deploys skip the ignored-build step, so the build runs. vercel redeploy and vercel deploy --prod are not interchangeable commands when a path-scoped ignore rule is in play, and the difference only shows up as a cancellation that looks like a no-op.

Turning preview deploys off with an ignore command

I do not use preview deploys, and I pay for the builds they generate. Every repo connected to Vercel carries an ignore command at the Root Directory that Vercel uses, which exits 0 (skip) for anything that is not production and exits 1 (build) for production:

{
"ignoreCommand": "[ \"$VERCEL_ENV\" != \"production\" ] && exit 0 || exit 1"
}

That rule kills previews and leaves production intact. A --prod CLI deploy runs with VERCEL_ENV=production, so it still builds, which is what makes the manual ship rituals above compatible with the suppression.

I have no dollar figure to put behind the rule, which is worth admitting in a chapter about discipline: I adopted it on the reasoning that a preview build I never open is a build I am paying to throw away, and I never measured what it saved.

The gap: a Vercel project with no vercel.json at its Root Directory has nothing to read, so the rule does not apply and previews keep building. Those projects need the equivalent setting through the dashboard. Checked on 2026-07-27: of 23 Vercel-linked repos on this machine, four carry no vercel.json at any root the platform would read, which are guitarhub, suede-composer, suede-muse, and suede-org-site-local. Those four keep building previews until someone sets the equivalent in the dashboard, and a rule that silently exempts four projects is a rule I only half enforce.

A merge that shipped nothing: the wrong branch in the push refspec

On 2026-07-11 I merged PR #570 on suede-home and it changed nothing real. Its title said “wave 8.” The merge commit message said “wave 6 satellite pages.”

The cause was a copy-paste in a push refspec:

git push origin home-wave6-20260710:home-wave8-20260710

That pushed the contents of a stale local branch, home-wave6-20260710, which had never been re-synced after the wave-6 squash-merge, under the wave-8 name. The branch holding the real wave-8 commit was home-wave7-20260710. Compounding it, an unrelated commit 9619a2dc had landed on main in the gap and the stale branch predated it.

The only signal that anything was wrong: a 10-minute live poll timed out. The normal deploy-to-live baseline for that project is about 75 to 90 seconds. A merge that deploys nothing produces no error anywhere in GitHub or Vercel, so the timing anomaly was the entire alarm.

I confirmed there was no regression from #570 by checking file contents on origin/main, caught a pending 25-line deletion with git diff origin/main <branch> --stat before merging the correction, rebased onto origin/main, and shipped the real fix as PR #571, merge 2ad0156. Both stale branches went away afterward.

The standing rule that came out of it: before any merge on that repo, run git diff origin/main <branch> --stat and confirm the file list matches intent. Commit messages lie, PR titles lie, and a first gh pr diff call under-reported before a rebase. After the merge, read the merge commit’s actual files with gh api repos/.../commits/<sha> --jq '.files[].filename'.

Empty bodies and empty env vars: read production from outside

The through-line across all of it: the config I intended and the behavior production exhibits are separate facts, and only one of them is checkable from outside.

Two incidents make the case harder than “curl your site.” Until 2026-06-11 Suede Social served an empty server-rendered body on every route, because the root frontend/src/components/theme-provider.tsx did if (!mounted) return null; around the whole app body. The build was fine. The deploy was fine. Crawlers saw nothing. PR #290, commit f6d2dd7, made children always render on the server and moved the dark-default localStorage migration into a blocking inline script ahead of next-themes’ pre-paint script, with suppressHydrationWarning on <html>. The spot-check I run after prod deploys now is curl -s <domain> | grep '<h1' on each domain, which would have caught it on day one.

Then 2026-07-16 on agents.suedeai.ai. Vercel production env vars DB_DRIVER and SUPABASE_* were pulling as empty strings, and the code took a fallback path to a non-durable SQLite repo inside serverless. No error. A same-day multi-lane audit found a false save/recovery conflict that auto-persisted a draft on page view, a disabled Run button, live /api/v2/* endpoints returning 503, and mobile and WCAG failures. PR #71 covered visual and UX, PR #72 covered DB security, and proper Supabase config was restored. Agent Studio’s tables now sit in the same shared Suede Supabase project as the rest of the estate, a temporary arrangement I chose rather than an accident, with RLS hardened, the service-role key removed from Vercel, atomic RPCs added, and an end-to-end canary covering create, bind, version, and run. Given how much churn happened in one day, that state deserves an independent spot-check before anyone relies on it.

An env var that reads as an empty string is worse than a missing one, because empty takes the fallback branch and missing throws. vercel env pull compounds this: it returns empty values for sensitive production variables and writes the file anyway, and even ANTHROPIC_API_KEY reads 0 characters. Use vercel env ls to confirm the variable names exist, and read runtime configuration off the live surface, such as /.well-known/x402, instead of off a local env file.