The deploy said OK at every step and the signup form was dead

Today's job was meant to be small: ship a mailing-list signup that had been sitting finished-but-undeployed. It shipped, the deploy reported success at every step, and the form was completely dead. Nobody could sign up for about twenty minutes.

The cause is worth writing down, because it is a whole category of bug rather than one mistake — and because the tooling gave no indication anything was wrong.

What we were trying to do#

Put a double opt-in email signup live: someone enters an address, it is stored encrypted, and once confirmation email is switched on they get a link to click. Addresses are held as ciphertext with a blind index, so the database can enforce "one row per person" without ever storing a readable address.

That meant a migration: the old plaintext columns had to be converted to encrypted ones and then removed. The setup script did this in two steps — encrypt every remaining plaintext row, then drop the dead columns — immediately before deploying.

The tools#

Cloudflare Pages with Functions for the API, D1 (their SQLite) for storage, and Wrangler as the command-line tool. One idempotent setup script does the whole deploy so there is no runbook to follow by hand and no step to forget.

The trap: a CLI that returns a summary instead of rows#

The setup script read the database through Wrangler. Reads went through the same helper every other statement used, which ran the query from a temporary file and asked for JSON.

Here is the problem. Given a file, that command does not return the query's rows. It returns a summary of the run:

[{ "results": [{ "Total queries executed": 1, "Rows read": 1, "Database size (MB)": "0.06" }] }]

Ask for the same query with the inline-command flag instead and you get the actual rows. Same tool, same JSON flag, two completely different shapes — and the file form is the one you reach for the moment a statement contains a quote or a bracket, because passing SQL on a command line is its own source of pain.

So every read came back as that summary object. The check for "does the plaintext column still exist" mapped the summary's keys and found nothing that looked like a column name, which reads identically to the column is already gone. The migration steps therefore printed:

OK  Nothing to convert — the plaintext columns are already gone
OK  Already scrubbed

Both were false. Neither step had ever done anything, on any run.

Why that killed the form#

The newly deployed code inserted only the encrypted columns. The old plaintext columns were still present, still NOT NULL, and still had no default. Every insert violated a constraint, so every signup failed and the visitor was bounced to an error.

Two safe-looking facts combined into an outage: code that no longer writes a column, and a migration that believed it had removed that column.

How it was caught#

Not by the deploy, which was green throughout. By poking the live endpoint afterwards:

curl -s -D - -o /dev/null -X POST "https://example.com/api/subscribe" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "[email protected]" \
  --data-urlencode "f=hero" | grep -iE "^HTTP|^location"

A successful signup redirects to the thank-you page. This one redirected to the error path, and no row appeared in the database. That single request is now a mandatory step after every deploy. A deploy that reports success is not evidence that the thing it deployed works.

The fix, in three parts#

Repair the data first, and prove it before destroying anything. The one existing subscriber was still plaintext. Encrypting it was easy; the important part was decrypting the result and comparing it to the original before dropping the plaintext column, because that was the last moment the comparison was possible at all. Then the dead columns went.

Make the failure loud. Reads now go through a separate function that uses the inline-command form, and — this is the part that matters — it inspects what came back and aborts with a clear message if it ever sees that summary shape again:

if rows and "Total queries executed" in rows[0]:
    die("A read came back as the tool's summary object instead of rows.")

A helper that silently returns the wrong shape is far more dangerous than one that crashes. The original swallowed a parse mismatch in a try/except and returned an empty list, and every caller read that as "nothing there".

Fix what the repair exposed. With the columns finally gone, the base schema no longer matched reality: it still declared the plaintext columns and indexed one of them, so re-running setup failed. An old migration also created an index on a column that had since been dropped. Both were corrected and verified by building a brand-new local database from scratch and running the full test suite against it — 116 checks, green.

What to take from it#

• Verify by reading back, never by trusting a success message. Every gate in that

script passed while doing nothing.

• Distrust helpers that can return more than one shape. If a function can return rows

or a summary depending on which flag you used, make it assert which one it got.

• Never let a catch-all swallow a parse failure into an empty result. Empty is a valid

answer, so it hides.

• Probe the live thing after deploying. One HTTP request would have caught this in

seconds instead of twenty minutes.

• Order destructive migrations so the proof comes before the deletion. Encrypt, verify

the round trip, then drop.

The wider lesson is the one this project keeps relearning: the dangerous failures are not crashes. They are the operations that return true, print OK, and change nothing.