fix(poller): cap Redis memory, and stop retrying periodic polls (v0.23.18-beta)
The two follow-ups left open by v0.23.17. One is a real fix; the other turned out to need explaining rather than fixing. **Redis had no memory ceiling.** v0.23.17 stopped the poller from growing Redis without bound, but nothing stopped anything else from doing so, and the host had no protection either way. Redis now runs with a 512 MB cap (REDIS_MAXMEMORY). The eviction policy is the part worth care. `allkeys-lru` would be actively dangerous here: BullMQ job data carries no TTL, so an LRU sweep across all keys could discard queued work with no trace. `volatile-lru` evicts only keys that set an expiry — which is exactly the poller's caches and in-flight claims, all of which are safe to lose because losing one merely causes an earlier re-poll. Verified rather than assumed: sampled BullMQ job keys report ttl=-1 and poller cache keys 528-3408s. If pressure ever exhausts the volatile keys Redis refuses writes instead of evicting queue state, which is loud and survivable. **The 14% fast-poll failure rate is not a bug.** Current rate across all devices and job kinds is 0% over 1,462 attempts. The historical 230,707 failures were genuine device downtime: device_availability records 324 outages totalling ~1,003 hours across four lab devices, which at a 30-second cadence and two attempts per cycle predicts ~240,700 failures — within 4% of what was recorded. Nothing to fix; retention had simply been hoarding an accurate count. It did expose real waste, though. Periodic polls no longer retry in-job, because the schedule already is the retry: a device that did not answer is unlikely to answer a second later, and each extra attempt occupies a worker for up to the job timeout. On a fleet with several devices down that doubled the capacity they consumed — the same capacity the rest of the fleet was short of in #114. It also doubled the recorded failure count, which is how ~1,000 hours of downtime came to look like 231k distinct faults. User-initiated full syncs keep their retries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
38d23cde02
commit
deb71363c9
@@ -159,12 +159,19 @@ export class PollerService {
|
||||
attempts: 3,
|
||||
backoff: { type: 'exponential', delay: 5000 },
|
||||
});
|
||||
// Periodic polls are not retried in-job: the schedule is the retry. A device
|
||||
// that just failed to answer is unlikely to answer a second later, and each
|
||||
// extra attempt occupies a worker for up to the job timeout. On a fleet with
|
||||
// several devices down that doubles the capacity they waste, which is exactly
|
||||
// the capacity the rest of the fleet is short of (#114). Historically this
|
||||
// also doubled the recorded failure count: 231k failed jobs on a four-device
|
||||
// fleet turned out to be ~1,000 hours of genuine downtime, counted twice.
|
||||
} else if (type === 'fast') {
|
||||
await this.fastQueue.add('device-fast-poll', jobData, { attempts: 2 });
|
||||
await this.fastQueue.add('device-fast-poll', jobData, { attempts: 1 });
|
||||
} else if (type === 'slow') {
|
||||
await this.slowQueue.add('device-slow-poll', jobData, { attempts: 2 });
|
||||
await this.slowQueue.add('device-slow-poll', jobData, { attempts: 1 });
|
||||
} else if (type === 'logs') {
|
||||
await this.logsQueue.add('device-logs-poll', jobData, { attempts: 2 });
|
||||
await this.logsQueue.add('device-logs-poll', jobData, { attempts: 1 });
|
||||
} else if (type === 'macscan') {
|
||||
await this.fastQueue.add('device-macscan', jobData, { attempts: 1 });
|
||||
} else if (type === 'spectral') {
|
||||
|
||||
Reference in New Issue
Block a user