fix(poller): stop the queue outrunning the workers, and cap job retention (v0.23.17-beta)
Addresses #114, where a ~60 device fleet saw devices stop reporting entirely while others crawled, new devices never started polling, and a manual Sync worked exactly once. All of it is one failure with two compounding causes. **The queue grew faster than it drained.** Worker concurrency was hardcoded to 3 with no per-job timeout, and the scheduler enqueued a fast poll per device every 30 seconds regardless of whether the previous one had finished. Measured against a live fleet a fast poll took ~1s at the median but 13s at p90, so three workers sustained about 0.87 jobs/sec against an arrival rate of 2.0/sec at sixty devices. The difference — roughly 98,000 jobs a day — accumulated in an unbounded FIFO, and devices at the back of it were simply never polled. That is why Sync appeared to work once and then stop: the Sync route runs inline and bypasses the queue entirely, so it was the only thing still reaching those devices. Three changes: concurrency is configurable and defaults to 12; every job runs under a timeout so one unreachable device cannot hold a worker while the fleet starves; and a device with a poll already queued or running is not enqueued again. That last one is the important one — it makes a backlog impossible rather than merely less likely. The claim is a Redis key with a TTL, so a worker that dies frees its device on the next cycle instead of wedging it forever. **Finished jobs were never removed.** BullMQ keeps them unless told otherwise. A four-device development instance had accumulated 1,412,935 completed and 230,707 failed jobs across the three queues — 2,570,474 Redis keys and 1.16 GB, against a Redis with no maxmemory configured. Left alone it grows until the host dies. Retention is now bounded by age and count, and a background pass on startup trims history from before the limits existed. The two faults fed each other, which is measurable: after trimming 2.16M jobs the mean fast poll on the same fleet fell from 3,448ms to 602ms and the slowest from 16,095ms to 1,205ms. Multi-million-element sorted sets were slowing every queue operation, which reduced throughput, which grew the backlog further. **The reporter's follow-up is the real lesson** and is addressed too: there was no way to tell whether a device had been missed, was unresponsive, or was merely slow, and no view of queue depth or per-poll cost — so polling frequency was being chosen by guesswork. device_poll_stats now records attempt and success separately per device and kind, with duration and last error; their difference is precisely the missing distinction. GET /api/system/poller reports queue depths, worker settings, measured poll cost, and a headroom figure that states whether the configured cadence is achievable at the current fleet size. Below 1.0 the backlog grows every cycle. Verified on the live instance: 2,570,474 keys reduced to 2,336, memory 1.16 GB to 32.25 MB, queues drained, per-device stats recording, no stranded in-flight claims, and headroom reported at 149x for four devices — the same arithmetic that would have read 0.43 for sixty devices before this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1332f7384d
commit
38d23cde02
@@ -6,7 +6,7 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<img src="https://img.shields.io/badge/version-0.23.16_Beta-blue" alt="Version" />
|
||||
<img src="https://img.shields.io/badge/version-0.23.17_Beta-blue" alt="Version" />
|
||||
<img src="https://img.shields.io/badge/license-AGPLv3-blue" alt="License" />
|
||||
<img src="https://img.shields.io/badge/docker-compose-2496ED?logo=docker&logoColor=white" alt="Docker" />
|
||||
<img src="https://img.shields.io/badge/TypeScript-5.3-3178C6?logo=typescript&logoColor=white" alt="TypeScript" />
|
||||
|
||||
Reference in New Issue
Block a user