fix(poller): stop the queue outrunning the workers, and cap job retention (v0.23.17-beta)

Addresses #114, where a ~60 device fleet saw devices stop reporting entirely
while others crawled, new devices never started polling, and a manual Sync
worked exactly once. All of it is one failure with two compounding causes.

**The queue grew faster than it drained.** Worker concurrency was hardcoded to 3
with no per-job timeout, and the scheduler enqueued a fast poll per device every
30 seconds regardless of whether the previous one had finished. Measured against
a live fleet a fast poll took ~1s at the median but 13s at p90, so three workers
sustained about 0.87 jobs/sec against an arrival rate of 2.0/sec at sixty
devices. The difference — roughly 98,000 jobs a day — accumulated in an unbounded
FIFO, and devices at the back of it were simply never polled. That is why Sync
appeared to work once and then stop: the Sync route runs inline and bypasses the
queue entirely, so it was the only thing still reaching those devices.

Three changes: concurrency is configurable and defaults to 12; every job runs
under a timeout so one unreachable device cannot hold a worker while the fleet
starves; and a device with a poll already queued or running is not enqueued
again. That last one is the important one — it makes a backlog impossible rather
than merely less likely. The claim is a Redis key with a TTL, so a worker that
dies frees its device on the next cycle instead of wedging it forever.

**Finished jobs were never removed.** BullMQ keeps them unless told otherwise.
A four-device development instance had accumulated 1,412,935 completed and
230,707 failed jobs across the three queues — 2,570,474 Redis keys and 1.16 GB,
against a Redis with no maxmemory configured. Left alone it grows until the host
dies. Retention is now bounded by age and count, and a background pass on
startup trims history from before the limits existed.

The two faults fed each other, which is measurable: after trimming 2.16M jobs
the mean fast poll on the same fleet fell from 3,448ms to 602ms and the slowest
from 16,095ms to 1,205ms. Multi-million-element sorted sets were slowing every
queue operation, which reduced throughput, which grew the backlog further.

**The reporter's follow-up is the real lesson** and is addressed too: there was
no way to tell whether a device had been missed, was unresponsive, or was merely
slow, and no view of queue depth or per-poll cost — so polling frequency was
being chosen by guesswork. device_poll_stats now records attempt and success
separately per device and kind, with duration and last error; their difference
is precisely the missing distinction. GET /api/system/poller reports queue
depths, worker settings, measured poll cost, and a headroom figure that states
whether the configured cadence is achievable at the current fleet size. Below
1.0 the backlog grows every cycle.

Verified on the live instance: 2,570,474 keys reduced to 2,336, memory 1.16 GB
to 32.25 MB, queues drained, per-device stats recording, no stranded in-flight
claims, and headroom reported at 149x for four devices — the same arithmetic
that would have read 0.43 for sixty devices before this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2GT-Rich
2026-08-31 18:28:51 -07:00
co-authored by Claude Opus 5
parent 1332f7384d
commit 38d23cde02
8 changed files with 314 additions and 23 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mikrotik-manager-frontend",
"version": "0.23.16-beta",
"version": "0.23.17-beta",
"private": true,
"type": "module",
"scripts": {
+1 -1
View File
@@ -1 +1 @@
export const APP_VERSION = 'v0.23.16 Beta';
export const APP_VERSION = 'v0.23.17 Beta';