fix(poller): stop the queue outrunning the workers, and cap job retention (v0.23.17-beta)

Addresses #114, where a ~60 device fleet saw devices stop reporting entirely
while others crawled, new devices never started polling, and a manual Sync
worked exactly once. All of it is one failure with two compounding causes.

**The queue grew faster than it drained.** Worker concurrency was hardcoded to 3
with no per-job timeout, and the scheduler enqueued a fast poll per device every
30 seconds regardless of whether the previous one had finished. Measured against
a live fleet a fast poll took ~1s at the median but 13s at p90, so three workers
sustained about 0.87 jobs/sec against an arrival rate of 2.0/sec at sixty
devices. The difference — roughly 98,000 jobs a day — accumulated in an unbounded
FIFO, and devices at the back of it were simply never polled. That is why Sync
appeared to work once and then stop: the Sync route runs inline and bypasses the
queue entirely, so it was the only thing still reaching those devices.

Three changes: concurrency is configurable and defaults to 12; every job runs
under a timeout so one unreachable device cannot hold a worker while the fleet
starves; and a device with a poll already queued or running is not enqueued
again. That last one is the important one — it makes a backlog impossible rather
than merely less likely. The claim is a Redis key with a TTL, so a worker that
dies frees its device on the next cycle instead of wedging it forever.

**Finished jobs were never removed.** BullMQ keeps them unless told otherwise.
A four-device development instance had accumulated 1,412,935 completed and
230,707 failed jobs across the three queues — 2,570,474 Redis keys and 1.16 GB,
against a Redis with no maxmemory configured. Left alone it grows until the host
dies. Retention is now bounded by age and count, and a background pass on
startup trims history from before the limits existed.

The two faults fed each other, which is measurable: after trimming 2.16M jobs
the mean fast poll on the same fleet fell from 3,448ms to 602ms and the slowest
from 16,095ms to 1,205ms. Multi-million-element sorted sets were slowing every
queue operation, which reduced throughput, which grew the backlog further.

**The reporter's follow-up is the real lesson** and is addressed too: there was
no way to tell whether a device had been missed, was unresponsive, or was merely
slow, and no view of queue depth or per-poll cost — so polling frequency was
being chosen by guesswork. device_poll_stats now records attempt and success
separately per device and kind, with duration and last error; their difference
is precisely the missing distinction. GET /api/system/poller reports queue
depths, worker settings, measured poll cost, and a headroom figure that states
whether the configured cadence is achievable at the current fleet size. Below
1.0 the backlog grows every cycle.

Verified on the live instance: 2,570,474 keys reduced to 2,336, memory 1.16 GB
to 32.25 MB, queues drained, per-device stats recording, no stranded in-flight
claims, and headroom reported at 149x for four devices — the same arithmetic
that would have read 0.43 for sixty devices before this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/src/routes/system.ts

@@ -4,10 +4,17 @@ import { readFileSync } from 'fs';
 import { join } from 'path';
 import { requireAuth } from '../middleware/auth';
 import { query } from '../config/database';
+import type { PollerService } from '../services/PollerService';
 
 const router = Router();
 router.use(requireAuth);
 
+// Injected from index.ts, matching how the devices routes receive it.
+let pollerService: PollerService | null = null;
+export function setPollerService(p: PollerService): void {
+  pollerService = p;
+}
+
 // The container runs `node dist/index.js` (not via npm), so npm_package_version
 // is unset there — read the bundled package.json instead, falling back to it.
 // Without this, current stays "0.0.0" and every release looks like an update.
@@ -46,6 +53,41 @@ function isNewer(latest: number[], current: number[]): boolean {
 
 // GET /api/system/version-check
 // Returns { current, latest, update_available } — cached 24 h in app_settings.
+/**
+ * GET /api/system/poller — is the poller keeping up?
+ *
+ * A fleet that outruns its workers looks, from the outside, exactly like a fleet
+ * of misbehaving devices: data goes stale and nothing says why. `headroom` below
+ * 1.0 means the backlog grows every cycle and some devices will stop reporting
+ * altogether (#114).
+ */
+router.get('/poller', async (_req: Request, res: Response) => {
+  try {
+    if (!pollerService) return res.status(503).json({ error: 'Poller not started' });
+    const health = await pollerService.getPollerHealth();
+
+    // Devices the poller has not managed to reach lately, worst first. This is
+    // the list that answers "which ones is it actually missing?".
+    const stale = await query(
+      `SELECT d.id, d.name, d.status, s.kind,
+              s.last_attempt_at, s.last_success_at, s.last_duration_ms, s.last_error,
+              s.attempts, s.failures,
+              EXTRACT(EPOCH FROM (NOW() - s.last_success_at))::int AS seconds_since_success
+         FROM devices d
+         JOIN device_poll_stats s ON s.device_id = d.id AND s.kind = 'fast'
+        WHERE d.status != 'disabled'
+          AND (s.last_success_at IS NULL OR s.last_success_at < NOW() - INTERVAL '3 minutes')
+        ORDER BY s.last_success_at ASC NULLS FIRST
+        LIMIT 50`
+    );
+
+    res.json({ ...health, stale_devices: stale });
+  } catch (error) {
+    console.error('Error building poller health:', error);
+    res.status(500).json({ error: 'Failed to read poller health' });
+  }
+});
+
 router.get('/version-check', async (_req: Request, res: Response) => {
   try {
     // Check cache