2GT-RichandClaude Opus 5
38d23cde02
Addresses #114, where a ~60 device fleet saw devices stop reporting entirely while others crawled, new devices never started polling, and a manual Sync worked exactly once. All of it is one failure with two compounding causes. **The queue grew faster than it drained.** Worker concurrency was hardcoded to 3 with no per-job timeout, and the scheduler enqueued a fast poll per device every 30 seconds regardless of whether the previous one had finished. Measured against a live fleet a fast poll took ~1s at the median but 13s at p90, so three workers sustained about 0.87 jobs/sec against an arrival rate of 2.0/sec at sixty devices. The difference — roughly 98,000 jobs a day — accumulated in an unbounded FIFO, and devices at the back of it were simply never polled. That is why Sync appeared to work once and then stop: the Sync route runs inline and bypasses the queue entirely, so it was the only thing still reaching those devices. Three changes: concurrency is configurable and defaults to 12; every job runs under a timeout so one unreachable device cannot hold a worker while the fleet starves; and a device with a poll already queued or running is not enqueued again. That last one is the important one — it makes a backlog impossible rather than merely less likely. The claim is a Redis key with a TTL, so a worker that dies frees its device on the next cycle instead of wedging it forever. **Finished jobs were never removed.** BullMQ keeps them unless told otherwise. A four-device development instance had accumulated 1,412,935 completed and 230,707 failed jobs across the three queues — 2,570,474 Redis keys and 1.16 GB, against a Redis with no maxmemory configured. Left alone it grows until the host dies. Retention is now bounded by age and count, and a background pass on startup trims history from before the limits existed. The two faults fed each other, which is measurable: after trimming 2.16M jobs the mean fast poll on the same fleet fell from 3,448ms to 602ms and the slowest from 16,095ms to 1,205ms. Multi-million-element sorted sets were slowing every queue operation, which reduced throughput, which grew the backlog further. **The reporter's follow-up is the real lesson** and is addressed too: there was no way to tell whether a device had been missed, was unresponsive, or was merely slow, and no view of queue depth or per-poll cost — so polling frequency was being chosen by guesswork. device_poll_stats now records attempt and success separately per device and kind, with duration and last error; their difference is precisely the missing distinction. GET /api/system/poller reports queue depths, worker settings, measured poll cost, and a headroom figure that states whether the configured cadence is achievable at the current fleet size. Below 1.0 the backlog grows every cycle. Verified on the live instance: 2,570,474 keys reduced to 2,336, memory 1.16 GB to 32.25 MB, queues drained, per-device stats recording, no stranded in-flight claims, and headroom reported at 149x for four devices — the same arithmetic that would have read 0.43 for sixty devices before this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
50 lines
1.3 KiB
JSON
50 lines
1.3 KiB
JSON
{
|
|
"name": "mikrotik-manager-frontend",
|
|
"version": "0.23.17-beta",
|
|
"private": true,
|
|
"type": "module",
|
|
"scripts": {
|
|
"dev": "vite",
|
|
"build": "tsc && vite build",
|
|
"preview": "vite preview",
|
|
"lint": "eslint src",
|
|
"test": "vitest run"
|
|
},
|
|
"dependencies": {
|
|
"@tanstack/react-query": "^5.101.0",
|
|
"@types/leaflet": "^1.9.21",
|
|
"@xterm/addon-fit": "^0.11.0",
|
|
"@xterm/xterm": "^6.0.0",
|
|
"@xyflow/react": "^12.11.1",
|
|
"axios": "^1.18.1",
|
|
"clsx": "^2.1.0",
|
|
"date-fns": "^3.2.0",
|
|
"leaflet": "^1.9.4",
|
|
"lucide-react": "^1.23.0",
|
|
"react": "^19.2.7",
|
|
"react-dom": "^19.2.7",
|
|
"react-router-dom": "^7.18.1",
|
|
"recharts": "^2.10.3",
|
|
"socket.io-client": "^4.7.3",
|
|
"tailwind-merge": "^3.5.0",
|
|
"zustand": "^4.4.7"
|
|
},
|
|
"devDependencies": {
|
|
"@eslint/js": "^9.39.4",
|
|
"@types/react": "^19.2.14",
|
|
"@types/react-dom": "^19.2.3",
|
|
"@vitejs/plugin-react": "^6.0.3",
|
|
"autoprefixer": "^10.4.17",
|
|
"eslint": "^9.39.4",
|
|
"eslint-plugin-react": "^7.37.5",
|
|
"eslint-plugin-react-hooks": "^7.0.1",
|
|
"eslint-plugin-security": "^4.0.0",
|
|
"postcss": "^8.5.10",
|
|
"tailwindcss": "^3.4.1",
|
|
"typescript": "^5.9.3",
|
|
"typescript-eslint": "^8.58.2",
|
|
"vite": "^8.0.16",
|
|
"vitest": "^4.1.8"
|
|
}
|
|
}
|