2GT-RichandClaude Opus 5
38d23cde02
Addresses #114, where a ~60 device fleet saw devices stop reporting entirely while others crawled, new devices never started polling, and a manual Sync worked exactly once. All of it is one failure with two compounding causes. **The queue grew faster than it drained.** Worker concurrency was hardcoded to 3 with no per-job timeout, and the scheduler enqueued a fast poll per device every 30 seconds regardless of whether the previous one had finished. Measured against a live fleet a fast poll took ~1s at the median but 13s at p90, so three workers sustained about 0.87 jobs/sec against an arrival rate of 2.0/sec at sixty devices. The difference — roughly 98,000 jobs a day — accumulated in an unbounded FIFO, and devices at the back of it were simply never polled. That is why Sync appeared to work once and then stop: the Sync route runs inline and bypasses the queue entirely, so it was the only thing still reaching those devices. Three changes: concurrency is configurable and defaults to 12; every job runs under a timeout so one unreachable device cannot hold a worker while the fleet starves; and a device with a poll already queued or running is not enqueued again. That last one is the important one — it makes a backlog impossible rather than merely less likely. The claim is a Redis key with a TTL, so a worker that dies frees its device on the next cycle instead of wedging it forever. **Finished jobs were never removed.** BullMQ keeps them unless told otherwise. A four-device development instance had accumulated 1,412,935 completed and 230,707 failed jobs across the three queues — 2,570,474 Redis keys and 1.16 GB, against a Redis with no maxmemory configured. Left alone it grows until the host dies. Retention is now bounded by age and count, and a background pass on startup trims history from before the limits existed. The two faults fed each other, which is measurable: after trimming 2.16M jobs the mean fast poll on the same fleet fell from 3,448ms to 602ms and the slowest from 16,095ms to 1,205ms. Multi-million-element sorted sets were slowing every queue operation, which reduced throughput, which grew the backlog further. **The reporter's follow-up is the real lesson** and is addressed too: there was no way to tell whether a device had been missed, was unresponsive, or was merely slow, and no view of queue depth or per-poll cost — so polling frequency was being chosen by guesswork. device_poll_stats now records attempt and success separately per device and kind, with duration and last error; their difference is precisely the missing distinction. GET /api/system/poller reports queue depths, worker settings, measured poll cost, and a headroom figure that states whether the configured cadence is achievable at the current fleet size. Below 1.0 the backlog grows every cycle. Verified on the live instance: 2,570,474 keys reduced to 2,336, memory 1.16 GB to 32.25 MB, queues drained, per-device stats recording, no stranded in-flight claims, and headroom reported at 149x for four devices — the same arithmetic that would have read 0.43 for sixty devices before this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
63 lines
1.7 KiB
JSON
63 lines
1.7 KiB
JSON
{
|
|
"name": "mikrotik-manager-backend",
|
|
"version": "0.23.17-beta",
|
|
"description": "Mikrotik Network Management Platform - Backend API",
|
|
"main": "dist/index.js",
|
|
"scripts": {
|
|
"build": "tsc",
|
|
"start": "node dist/index.js",
|
|
"dev": "nodemon --watch src --ext ts --exec ts-node src/index.ts",
|
|
"lint": "eslint src",
|
|
"migrate": "ts-node src/db/migrate.ts",
|
|
"test": "jest",
|
|
"test:coverage": "jest --coverage"
|
|
},
|
|
"dependencies": {
|
|
"@influxdata/influxdb-client": "^1.33.2",
|
|
"@influxdata/influxdb-client-apis": "^1.35.0",
|
|
"bcryptjs": "^2.4.3",
|
|
"bullmq": "^5.1.1",
|
|
"compression": "^1.7.4",
|
|
"cors": "^2.8.5",
|
|
"dotenv": "^17.4.2",
|
|
"express": "^4.18.2",
|
|
"express-async-errors": "^3.1.1",
|
|
"helmet": "^8.1.0",
|
|
"ioredis": "^5.10.1",
|
|
"jsonwebtoken": "^9.0.2",
|
|
"morgan": "^1.10.0",
|
|
"nodemailer": "^9.0.3",
|
|
"openid-client": "^5.7.1",
|
|
"otpauth": "^9.5.1",
|
|
"pg": "^8.21.0",
|
|
"qrcode": "^1.5.4",
|
|
"socket.io": "^4.6.1",
|
|
"ssh2": "^1.14.0"
|
|
},
|
|
"devDependencies": {
|
|
"@eslint/js": "^10.0.1",
|
|
"@types/bcryptjs": "^2.4.6",
|
|
"@types/compression": "^1.7.5",
|
|
"@types/cors": "^2.8.17",
|
|
"@types/express": "^4.17.21",
|
|
"@types/jest": "^30.0.0",
|
|
"@types/jsonwebtoken": "^9.0.5",
|
|
"@types/morgan": "^1.9.9",
|
|
"@types/node": "^26.1.1",
|
|
"@types/nodemailer": "^6.4.14",
|
|
"@types/pg": "^8.10.9",
|
|
"@types/qrcode": "^1.5.6",
|
|
"@types/ssh2": "^1.11.17",
|
|
"@types/supertest": "^7.2.0",
|
|
"eslint": "^10.7.0",
|
|
"eslint-plugin-security": "^4.0.1",
|
|
"jest": "^30.4.2",
|
|
"nodemon": "^3.0.2",
|
|
"supertest": "^7.2.2",
|
|
"ts-jest": "^29.4.9",
|
|
"ts-node": "^10.9.2",
|
|
"typescript": "^6.0.3",
|
|
"typescript-eslint": "^8.64.0"
|
|
}
|
|
}
|