2GT-RichandClaude Opus 5
98a72e1f7a
Closes #112. Source stays `docs/*.md`, rendered by MkDocs-Material and published to GitHub Pages by Actions. Same repository on purpose: documentation drifts, and co-location is the only thing that reliably prevents it — the pull request that changes behaviour changes the description, and review sees both. A separate repo would double the ceremony and guarantee divergence. The markdown remains readable when browsing the repository, so the site renders those files rather than replacing them. **Versioned from the start**, using mike. Images are published to GHCR and this project ships betas most days, so readers are routinely running something other than the newest build; a single "latest" would confidently describe a release they do not have. Versions follow the product rather than the docs, so the site and the running image carry the same number. **Content came first, and it had to.** The docs were eleven days and sixteen releases stale — SSH keys, bulk commands, the data-cap SMS, poller health and the timezone setting appeared in exactly zero files between them. Publishing that would have given documentation missing five of the most consequential features a polished, authoritative home, and a well-presented wrong answer is worse than an obviously rough one. Four pages now cover them: - cellular.md — signal grading, carrier aggregation, the uplink anchor, observed bands, and the data-cap SMS including why it fires early and how to calibrate the margin - ssh-keys.md — leading with what installing a key costs, because RouterOS disables password SSH for that user and the database becomes load-bearing - commands.md — why waves exist, and why reading output matters when RouterOS reports failure in text rather than an exit code - scaling.md — headroom, why it is insufficient on its own, and every tunable Pull requests build the site with `--strict`, which turns warnings into errors. That is the point of running it there at all, and it earned itself immediately: the first build failed on two links to the project README, which sits outside the docs tree. They are absolute now, so they resolve both in the rendered site and when reading the markdown on GitHub. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4.0 KiB
Polling and scaling
How the poller works, how to tell whether it is keeping up, and what to change when it is not.
The model
Every device is polled on three cadences, dispatched through Redis-backed queues:
| Poll | Default | Collects |
|---|---|---|
| Fast | 30 s | Interface traffic, resources, clients, status |
| Logs | 60 s | Device log events |
| Slow | 5 min | Interfaces, VLANs, system info, wireless, STP |
Enqueue rate is not connect rate. Workers process jobs with limited concurrency, so what matters is whether they drain faster than the schedule fills.
Is it keeping up?
Settings → Polling, or GET /api/system/poller.
The number to read is headroom — service capacity divided by demand:
arrival devices / poll interval
service worker concurrency / mean poll duration
headroom service / arrival
Below 1.0 the backlog grows every cycle and devices go stale. Comfortably above it, the fleet is being polled on schedule.
Headroom alone is not enough, which is why backlog and drain estimate sit beside it. It describes flow — whether the workers can keep up from here — and is blind to work already queued. A healthy headroom next to a large backlog means capacity is fine but you are still working through a queue.
A single status summarises all of it: ok, draining, or saturated.
Which devices are being missed
The same page lists devices not polled recently, and distinguishes two states that look identical from the outside:
- not attempted recently — the poller never reached it
- responded with… — it was reached and did not answer
device_poll_stats records attempt and success separately per device and per kind, with
duration and last error. Their difference is that distinction.
Tuning
All of these are environment variables and need a container restart.
| Variable | Default | Effect |
|---|---|---|
POLLER_CONCURRENCY |
12 | Workers per queue. The first lever when headroom is low |
POLLER_INTERVAL_MS |
30000 | Scheduler tick. Lengthening reduces demand |
POLLER_JOB_TIMEOUT_MS |
45000 | Ceiling on one poll, so a dead device cannot hold a worker |
POLLER_JOB_RETENTION_SEC |
3600 | How long finished jobs are kept |
POLLER_JOB_RETENTION_COUNT |
5000 | How many finished jobs are kept |
REDIS_MAXMEMORY |
512mb | Ceiling on Redis |
Raise concurrency first. Lengthen the interval second — it reduces freshness, which is the thing you are paying for.
Behaviour worth knowing
A device already queued is not enqueued again. Deduplication is by job id, so a backlog cannot compound: one outstanding poll per device and kind, however long it waits.
Periodic polls are not retried in-job. The schedule is the retry — a device that did not answer is unlikely to answer a second later, and a retry occupies a worker the rest of the fleet needs.
Stale queued polls are discarded, at startup and every five minutes. A periodic poll that has waited longer than its own cadence is worthless: running it produces a reading the next cycle would take anyway. Discarding is safe precisely because the work is periodic — the schedule re-enqueues within one interval.
Job history is bounded. Redis runs with noeviction rather than an LRU policy on
purpose: BullMQ's job locks and stalled-check keys carry TTLs, and evicting one causes
duplicate processing or a stalled-job storm. A memory ceiling protects the host; retention
keeps usage far below it.
Clearing a backlog
Settings → Polling → Clear queued, or POST /api/system/poller/drain.
Nothing is lost. The scheduler re-enqueues whatever is still due on its next tick, so the cost is at most one interval of freshness.
Scheduled work and timezones
Scheduled tasks are evaluated in the timezone set under Settings → General, not the container's. The container runs UTC, so without this a backup set for 02:00 would fire at 02:00 UTC wherever you are — and on the wrong day far enough east.