2GT-RichandClaude Opus 5
98a72e1f7a
Closes #112. Source stays `docs/*.md`, rendered by MkDocs-Material and published to GitHub Pages by Actions. Same repository on purpose: documentation drifts, and co-location is the only thing that reliably prevents it — the pull request that changes behaviour changes the description, and review sees both. A separate repo would double the ceremony and guarantee divergence. The markdown remains readable when browsing the repository, so the site renders those files rather than replacing them. **Versioned from the start**, using mike. Images are published to GHCR and this project ships betas most days, so readers are routinely running something other than the newest build; a single "latest" would confidently describe a release they do not have. Versions follow the product rather than the docs, so the site and the running image carry the same number. **Content came first, and it had to.** The docs were eleven days and sixteen releases stale — SSH keys, bulk commands, the data-cap SMS, poller health and the timezone setting appeared in exactly zero files between them. Publishing that would have given documentation missing five of the most consequential features a polished, authoritative home, and a well-presented wrong answer is worse than an obviously rough one. Four pages now cover them: - cellular.md — signal grading, carrier aggregation, the uplink anchor, observed bands, and the data-cap SMS including why it fires early and how to calibrate the margin - ssh-keys.md — leading with what installing a key costs, because RouterOS disables password SSH for that user and the database becomes load-bearing - commands.md — why waves exist, and why reading output matters when RouterOS reports failure in text rather than an exit code - scaling.md — headroom, why it is insufficient on its own, and every tunable Pull requests build the site with `--strict`, which turns warnings into errors. That is the point of running it there at all, and it earned itself immediately: the first build failed on two links to the project README, which sits outside the docs tree. They are absolute now, so they resolve both in the rendered site and when reading the markdown on GitHub. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
101 lines
4.0 KiB
Markdown
101 lines
4.0 KiB
Markdown
# Polling and scaling
|
|
|
|
[← Documentation index](README.md)
|
|
|
|
How the poller works, how to tell whether it is keeping up, and what to change when it is
|
|
not.
|
|
|
|
## The model
|
|
|
|
Every device is polled on three cadences, dispatched through Redis-backed queues:
|
|
|
|
| Poll | Default | Collects |
|
|
|---|---|---|
|
|
| Fast | 30 s | Interface traffic, resources, clients, status |
|
|
| Logs | 60 s | Device log events |
|
|
| Slow | 5 min | Interfaces, VLANs, system info, wireless, STP |
|
|
|
|
Enqueue rate is not connect rate. Workers process jobs with limited concurrency, so what
|
|
matters is whether they drain faster than the schedule fills.
|
|
|
|
## Is it keeping up?
|
|
|
|
**Settings → Polling**, or `GET /api/system/poller`.
|
|
|
|
The number to read is **headroom** — service capacity divided by demand:
|
|
|
|
```
|
|
arrival devices / poll interval
|
|
service worker concurrency / mean poll duration
|
|
headroom service / arrival
|
|
```
|
|
|
|
Below **1.0** the backlog grows every cycle and devices go stale. Comfortably above it,
|
|
the fleet is being polled on schedule.
|
|
|
|
**Headroom alone is not enough**, which is why backlog and drain estimate sit beside it. It
|
|
describes flow — whether the workers can keep up *from here* — and is blind to work already
|
|
queued. A healthy headroom next to a large backlog means capacity is fine but you are still
|
|
working through a queue.
|
|
|
|
A single `status` summarises all of it: `ok`, `draining`, or `saturated`.
|
|
|
|
## Which devices are being missed
|
|
|
|
The same page lists devices not polled recently, and distinguishes two states that look
|
|
identical from the outside:
|
|
|
|
- **not attempted recently** — the poller never reached it
|
|
- **responded with…** — it was reached and did not answer
|
|
|
|
`device_poll_stats` records attempt and success separately per device and per kind, with
|
|
duration and last error. Their difference is that distinction.
|
|
|
|
## Tuning
|
|
|
|
All of these are environment variables and need a container restart.
|
|
|
|
| Variable | Default | Effect |
|
|
|---|---|---|
|
|
| `POLLER_CONCURRENCY` | 12 | Workers per queue. The first lever when headroom is low |
|
|
| `POLLER_INTERVAL_MS` | 30000 | Scheduler tick. Lengthening reduces demand |
|
|
| `POLLER_JOB_TIMEOUT_MS` | 45000 | Ceiling on one poll, so a dead device cannot hold a worker |
|
|
| `POLLER_JOB_RETENTION_SEC` | 3600 | How long finished jobs are kept |
|
|
| `POLLER_JOB_RETENTION_COUNT` | 5000 | How many finished jobs are kept |
|
|
| `REDIS_MAXMEMORY` | 512mb | Ceiling on Redis |
|
|
|
|
Raise concurrency first. Lengthen the interval second — it reduces freshness, which is the
|
|
thing you are paying for.
|
|
|
|
## Behaviour worth knowing
|
|
|
|
**A device already queued is not enqueued again.** Deduplication is by job id, so a
|
|
backlog cannot compound: one outstanding poll per device and kind, however long it waits.
|
|
|
|
**Periodic polls are not retried in-job.** The schedule *is* the retry — a device that did
|
|
not answer is unlikely to answer a second later, and a retry occupies a worker the rest of
|
|
the fleet needs.
|
|
|
|
**Stale queued polls are discarded**, at startup and every five minutes. A periodic poll
|
|
that has waited longer than its own cadence is worthless: running it produces a reading the
|
|
next cycle would take anyway. Discarding is safe precisely because the work is periodic —
|
|
the schedule re-enqueues within one interval.
|
|
|
|
**Job history is bounded.** Redis runs with `noeviction` rather than an LRU policy on
|
|
purpose: BullMQ's job locks and stalled-check keys carry TTLs, and evicting one causes
|
|
duplicate processing or a stalled-job storm. A memory ceiling protects the host; retention
|
|
keeps usage far below it.
|
|
|
|
## Clearing a backlog
|
|
|
|
**Settings → Polling → Clear queued**, or `POST /api/system/poller/drain`.
|
|
|
|
Nothing is lost. The scheduler re-enqueues whatever is still due on its next tick, so the
|
|
cost is at most one interval of freshness.
|
|
|
|
## Scheduled work and timezones
|
|
|
|
Scheduled tasks are evaluated in the timezone set under **Settings → General**, not the
|
|
container's. The container runs UTC, so without this a backup set for 02:00 would fire at
|
|
02:00 UTC wherever you are — and on the wrong day far enough east.
|