docs: versioned documentation site, and the four pages that were missing (v0.23.28-beta)

Closes #112.

Source stays `docs/*.md`, rendered by MkDocs-Material and published to GitHub
Pages by Actions. Same repository on purpose: documentation drifts, and
co-location is the only thing that reliably prevents it — the pull request that
changes behaviour changes the description, and review sees both. A separate repo
would double the ceremony and guarantee divergence.

The markdown remains readable when browsing the repository, so the site renders
those files rather than replacing them.

**Versioned from the start**, using mike. Images are published to GHCR and this
project ships betas most days, so readers are routinely running something other
than the newest build; a single "latest" would confidently describe a release
they do not have. Versions follow the product rather than the docs, so the site
and the running image carry the same number.

**Content came first, and it had to.** The docs were eleven days and sixteen
releases stale — SSH keys, bulk commands, the data-cap SMS, poller health and the
timezone setting appeared in exactly zero files between them. Publishing that
would have given documentation missing five of the most consequential features a
polished, authoritative home, and a well-presented wrong answer is worse than an
obviously rough one. Four pages now cover them:

  - cellular.md — signal grading, carrier aggregation, the uplink anchor,
    observed bands, and the data-cap SMS including why it fires early and how to
    calibrate the margin
  - ssh-keys.md — leading with what installing a key costs, because RouterOS
    disables password SSH for that user and the database becomes load-bearing
  - commands.md — why waves exist, and why reading output matters when RouterOS
    reports failure in text rather than an exit code
  - scaling.md — headroom, why it is insufficient on its own, and every tunable

Pull requests build the site with `--strict`, which turns warnings into errors.
That is the point of running it there at all, and it earned itself immediately:
the first build failed on two links to the project README, which sits outside the
docs tree. They are absolute now, so they resolve both in the rendered site and
when reading the markdown on GitHub.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2GT-Rich
2026-09-02 18:42:40 -07:00
co-authored by Claude Opus 5
parent 0e364b1558
commit 98a72e1f7a
14 changed files with 556 additions and 7 deletions
+100
View File
@@ -0,0 +1,100 @@
# Polling and scaling
[← Documentation index](README.md)
How the poller works, how to tell whether it is keeping up, and what to change when it is
not.
## The model
Every device is polled on three cadences, dispatched through Redis-backed queues:
| Poll | Default | Collects |
|---|---|---|
| Fast | 30 s | Interface traffic, resources, clients, status |
| Logs | 60 s | Device log events |
| Slow | 5 min | Interfaces, VLANs, system info, wireless, STP |
Enqueue rate is not connect rate. Workers process jobs with limited concurrency, so what
matters is whether they drain faster than the schedule fills.
## Is it keeping up?
**Settings → Polling**, or `GET /api/system/poller`.
The number to read is **headroom** — service capacity divided by demand:
```
arrival devices / poll interval
service worker concurrency / mean poll duration
headroom service / arrival
```
Below **1.0** the backlog grows every cycle and devices go stale. Comfortably above it,
the fleet is being polled on schedule.
**Headroom alone is not enough**, which is why backlog and drain estimate sit beside it. It
describes flow — whether the workers can keep up *from here* — and is blind to work already
queued. A healthy headroom next to a large backlog means capacity is fine but you are still
working through a queue.
A single `status` summarises all of it: `ok`, `draining`, or `saturated`.
## Which devices are being missed
The same page lists devices not polled recently, and distinguishes two states that look
identical from the outside:
- **not attempted recently** — the poller never reached it
- **responded with…** — it was reached and did not answer
`device_poll_stats` records attempt and success separately per device and per kind, with
duration and last error. Their difference is that distinction.
## Tuning
All of these are environment variables and need a container restart.
| Variable | Default | Effect |
|---|---|---|
| `POLLER_CONCURRENCY` | 12 | Workers per queue. The first lever when headroom is low |
| `POLLER_INTERVAL_MS` | 30000 | Scheduler tick. Lengthening reduces demand |
| `POLLER_JOB_TIMEOUT_MS` | 45000 | Ceiling on one poll, so a dead device cannot hold a worker |
| `POLLER_JOB_RETENTION_SEC` | 3600 | How long finished jobs are kept |
| `POLLER_JOB_RETENTION_COUNT` | 5000 | How many finished jobs are kept |
| `REDIS_MAXMEMORY` | 512mb | Ceiling on Redis |
Raise concurrency first. Lengthen the interval second — it reduces freshness, which is the
thing you are paying for.
## Behaviour worth knowing
**A device already queued is not enqueued again.** Deduplication is by job id, so a
backlog cannot compound: one outstanding poll per device and kind, however long it waits.
**Periodic polls are not retried in-job.** The schedule *is* the retry — a device that did
not answer is unlikely to answer a second later, and a retry occupies a worker the rest of
the fleet needs.
**Stale queued polls are discarded**, at startup and every five minutes. A periodic poll
that has waited longer than its own cadence is worthless: running it produces a reading the
next cycle would take anyway. Discarding is safe precisely because the work is periodic —
the schedule re-enqueues within one interval.
**Job history is bounded.** Redis runs with `noeviction` rather than an LRU policy on
purpose: BullMQ's job locks and stalled-check keys carry TTLs, and evicting one causes
duplicate processing or a stalled-job storm. A memory ceiling protects the host; retention
keeps usage far below it.
## Clearing a backlog
**Settings → Polling → Clear queued**, or `POST /api/system/poller/drain`.
Nothing is lost. The scheduler re-enqueues whatever is still due on its next tick, so the
cost is at most one interval of freshness.
## Scheduled work and timezones
Scheduled tasks are evaluated in the timezone set under **Settings → General**, not the
container's. The container runs UTC, so without this a backup set for 02:00 would fire at
02:00 UTC wherever you are — and on the wrong day far enough east.