diff --git a/README.md b/README.md index 628e62b..93ebf49 100644 --- a/README.md +++ b/README.md @@ -370,6 +370,8 @@ so it moves when behaviour does. | [Configuration](docs/configuration.md) | Environment variables, secret management, key rotation, TLS | | [Change Guard and Config Health](docs/change-guard.md) | How the safety system works, its settings, and what to expect when it fires | | [Sites](docs/sites.md) | Grouping devices into sites, switching between them, and what each site scopes | +| [Firmware rollouts](docs/firmware.md) | Waves, the upgrade pipeline, and what upgrading several at once costs | +| [Backups](docs/backups.md) | Finding, reading, comparing and deleting configuration backups | | [Alerting](docs/alerting.md) | Rules, channels, ntfy priority mapping, webhooks, maintenance windows | | [Single sign-on (OIDC)](docs/sso-oidc.md) | Identity provider setup, group-to-role mapping, break-glass behaviour | | [CAPsMAN](docs/capsman.md) | How centrally provisioned APs are modelled, and current limits | diff --git a/docs/README.md b/docs/README.md index 1ebd251..5e44aaf 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,6 +14,8 @@ whether to try it, the [project README](https://github.com/2GT-Media-Group-LLC/m | [Cellular (LTE)](cellular.md) | Signal, carriers, tower movement, and the data-cap reset SMS | | [SSH keys](ssh-keys.md) | Per-device keypairs, rotation, and what installing one costs | | [Bulk commands](commands.md) | Running one command across a fleet, in waves | +| [Firmware rollouts](firmware.md) | Waves, the upgrade pipeline, and what upgrading several at once costs | +| [Backups](backups.md) | Finding, reading, comparing and deleting configuration backups | | [Polling and scaling](scaling.md) | Poller health, headroom, tuning, and clearing a backlog | | [API and automation](api.md) | Scoped tokens, authentication, and driving the platform from scripts | | [Architecture](architecture.md) | How the system is put together, for contributors | diff --git a/docs/backups.md b/docs/backups.md new file mode 100644 index 0000000..7eab2dd --- /dev/null +++ b/docs/backups.md @@ -0,0 +1,85 @@ +# Backups + +[← Documentation index](README.md) + +Configuration backups for every device, on demand or on a schedule, with the tools to +find, read and compare them. + +## What a backup actually is + +`/export compact` run over SSH, stored as a `.rsc` file — **plain RouterOS script, not +the binary `.backup` blob**. + +That distinction shapes everything else on this page. A `.rsc` can be read, diffed, +edited and applied selectively; a `.backup` can only be restored wholesale onto the +same device. It also means a backup is human-readable text, which is why previewing +and comparing them is possible at all. + +It also means a backup does **not** capture things `/export` omits: user passwords, +certificates and files on disk. A `.rsc` restores a configuration, not a device. + +## Finding one + +The table filters by **device**, **type**, a **date range** and free text across +device name, filename and notes. Date ranges include both ends — "to: 3 September" +covers everything that happened on the 3rd. + +Types: + +| Type | Created by | +|---|---| +| `manual` | The Create Backup button | +| `scheduled` | The daily/weekly/monthly schedule | +| `pre-upgrade` | A firmware rollout, before it touches the device | +| `config-snapshot` | Config History, when it captures a change | + +## Reading and comparing + +Click any row to read the backup in place. Select exactly two and choose **Compare** +for a line diff — the same diff Config History uses for snapshots. + +The comparison is always ordered oldest to newest regardless of the order you clicked, +so additions read as additions. + +The **filename** is not a column in the table; it appears in the preview and diff +headers instead. It is a device name and a timestamp — useful when you are looking at +a specific file or quoting it to someone, less useful as a column competing for width +with the device and date you actually filter on. + +## Deleting, and the one thing to know first + +A backup and its Config History snapshot are **one artifact**. They are joined in the +database with `ON DELETE CASCADE` precisely so they cannot drift apart — a snapshot +whose restorable `.rsc` had been deleted would offer a rollback it could not perform. + +So deleting a `config-snapshot` backup also removes that snapshot from the device's +Config History, and the ability to roll back to it. + +Single deletes of a snapshot-linked backup ask for confirmation and say this plainly. +**Bulk delete** does the same, but with the numbers: it reports how many backups per +device will go, and how many Config History snapshots will go with them, before you +confirm. There is no undo. + +Bulk delete is capped at 200 per action and is scoped to the [current site](sites.md), +so it cannot reach into another site's history. + +## Restoring + +Restore uploads the `.rsc` and applies it. This is a real configuration change with +all the risk that implies — read the backup first if you are not certain what is in +it, which is what the preview is for. + +## API + +| Method | Path | Purpose | +|---|---|---| +| `GET` | `/api/backups` | List, with `deviceId`, `type`, `from`, `to`, `search` | +| `GET` | `/api/backups/types` | Types present, with counts | +| `POST` | `/api/backups` | Create one for a device | +| `GET` | `/api/backups/:id/content` | Read the text (capped at 2 MB) | +| `GET` | `/api/backups/:from/diff/:to` | Both texts, for a client-side diff | +| `GET` | `/api/backups/:id/download` | Download the `.rsc` | +| `POST` | `/api/backups/:id/restore` | Apply it to the device | +| `DELETE` | `/api/backups/:id` | Delete one | +| `GET` | `/api/backups/bulk-delete/preview` | What a bulk delete would remove | +| `POST` | `/api/backups/bulk-delete` | Delete many | diff --git a/docs/firmware.md b/docs/firmware.md new file mode 100644 index 0000000..0f54fd8 --- /dev/null +++ b/docs/firmware.md @@ -0,0 +1,103 @@ +# Firmware rollouts + +[← Documentation index](README.md) + +Upgrade RouterOS across a fleet in waves, with a verified pipeline per device and +a stopping point before a bad build reaches everything. + +## The pipeline + +Each device goes through the same sequence: + +1. **Pre-upgrade backup** (optional, on by default) +2. **Check for updates** — a device already on the target version is *skipped*, not failed +3. **Download the image, and confirm it landed** +4. **Reboot** +5. **Prove the reboot happened** — uptime must have gone *backwards* +6. **Verify the version moved** +7. **RouterBOOT** (optional) — a second flash and a second reboot + +Steps 3 and 5 exist because of a specific failure. `/system/package/update/install` +bundles download and reboot into one call whose progress cannot be observed, and a +CCR2216 was seen to accept it, do neither, and report nothing. That surfaced as +"rebooted but still reports 7.24.1", because nothing could tell a device that never +restarted from one that restarted unchanged. + +Downloading first makes success verifiable *before* anything reboots, so a device is +never restarted for an image it does not have. + +!!! note "Downloads are slow, and that is normal" + `/system/package/update/download` blocks until the image has landed — a minute or + more on an ordinary link. It is given a ten-minute budget. If the command does not + return cleanly the device is asked again on a fresh connection what actually + happened, rather than being declared failed: a slow download is not a broken one. + +## Waves + +Waves are the point. Devices are assigned to wave 1, 2 or 3, and **waves run strictly +in sequence**. Wave 1 is your canary: put one device in it, let it finish, and the +rest of the fleet is still untouched if it goes wrong. + +**Halt on failure** stops the rollout rather than continuing into the next wave. +Devices never reached are marked `skipped`, not `failed` — they did not run. + +A device that comes back on the *old* version counts as a failure. + +## Upgrading several devices at once + +By default the devices inside a wave upgrade **one at a time**. That is safe and slow: +a CRS switch can take five minutes to reboot, so ten of them is most of an hour spent +waiting. + +The **At once** control raises how many devices in the same wave may upgrade +simultaneously. Waves remain sequential either way, so the canary still gates +everything after it. + +There are two costs, and both are worth understanding before you raise it. + +**Halting becomes less precise.** With one device at a time, a failure stops the +rollout immediately. Running several at once, the wave finishes first — so a wave of +five can put a bad build on five devices before anything stops. At the default of 1 +these are the same moment. + +**Devices reboot together.** That is fine for independent routers. It is not fine for +a switch that the others are reached *through*: rebooting it alongside them cuts the +path to devices that are mid-upgrade. Change Guard does not cover this — it protects +configuration, and a device that is unreachable because its uplink rebooted has no +configuration change to revert. + +The rollout builder warns when it can see that one selected device carries another's +traffic. That warning is deliberately conservative: it fires only when a link is the +device's spanning-tree root port *and* that port resolves to exactly one neighbour. +Discovered topology is often ambiguous — a trunk port can resolve to several +neighbours at once — and naming the wrong switch would be worse than saying nothing. +**Absence of a warning is not proof that it is safe to reboot everything together.** + +A reasonable pattern for a large fleet on a well-known patch release: + +| Wave | Devices | At once | +|---|---|---| +| 1 | one representative device | 1 | +| 2 | the access layer | 5–10 | +| 3 | anything upstream of wave 2 | 1 | + +## Scheduling and cancelling + +A rollout can be scheduled; pair it with a maintenance window by scheduling inside +one. One rollout runs at a time. + +Devices already belonging to an unfinished rollout are refused by name when you try +to create another for them, before anything is written. + +**Cancelling never interrupts an in-flight flash.** It stops before the *next* device, +because interrupting a device mid-write is how you brick it. The UI says so when you +press the button — the currently upgrading device will finish first. + +## When it fails + +Failure messages name the mechanism rather than the symptom. "Device did not finish +downloading within ten minutes. Nothing was rebooted. The device reports 91.6 MB free" +tells you where to look; "upgrade failed" does not. + +If a rollout reports a device stuck at the old version, check the device's own +`/system/package/update` state before re-running — it may have the image already. diff --git a/mkdocs.yml b/mkdocs.yml index 7f71417..54939d5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -65,6 +65,8 @@ nav: - Alerting: alerting.md - SSH keys: ssh-keys.md - Bulk commands: commands.md + - Firmware rollouts: firmware.md + - Backups: backups.md - Devices: - Cellular (LTE): cellular.md - CAPsMAN: capsman.md