docs: firmware rollouts and backups
Neither area had a page, and both changed materially in v0.24.2/v0.24.3. The firmware page records why the pipeline downloads and verifies before rebooting, and what raising "at once" actually costs: halting becomes less precise, devices reboot together, and the upstream warning is conservative enough that its absence is not proof of safety. The backups page leads with the fact that a backup is /export text rather than a binary blob, since that is what makes preview and diff possible and what limits what a restore can do. It also states plainly that a backup and its Config History snapshot are one artifact, so deleting one removes the other's rollback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2GT-Rich
co-authored by
Claude Opus 5
Claude Opus 5
parent
df8f6dca44
commit
6c3d2aaa35
@@ -370,6 +370,8 @@ so it moves when behaviour does.
|
||||
| [Configuration](docs/configuration.md) | Environment variables, secret management, key rotation, TLS |
|
||||
| [Change Guard and Config Health](docs/change-guard.md) | How the safety system works, its settings, and what to expect when it fires |
|
||||
| [Sites](docs/sites.md) | Grouping devices into sites, switching between them, and what each site scopes |
|
||||
| [Firmware rollouts](docs/firmware.md) | Waves, the upgrade pipeline, and what upgrading several at once costs |
|
||||
| [Backups](docs/backups.md) | Finding, reading, comparing and deleting configuration backups |
|
||||
| [Alerting](docs/alerting.md) | Rules, channels, ntfy priority mapping, webhooks, maintenance windows |
|
||||
| [Single sign-on (OIDC)](docs/sso-oidc.md) | Identity provider setup, group-to-role mapping, break-glass behaviour |
|
||||
| [CAPsMAN](docs/capsman.md) | How centrally provisioned APs are modelled, and current limits |
|
||||
|
||||
@@ -14,6 +14,8 @@ whether to try it, the [project README](https://github.com/2GT-Media-Group-LLC/m
|
||||
| [Cellular (LTE)](cellular.md) | Signal, carriers, tower movement, and the data-cap reset SMS |
|
||||
| [SSH keys](ssh-keys.md) | Per-device keypairs, rotation, and what installing one costs |
|
||||
| [Bulk commands](commands.md) | Running one command across a fleet, in waves |
|
||||
| [Firmware rollouts](firmware.md) | Waves, the upgrade pipeline, and what upgrading several at once costs |
|
||||
| [Backups](backups.md) | Finding, reading, comparing and deleting configuration backups |
|
||||
| [Polling and scaling](scaling.md) | Poller health, headroom, tuning, and clearing a backlog |
|
||||
| [API and automation](api.md) | Scoped tokens, authentication, and driving the platform from scripts |
|
||||
| [Architecture](architecture.md) | How the system is put together, for contributors |
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# Backups
|
||||
|
||||
[← Documentation index](README.md)
|
||||
|
||||
Configuration backups for every device, on demand or on a schedule, with the tools to
|
||||
find, read and compare them.
|
||||
|
||||
## What a backup actually is
|
||||
|
||||
`/export compact` run over SSH, stored as a `.rsc` file — **plain RouterOS script, not
|
||||
the binary `.backup` blob**.
|
||||
|
||||
That distinction shapes everything else on this page. A `.rsc` can be read, diffed,
|
||||
edited and applied selectively; a `.backup` can only be restored wholesale onto the
|
||||
same device. It also means a backup is human-readable text, which is why previewing
|
||||
and comparing them is possible at all.
|
||||
|
||||
It also means a backup does **not** capture things `/export` omits: user passwords,
|
||||
certificates and files on disk. A `.rsc` restores a configuration, not a device.
|
||||
|
||||
## Finding one
|
||||
|
||||
The table filters by **device**, **type**, a **date range** and free text across
|
||||
device name, filename and notes. Date ranges include both ends — "to: 3 September"
|
||||
covers everything that happened on the 3rd.
|
||||
|
||||
Types:
|
||||
|
||||
| Type | Created by |
|
||||
|---|---|
|
||||
| `manual` | The Create Backup button |
|
||||
| `scheduled` | The daily/weekly/monthly schedule |
|
||||
| `pre-upgrade` | A firmware rollout, before it touches the device |
|
||||
| `config-snapshot` | Config History, when it captures a change |
|
||||
|
||||
## Reading and comparing
|
||||
|
||||
Click any row to read the backup in place. Select exactly two and choose **Compare**
|
||||
for a line diff — the same diff Config History uses for snapshots.
|
||||
|
||||
The comparison is always ordered oldest to newest regardless of the order you clicked,
|
||||
so additions read as additions.
|
||||
|
||||
The **filename** is not a column in the table; it appears in the preview and diff
|
||||
headers instead. It is a device name and a timestamp — useful when you are looking at
|
||||
a specific file or quoting it to someone, less useful as a column competing for width
|
||||
with the device and date you actually filter on.
|
||||
|
||||
## Deleting, and the one thing to know first
|
||||
|
||||
A backup and its Config History snapshot are **one artifact**. They are joined in the
|
||||
database with `ON DELETE CASCADE` precisely so they cannot drift apart — a snapshot
|
||||
whose restorable `.rsc` had been deleted would offer a rollback it could not perform.
|
||||
|
||||
So deleting a `config-snapshot` backup also removes that snapshot from the device's
|
||||
Config History, and the ability to roll back to it.
|
||||
|
||||
Single deletes of a snapshot-linked backup ask for confirmation and say this plainly.
|
||||
**Bulk delete** does the same, but with the numbers: it reports how many backups per
|
||||
device will go, and how many Config History snapshots will go with them, before you
|
||||
confirm. There is no undo.
|
||||
|
||||
Bulk delete is capped at 200 per action and is scoped to the [current site](sites.md),
|
||||
so it cannot reach into another site's history.
|
||||
|
||||
## Restoring
|
||||
|
||||
Restore uploads the `.rsc` and applies it. This is a real configuration change with
|
||||
all the risk that implies — read the backup first if you are not certain what is in
|
||||
it, which is what the preview is for.
|
||||
|
||||
## API
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/backups` | List, with `deviceId`, `type`, `from`, `to`, `search` |
|
||||
| `GET` | `/api/backups/types` | Types present, with counts |
|
||||
| `POST` | `/api/backups` | Create one for a device |
|
||||
| `GET` | `/api/backups/:id/content` | Read the text (capped at 2 MB) |
|
||||
| `GET` | `/api/backups/:from/diff/:to` | Both texts, for a client-side diff |
|
||||
| `GET` | `/api/backups/:id/download` | Download the `.rsc` |
|
||||
| `POST` | `/api/backups/:id/restore` | Apply it to the device |
|
||||
| `DELETE` | `/api/backups/:id` | Delete one |
|
||||
| `GET` | `/api/backups/bulk-delete/preview` | What a bulk delete would remove |
|
||||
| `POST` | `/api/backups/bulk-delete` | Delete many |
|
||||
@@ -0,0 +1,103 @@
|
||||
# Firmware rollouts
|
||||
|
||||
[← Documentation index](README.md)
|
||||
|
||||
Upgrade RouterOS across a fleet in waves, with a verified pipeline per device and
|
||||
a stopping point before a bad build reaches everything.
|
||||
|
||||
## The pipeline
|
||||
|
||||
Each device goes through the same sequence:
|
||||
|
||||
1. **Pre-upgrade backup** (optional, on by default)
|
||||
2. **Check for updates** — a device already on the target version is *skipped*, not failed
|
||||
3. **Download the image, and confirm it landed**
|
||||
4. **Reboot**
|
||||
5. **Prove the reboot happened** — uptime must have gone *backwards*
|
||||
6. **Verify the version moved**
|
||||
7. **RouterBOOT** (optional) — a second flash and a second reboot
|
||||
|
||||
Steps 3 and 5 exist because of a specific failure. `/system/package/update/install`
|
||||
bundles download and reboot into one call whose progress cannot be observed, and a
|
||||
CCR2216 was seen to accept it, do neither, and report nothing. That surfaced as
|
||||
"rebooted but still reports 7.24.1", because nothing could tell a device that never
|
||||
restarted from one that restarted unchanged.
|
||||
|
||||
Downloading first makes success verifiable *before* anything reboots, so a device is
|
||||
never restarted for an image it does not have.
|
||||
|
||||
!!! note "Downloads are slow, and that is normal"
|
||||
`/system/package/update/download` blocks until the image has landed — a minute or
|
||||
more on an ordinary link. It is given a ten-minute budget. If the command does not
|
||||
return cleanly the device is asked again on a fresh connection what actually
|
||||
happened, rather than being declared failed: a slow download is not a broken one.
|
||||
|
||||
## Waves
|
||||
|
||||
Waves are the point. Devices are assigned to wave 1, 2 or 3, and **waves run strictly
|
||||
in sequence**. Wave 1 is your canary: put one device in it, let it finish, and the
|
||||
rest of the fleet is still untouched if it goes wrong.
|
||||
|
||||
**Halt on failure** stops the rollout rather than continuing into the next wave.
|
||||
Devices never reached are marked `skipped`, not `failed` — they did not run.
|
||||
|
||||
A device that comes back on the *old* version counts as a failure.
|
||||
|
||||
## Upgrading several devices at once
|
||||
|
||||
By default the devices inside a wave upgrade **one at a time**. That is safe and slow:
|
||||
a CRS switch can take five minutes to reboot, so ten of them is most of an hour spent
|
||||
waiting.
|
||||
|
||||
The **At once** control raises how many devices in the same wave may upgrade
|
||||
simultaneously. Waves remain sequential either way, so the canary still gates
|
||||
everything after it.
|
||||
|
||||
There are two costs, and both are worth understanding before you raise it.
|
||||
|
||||
**Halting becomes less precise.** With one device at a time, a failure stops the
|
||||
rollout immediately. Running several at once, the wave finishes first — so a wave of
|
||||
five can put a bad build on five devices before anything stops. At the default of 1
|
||||
these are the same moment.
|
||||
|
||||
**Devices reboot together.** That is fine for independent routers. It is not fine for
|
||||
a switch that the others are reached *through*: rebooting it alongside them cuts the
|
||||
path to devices that are mid-upgrade. Change Guard does not cover this — it protects
|
||||
configuration, and a device that is unreachable because its uplink rebooted has no
|
||||
configuration change to revert.
|
||||
|
||||
The rollout builder warns when it can see that one selected device carries another's
|
||||
traffic. That warning is deliberately conservative: it fires only when a link is the
|
||||
device's spanning-tree root port *and* that port resolves to exactly one neighbour.
|
||||
Discovered topology is often ambiguous — a trunk port can resolve to several
|
||||
neighbours at once — and naming the wrong switch would be worse than saying nothing.
|
||||
**Absence of a warning is not proof that it is safe to reboot everything together.**
|
||||
|
||||
A reasonable pattern for a large fleet on a well-known patch release:
|
||||
|
||||
| Wave | Devices | At once |
|
||||
|---|---|---|
|
||||
| 1 | one representative device | 1 |
|
||||
| 2 | the access layer | 5–10 |
|
||||
| 3 | anything upstream of wave 2 | 1 |
|
||||
|
||||
## Scheduling and cancelling
|
||||
|
||||
A rollout can be scheduled; pair it with a maintenance window by scheduling inside
|
||||
one. One rollout runs at a time.
|
||||
|
||||
Devices already belonging to an unfinished rollout are refused by name when you try
|
||||
to create another for them, before anything is written.
|
||||
|
||||
**Cancelling never interrupts an in-flight flash.** It stops before the *next* device,
|
||||
because interrupting a device mid-write is how you brick it. The UI says so when you
|
||||
press the button — the currently upgrading device will finish first.
|
||||
|
||||
## When it fails
|
||||
|
||||
Failure messages name the mechanism rather than the symptom. "Device did not finish
|
||||
downloading within ten minutes. Nothing was rebooted. The device reports 91.6 MB free"
|
||||
tells you where to look; "upgrade failed" does not.
|
||||
|
||||
If a rollout reports a device stuck at the old version, check the device's own
|
||||
`/system/package/update` state before re-running — it may have the image already.
|
||||
@@ -65,6 +65,8 @@ nav:
|
||||
- Alerting: alerting.md
|
||||
- SSH keys: ssh-keys.md
|
||||
- Bulk commands: commands.md
|
||||
- Firmware rollouts: firmware.md
|
||||
- Backups: backups.md
|
||||
- Devices:
|
||||
- Cellular (LTE): cellular.md
|
||||
- CAPsMAN: capsman.md
|
||||
|
||||
Reference in New Issue
Block a user