2GT-RichandClaude Opus 5
6c3d2aaa35
Neither area had a page, and both changed materially in v0.24.2/v0.24.3. The firmware page records why the pipeline downloads and verifies before rebooting, and what raising "at once" actually costs: halting becomes less precise, devices reboot together, and the upstream warning is conservative enough that its absence is not proof of safety. The backups page leads with the fact that a backup is /export text rather than a binary blob, since that is what makes preview and diff possible and what limits what a restore can do. It also states plainly that a backup and its Config History snapshot are one artifact, so deleting one removes the other's rollback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4.7 KiB
Firmware rollouts
Upgrade RouterOS across a fleet in waves, with a verified pipeline per device and a stopping point before a bad build reaches everything.
The pipeline
Each device goes through the same sequence:
- Pre-upgrade backup (optional, on by default)
- Check for updates — a device already on the target version is skipped, not failed
- Download the image, and confirm it landed
- Reboot
- Prove the reboot happened — uptime must have gone backwards
- Verify the version moved
- RouterBOOT (optional) — a second flash and a second reboot
Steps 3 and 5 exist because of a specific failure. /system/package/update/install
bundles download and reboot into one call whose progress cannot be observed, and a
CCR2216 was seen to accept it, do neither, and report nothing. That surfaced as
"rebooted but still reports 7.24.1", because nothing could tell a device that never
restarted from one that restarted unchanged.
Downloading first makes success verifiable before anything reboots, so a device is never restarted for an image it does not have.
!!! note "Downloads are slow, and that is normal"
/system/package/update/download blocks until the image has landed — a minute or
more on an ordinary link. It is given a ten-minute budget. If the command does not
return cleanly the device is asked again on a fresh connection what actually
happened, rather than being declared failed: a slow download is not a broken one.
Waves
Waves are the point. Devices are assigned to wave 1, 2 or 3, and waves run strictly in sequence. Wave 1 is your canary: put one device in it, let it finish, and the rest of the fleet is still untouched if it goes wrong.
Halt on failure stops the rollout rather than continuing into the next wave.
Devices never reached are marked skipped, not failed — they did not run.
A device that comes back on the old version counts as a failure.
Upgrading several devices at once
By default the devices inside a wave upgrade one at a time. That is safe and slow: a CRS switch can take five minutes to reboot, so ten of them is most of an hour spent waiting.
The At once control raises how many devices in the same wave may upgrade simultaneously. Waves remain sequential either way, so the canary still gates everything after it.
There are two costs, and both are worth understanding before you raise it.
Halting becomes less precise. With one device at a time, a failure stops the rollout immediately. Running several at once, the wave finishes first — so a wave of five can put a bad build on five devices before anything stops. At the default of 1 these are the same moment.
Devices reboot together. That is fine for independent routers. It is not fine for a switch that the others are reached through: rebooting it alongside them cuts the path to devices that are mid-upgrade. Change Guard does not cover this — it protects configuration, and a device that is unreachable because its uplink rebooted has no configuration change to revert.
The rollout builder warns when it can see that one selected device carries another's traffic. That warning is deliberately conservative: it fires only when a link is the device's spanning-tree root port and that port resolves to exactly one neighbour. Discovered topology is often ambiguous — a trunk port can resolve to several neighbours at once — and naming the wrong switch would be worse than saying nothing. Absence of a warning is not proof that it is safe to reboot everything together.
A reasonable pattern for a large fleet on a well-known patch release:
| Wave | Devices | At once |
|---|---|---|
| 1 | one representative device | 1 |
| 2 | the access layer | 5–10 |
| 3 | anything upstream of wave 2 | 1 |
Scheduling and cancelling
A rollout can be scheduled; pair it with a maintenance window by scheduling inside one. One rollout runs at a time.
Devices already belonging to an unfinished rollout are refused by name when you try to create another for them, before anything is written.
Cancelling never interrupts an in-flight flash. It stops before the next device, because interrupting a device mid-write is how you brick it. The UI says so when you press the button — the currently upgrading device will finish first.
When it fails
Failure messages name the mechanism rather than the symptom. "Device did not finish downloading within ten minutes. Nothing was rebooted. The device reports 91.6 MB free" tells you where to look; "upgrade failed" does not.
If a rollout reports a device stuck at the old version, check the device's own
/system/package/update state before re-running — it may have the image already.