2GT-RichandClaude Opus 5
1a4b635052
A regression I introduced in v0.24.3, and described inaccurately in its own commit message. Adding concurrency moved the halt check to the end of a wave. I wrote that "at the default of 1 those are the same moment". That is only true of a wave holding one device. With several devices in one wave and the default sequential setting, the old code stopped the moment one failed; mine went on to attempt every remaining device in that wave first. Anyone relying on halt-on-failure with a wide wave 1 got a wider blast radius than before, without changing a setting. Halting now stops further devices from *starting* as soon as a failure is known, whatever the concurrency. Devices already in flight are allowed to finish, because interrupting one mid-write is how you brick it. At concurrency 1 that is precisely the previous behaviour. Above 1, only the devices genuinely running side by side can carry a bad build -- which is the unavoidable part of the trade in #135, rather than a cost imposed on rollouts that never asked for concurrency. The firmware documentation said the same wrong thing and is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4.8 KiB
Firmware rollouts
Upgrade RouterOS across a fleet in waves, with a verified pipeline per device and a stopping point before a bad build reaches everything.
The pipeline
Each device goes through the same sequence:
- Pre-upgrade backup (optional, on by default)
- Check for updates — a device already on the target version is skipped, not failed
- Download the image, and confirm it landed
- Reboot
- Prove the reboot happened — uptime must have gone backwards
- Verify the version moved
- RouterBOOT (optional) — a second flash and a second reboot
Steps 3 and 5 exist because of a specific failure. /system/package/update/install
bundles download and reboot into one call whose progress cannot be observed, and a
CCR2216 was seen to accept it, do neither, and report nothing. That surfaced as
"rebooted but still reports 7.24.1", because nothing could tell a device that never
restarted from one that restarted unchanged.
Downloading first makes success verifiable before anything reboots, so a device is never restarted for an image it does not have.
!!! note "Downloads are slow, and that is normal"
/system/package/update/download blocks until the image has landed — a minute or
more on an ordinary link. It is given a ten-minute budget. If the command does not
return cleanly the device is asked again on a fresh connection what actually
happened, rather than being declared failed: a slow download is not a broken one.
Waves
Waves are the point. Devices are assigned to wave 1, 2 or 3, and waves run strictly in sequence. Wave 1 is your canary: put one device in it, let it finish, and the rest of the fleet is still untouched if it goes wrong.
Halt on failure stops the rollout rather than continuing into the next wave.
Devices never reached are marked skipped, not failed — they did not run.
A device that comes back on the old version counts as a failure.
Upgrading several devices at once
By default the devices inside a wave upgrade one at a time. That is safe and slow: a CRS switch can take five minutes to reboot, so ten of them is most of an hour spent waiting.
The At once control raises how many devices in the same wave may upgrade simultaneously. Waves remain sequential either way, so the canary still gates everything after it.
There are two costs, and both are worth understanding before you raise it.
Halting becomes less precise. A failure stops any further device from starting, immediately — but devices already running finish, because interrupting one mid-write is how you brick it. So with At once set to five, up to five devices can carry a bad build before anything stops. At the default of 1 nothing else has started, so a failure stops the rollout exactly as it always has.
Devices reboot together. That is fine for independent routers. It is not fine for a switch that the others are reached through: rebooting it alongside them cuts the path to devices that are mid-upgrade. Change Guard does not cover this — it protects configuration, and a device that is unreachable because its uplink rebooted has no configuration change to revert.
The rollout builder warns when it can see that one selected device carries another's traffic. That warning is deliberately conservative: it fires only when a link is the device's spanning-tree root port and that port resolves to exactly one neighbour. Discovered topology is often ambiguous — a trunk port can resolve to several neighbours at once — and naming the wrong switch would be worse than saying nothing. Absence of a warning is not proof that it is safe to reboot everything together.
A reasonable pattern for a large fleet on a well-known patch release:
| Wave | Devices | At once |
|---|---|---|
| 1 | one representative device | 1 |
| 2 | the access layer | 5–10 |
| 3 | anything upstream of wave 2 | 1 |
Scheduling and cancelling
A rollout can be scheduled; pair it with a maintenance window by scheduling inside one. One rollout runs at a time.
Devices already belonging to an unfinished rollout are refused by name when you try to create another for them, before anything is written.
Cancelling never interrupts an in-flight flash. It stops before the next device, because interrupting a device mid-write is how you brick it. The UI says so when you press the button — the currently upgrading device will finish first.
When it fails
Failure messages name the mechanism rather than the symptom. "Device did not finish downloading within ten minutes. Nothing was rebooted. The device reports 91.6 MB free" tells you where to look; "upgrade failed" does not.
If a rollout reports a device stuck at the old version, check the device's own
/system/package/update state before re-running — it may have the image already.