2GT-RichandClaude Opus 5
1a4b635052
CI / Deployment files in sync (push) Canceled after 0s
CI / Frontend — Lint, Type-check, Test & Build (push) Canceled after 0s
CI / Backend — Lint, Type-check & Build (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CodeQL Security Analysis / Analyze (javascript-typescript) (push) Canceled after 0s
Documentation / Build (check only) (push) Canceled after 0s
Documentation / Publish versioned site (push) Canceled after 0s
Publish to GHCR / Build & Push Backend (push) Canceled after 0s
Publish to GHCR / Build & Push Nginx (Frontend) (push) Canceled after 0s
Release / release (push) Canceled after 0s
A regression I introduced in v0.24.3, and described inaccurately in its own commit message. Adding concurrency moved the halt check to the end of a wave. I wrote that "at the default of 1 those are the same moment". That is only true of a wave holding one device. With several devices in one wave and the default sequential setting, the old code stopped the moment one failed; mine went on to attempt every remaining device in that wave first. Anyone relying on halt-on-failure with a wide wave 1 got a wider blast radius than before, without changing a setting. Halting now stops further devices from *starting* as soon as a failure is known, whatever the concurrency. Devices already in flight are allowed to finish, because interrupting one mid-write is how you brick it. At concurrency 1 that is precisely the previous behaviour. Above 1, only the devices genuinely running side by side can carry a bad build -- which is the unavoidable part of the trade in #135, rather than a cost imposed on rollouts that never asked for concurrency. The firmware documentation said the same wrong thing and is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
105 lines
4.8 KiB
Markdown
105 lines
4.8 KiB
Markdown
# Firmware rollouts
|
||
|
||
[← Documentation index](README.md)
|
||
|
||
Upgrade RouterOS across a fleet in waves, with a verified pipeline per device and
|
||
a stopping point before a bad build reaches everything.
|
||
|
||
## The pipeline
|
||
|
||
Each device goes through the same sequence:
|
||
|
||
1. **Pre-upgrade backup** (optional, on by default)
|
||
2. **Check for updates** — a device already on the target version is *skipped*, not failed
|
||
3. **Download the image, and confirm it landed**
|
||
4. **Reboot**
|
||
5. **Prove the reboot happened** — uptime must have gone *backwards*
|
||
6. **Verify the version moved**
|
||
7. **RouterBOOT** (optional) — a second flash and a second reboot
|
||
|
||
Steps 3 and 5 exist because of a specific failure. `/system/package/update/install`
|
||
bundles download and reboot into one call whose progress cannot be observed, and a
|
||
CCR2216 was seen to accept it, do neither, and report nothing. That surfaced as
|
||
"rebooted but still reports 7.24.1", because nothing could tell a device that never
|
||
restarted from one that restarted unchanged.
|
||
|
||
Downloading first makes success verifiable *before* anything reboots, so a device is
|
||
never restarted for an image it does not have.
|
||
|
||
!!! note "Downloads are slow, and that is normal"
|
||
`/system/package/update/download` blocks until the image has landed — a minute or
|
||
more on an ordinary link. It is given a ten-minute budget. If the command does not
|
||
return cleanly the device is asked again on a fresh connection what actually
|
||
happened, rather than being declared failed: a slow download is not a broken one.
|
||
|
||
## Waves
|
||
|
||
Waves are the point. Devices are assigned to wave 1, 2 or 3, and **waves run strictly
|
||
in sequence**. Wave 1 is your canary: put one device in it, let it finish, and the
|
||
rest of the fleet is still untouched if it goes wrong.
|
||
|
||
**Halt on failure** stops the rollout rather than continuing into the next wave.
|
||
Devices never reached are marked `skipped`, not `failed` — they did not run.
|
||
|
||
A device that comes back on the *old* version counts as a failure.
|
||
|
||
## Upgrading several devices at once
|
||
|
||
By default the devices inside a wave upgrade **one at a time**. That is safe and slow:
|
||
a CRS switch can take five minutes to reboot, so ten of them is most of an hour spent
|
||
waiting.
|
||
|
||
The **At once** control raises how many devices in the same wave may upgrade
|
||
simultaneously. Waves remain sequential either way, so the canary still gates
|
||
everything after it.
|
||
|
||
There are two costs, and both are worth understanding before you raise it.
|
||
|
||
**Halting becomes less precise.** A failure stops any further device from *starting*,
|
||
immediately — but devices already running finish, because interrupting one mid-write is
|
||
how you brick it. So with **At once** set to five, up to five devices can carry a bad
|
||
build before anything stops. At the default of 1 nothing else has started, so a failure
|
||
stops the rollout exactly as it always has.
|
||
|
||
**Devices reboot together.** That is fine for independent routers. It is not fine for
|
||
a switch that the others are reached *through*: rebooting it alongside them cuts the
|
||
path to devices that are mid-upgrade. Change Guard does not cover this — it protects
|
||
configuration, and a device that is unreachable because its uplink rebooted has no
|
||
configuration change to revert.
|
||
|
||
The rollout builder warns when it can see that one selected device carries another's
|
||
traffic. That warning is deliberately conservative: it fires only when a link is the
|
||
device's spanning-tree root port *and* that port resolves to exactly one neighbour.
|
||
Discovered topology is often ambiguous — a trunk port can resolve to several
|
||
neighbours at once — and naming the wrong switch would be worse than saying nothing.
|
||
**Absence of a warning is not proof that it is safe to reboot everything together.**
|
||
|
||
A reasonable pattern for a large fleet on a well-known patch release:
|
||
|
||
| Wave | Devices | At once |
|
||
|---|---|---|
|
||
| 1 | one representative device | 1 |
|
||
| 2 | the access layer | 5–10 |
|
||
| 3 | anything upstream of wave 2 | 1 |
|
||
|
||
## Scheduling and cancelling
|
||
|
||
A rollout can be scheduled; pair it with a maintenance window by scheduling inside
|
||
one. One rollout runs at a time.
|
||
|
||
Devices already belonging to an unfinished rollout are refused by name when you try
|
||
to create another for them, before anything is written.
|
||
|
||
**Cancelling never interrupts an in-flight flash.** It stops before the *next* device,
|
||
because interrupting a device mid-write is how you brick it. The UI says so when you
|
||
press the button — the currently upgrading device will finish first.
|
||
|
||
## When it fails
|
||
|
||
Failure messages name the mechanism rather than the symptom. "Device did not finish
|
||
downloading within ten minutes. Nothing was rebooted. The device reports 91.6 MB free"
|
||
tells you where to look; "upgrade failed" does not.
|
||
|
||
If a rollout reports a device stuck at the old version, check the device's own
|
||
`/system/package/update` state before re-running — it may have the image already.
|