feat(firmware): optional parallel upgrades within a wave (v0.24.3-beta)

Closes #135. The reporter's numbers: CRS switches take ~5 minutes each to
reboot, so ten devices is an hour of mostly waiting.

Devices in a wave can now upgrade concurrently, up to a per-rollout limit.
Waves themselves stay strictly sequential — that is the decision point, and
what makes wave 1 a canary. The default is 1, which is the behaviour every
existing rollout was built around, and existing rows backfill to it.

CommandRunner already runs a wave with Promise.all, but an upgrade is not a
bulk command: it holds an API connection for minutes across a download, a
reboot and a verification, and connection pressure is exactly where this
system has been shown to break. So concurrency is bounded rather than
unlimited, via mapWithConcurrency, capped at 20 and clamped server-side.

The trade is stated rather than hidden. Halting is now evaluated at the end
of a wave instead of the instant a device fails, so a wave of five can put a
bad build on five devices before stopping. At the default of 1 those are the
same moment. The UI says so plainly whenever the setting is above 1.

An upstream warning covers the risk Change Guard cannot: several devices
rebooting together is fine for independent routers and dangerous for a switch
the others are reached through, and a device unreachable because its uplink
rebooted has no config change to revert.

The first implementation of that warning was wrong in an instructive way. It
flagged any device another selected device linked to — and on the real
four-device fleet it flagged all four, because discovered links are
bidirectional and a trunk port often resolves to several neighbours at once.
A warning that fires on everything is worse than none: it teaches people to
dismiss it. It now requires the link to be the reporting device's STP root
port *and* that port to resolve to exactly one neighbour, so it stays quiet
when it cannot tell — which, on this fleet's ambiguous topology, it correctly
does.

Verified on hardware with devices already up to date, so the wave machinery
ran end to end without flashing anything:

  - sequential: 04:20:21→:26, then :26→:31. No overlap, 12s.
  - concurrency 2: both started within 10ms of each other, 7s.
  - concurrency 5 across two waves: wave 2 started 3ms *after* wave 1
    finished, never during it.

Clamping checked at the API: 999 stored as 20, 0 and "abc" as 1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2GT-Rich
2026-09-07 21:23:08 -07:00
co-authored by Claude Opus 5
parent 719563dd22
commit df8f6dca44
13 changed files with 404 additions and 19 deletions
+2 -2
View File
@@ -6,7 +6,7 @@
</p>
<p align="center">
<img src="https://img.shields.io/badge/version-0.24.2_Beta-blue" alt="Version" />
<img src="https://img.shields.io/badge/version-0.24.3_Beta-blue" alt="Version" />
<img src="https://img.shields.io/badge/license-AGPLv3-blue" alt="License" />
<img src="https://img.shields.io/badge/docker-compose-2496ED?logo=docker&logoColor=white" alt="Docker" />
<img src="https://img.shields.io/badge/TypeScript-5.3-3178C6?logo=typescript&logoColor=white" alt="TypeScript" />
@@ -339,7 +339,7 @@ Each network service supports multi-device management with conflict detection:
### Fleet operations
- **Firmware orchestration** — fleet version overview, live update checks, and MikroTik's official release notes in-app. Staged rollouts run in **waves** (wave 1 = canary) through a verified pipeline per device: pre-upgrade backup → install → ride out the reboot → confirm it returned healthy on the new version → next. **Halt on failure** stops the rollout if any device fails, and a device that comes back on the *old* version counts as a failure. Rollouts can be scheduled, and cancelling never interrupts an in-flight flash
- **Firmware orchestration** — fleet version overview, live update checks, and MikroTik's official release notes in-app. Staged rollouts run in **waves** (wave 1 = canary) through a verified pipeline per device: pre-upgrade backup → download and confirm the image landed → reboot → prove it restarted and came back healthy on the new version → next. **Halt on failure** stops the rollout if any device fails, and a device that comes back on the *old* version counts as a failure. Devices in a wave upgrade one at a time by default; raise **at once** to run them concurrently when a fleet of five-minute reboots would otherwise take all evening — waves stay sequential either way, so the canary still gates the rest. Rollouts can be scheduled, and cancelling never interrupts an in-flight flash
- **Backups** — on demand or on a daily/weekly/monthly schedule for all online devices. Filter by device, type and date range; read one in place instead of downloading it; and compare any two as a diff — they are `/export` text, not opaque blobs. Config snapshots and their restorable `.rsc` are one artifact: delete either and both go, so they never drift apart, and a bulk delete says how many snapshots it will take with it
- **Configuration templates** — reusable sets (DNS, NTP, syslog) pushed to many devices with per-device result reporting
- **Bulk device add** — "Try All" on discovered devices runs as a server-side job that survives a closed browser tab, with live progress and cancel