fix(firmware): halt-on-failure stopped waiting for the wave to finish (v0.24.5-beta)
CI / Deployment files in sync (push) Canceled after 0s
CI / Frontend — Lint, Type-check, Test & Build (push) Canceled after 0s
CI / Backend — Lint, Type-check & Build (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CodeQL Security Analysis / Analyze (javascript-typescript) (push) Canceled after 0s
Documentation / Build (check only) (push) Canceled after 0s
Documentation / Publish versioned site (push) Canceled after 0s
Publish to GHCR / Build & Push Backend (push) Canceled after 0s
Publish to GHCR / Build & Push Nginx (Frontend) (push) Canceled after 0s
Release / release (push) Canceled after 0s

A regression I introduced in v0.24.3, and described inaccurately in its own
commit message.

Adding concurrency moved the halt check to the end of a wave. I wrote that
"at the default of 1 those are the same moment". That is only true of a wave
holding one device. With several devices in one wave and the default
sequential setting, the old code stopped the moment one failed; mine went on
to attempt every remaining device in that wave first. Anyone relying on
halt-on-failure with a wide wave 1 got a wider blast radius than before,
without changing a setting.

Halting now stops further devices from *starting* as soon as a failure is
known, whatever the concurrency. Devices already in flight are allowed to
finish, because interrupting one mid-write is how you brick it. At
concurrency 1 that is precisely the previous behaviour. Above 1, only the
devices genuinely running side by side can carry a bad build -- which is the
unavoidable part of the trade in #135, rather than a cost imposed on rollouts
that never asked for concurrency.

The firmware documentation said the same wrong thing and is corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2GT-Rich
2026-09-08 08:54:51 -07:00
co-authored by Claude Opus 5
parent d9fa120abc
commit 1a4b635052
6 changed files with 28 additions and 19 deletions
+1 -1
View File
@@ -6,7 +6,7 @@
</p>
<p align="center">
<img src="https://img.shields.io/badge/version-0.24.4_Beta-blue" alt="Version" />
<img src="https://img.shields.io/badge/version-0.24.5_Beta-blue" alt="Version" />
<img src="https://img.shields.io/badge/license-AGPLv3-blue" alt="License" />
<img src="https://img.shields.io/badge/docker-compose-2496ED?logo=docker&logoColor=white" alt="Docker" />
<img src="https://img.shields.io/badge/TypeScript-5.3-3178C6?logo=typescript&logoColor=white" alt="TypeScript" />
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mikrotik-manager-backend",
"version": "0.24.4-beta",
"version": "0.24.5-beta",
"description": "Mikrotik Network Management Platform - Backend API",
"main": "dist/index.js",
"scripts": {
+19 -11
View File
@@ -98,12 +98,16 @@ export class FirmwareOrchestrator {
// Waves stay strictly sequential -- that is the decision point, and it is
// what makes wave 1 a canary. Concurrency applies only *within* a wave.
//
// At the default of 1 this is the previous behaviour with one difference
// worth stating: halting is evaluated at the end of a wave rather than the
// instant a device fails. With a wave of one those are the same moment. With
// a larger wave it is the trade the operator asked for (#135) -- more devices
// carry a bad build before anything stops, in exchange for not waiting out
// ten five-minute reboots in series.
// Halt-on-failure stops *starting* further devices the moment a failure is
// known, rather than at the end of the wave. Devices already in flight are
// allowed to finish, because interrupting a device mid-write is how you
// brick it.
//
// At the default concurrency of 1 that is exactly the previous behaviour:
// a failure stops the next device immediately. Only devices genuinely
// running side by side can carry a bad build, which is the unavoidable part
// of the trade in #135 -- not a wider blast radius imposed on rollouts that
// never asked for concurrency.
for (const wave of waves) {
if (this.cancelRequested || halted) break;
const inWave = items.filter((i) => i.wave === wave);
@@ -113,11 +117,15 @@ export class FirmwareOrchestrator {
`${inWave.length} device(s), up to ${concurrency} at once`);
}
const results = await mapWithConcurrency(inWave, concurrency, (item) =>
this.cancelRequested
? Promise.resolve(false)
: this.upgradeDevice(rollout, item)
);
let failedInWave = false;
const results = await mapWithConcurrency(inWave, concurrency, async (item) => {
if (this.cancelRequested) return false;
// Do not begin another device once this wave has already failed.
if (rollout.halt_on_failure && failedInWave) return false;
const ok = await this.upgradeDevice(rollout, item);
if (!ok) failedInWave = true;
return ok;
});
if (rollout.halt_on_failure && results.some((ok) => !ok)) {
halted = true;
+5 -4
View File
@@ -55,10 +55,11 @@ everything after it.
There are two costs, and both are worth understanding before you raise it.
**Halting becomes less precise.** With one device at a time, a failure stops the
rollout immediately. Running several at once, the wave finishes first — so a wave of
five can put a bad build on five devices before anything stops. At the default of 1
these are the same moment.
**Halting becomes less precise.** A failure stops any further device from *starting*,
immediately — but devices already running finish, because interrupting one mid-write is
how you brick it. So with **At once** set to five, up to five devices can carry a bad
build before anything stops. At the default of 1 nothing else has started, so a failure
stops the rollout exactly as it always has.
**Devices reboot together.** That is fine for independent routers. It is not fine for
a switch that the others are reached *through*: rebooting it alongside them cuts the
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mikrotik-manager-frontend",
"version": "0.24.4-beta",
"version": "0.24.5-beta",
"private": true,
"type": "module",
"scripts": {
+1 -1
View File
@@ -1 +1 @@
export const APP_VERSION = 'v0.24.4 Beta';
export const APP_VERSION = 'v0.24.5 Beta';