fix(firmware): halt-on-failure stopped waiting for the wave to finish (v0.24.5-beta)
CI / Deployment files in sync (push) Canceled after 0s
CI / Frontend — Lint, Type-check, Test & Build (push) Canceled after 0s
CI / Backend — Lint, Type-check & Build (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CodeQL Security Analysis / Analyze (javascript-typescript) (push) Canceled after 0s
Documentation / Build (check only) (push) Canceled after 0s
Documentation / Publish versioned site (push) Canceled after 0s
Publish to GHCR / Build & Push Backend (push) Canceled after 0s
Publish to GHCR / Build & Push Nginx (Frontend) (push) Canceled after 0s
Release / release (push) Canceled after 0s
CI / Deployment files in sync (push) Canceled after 0s
CI / Frontend — Lint, Type-check, Test & Build (push) Canceled after 0s
CI / Backend — Lint, Type-check & Build (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CodeQL Security Analysis / Analyze (javascript-typescript) (push) Canceled after 0s
Documentation / Build (check only) (push) Canceled after 0s
Documentation / Publish versioned site (push) Canceled after 0s
Publish to GHCR / Build & Push Backend (push) Canceled after 0s
Publish to GHCR / Build & Push Nginx (Frontend) (push) Canceled after 0s
Release / release (push) Canceled after 0s
A regression I introduced in v0.24.3, and described inaccurately in its own commit message. Adding concurrency moved the halt check to the end of a wave. I wrote that "at the default of 1 those are the same moment". That is only true of a wave holding one device. With several devices in one wave and the default sequential setting, the old code stopped the moment one failed; mine went on to attempt every remaining device in that wave first. Anyone relying on halt-on-failure with a wide wave 1 got a wider blast radius than before, without changing a setting. Halting now stops further devices from *starting* as soon as a failure is known, whatever the concurrency. Devices already in flight are allowed to finish, because interrupting one mid-write is how you brick it. At concurrency 1 that is precisely the previous behaviour. Above 1, only the devices genuinely running side by side can carry a bad build -- which is the unavoidable part of the trade in #135, rather than a cost imposed on rollouts that never asked for concurrency. The firmware documentation said the same wrong thing and is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
d9fa120abc
commit
1a4b635052
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "mikrotik-manager-backend",
|
||||
"version": "0.24.4-beta",
|
||||
"version": "0.24.5-beta",
|
||||
"description": "Mikrotik Network Management Platform - Backend API",
|
||||
"main": "dist/index.js",
|
||||
"scripts": {
|
||||
|
||||
@@ -98,12 +98,16 @@ export class FirmwareOrchestrator {
|
||||
// Waves stay strictly sequential -- that is the decision point, and it is
|
||||
// what makes wave 1 a canary. Concurrency applies only *within* a wave.
|
||||
//
|
||||
// At the default of 1 this is the previous behaviour with one difference
|
||||
// worth stating: halting is evaluated at the end of a wave rather than the
|
||||
// instant a device fails. With a wave of one those are the same moment. With
|
||||
// a larger wave it is the trade the operator asked for (#135) -- more devices
|
||||
// carry a bad build before anything stops, in exchange for not waiting out
|
||||
// ten five-minute reboots in series.
|
||||
// Halt-on-failure stops *starting* further devices the moment a failure is
|
||||
// known, rather than at the end of the wave. Devices already in flight are
|
||||
// allowed to finish, because interrupting a device mid-write is how you
|
||||
// brick it.
|
||||
//
|
||||
// At the default concurrency of 1 that is exactly the previous behaviour:
|
||||
// a failure stops the next device immediately. Only devices genuinely
|
||||
// running side by side can carry a bad build, which is the unavoidable part
|
||||
// of the trade in #135 -- not a wider blast radius imposed on rollouts that
|
||||
// never asked for concurrency.
|
||||
for (const wave of waves) {
|
||||
if (this.cancelRequested || halted) break;
|
||||
const inWave = items.filter((i) => i.wave === wave);
|
||||
@@ -113,11 +117,15 @@ export class FirmwareOrchestrator {
|
||||
`${inWave.length} device(s), up to ${concurrency} at once`);
|
||||
}
|
||||
|
||||
const results = await mapWithConcurrency(inWave, concurrency, (item) =>
|
||||
this.cancelRequested
|
||||
? Promise.resolve(false)
|
||||
: this.upgradeDevice(rollout, item)
|
||||
);
|
||||
let failedInWave = false;
|
||||
const results = await mapWithConcurrency(inWave, concurrency, async (item) => {
|
||||
if (this.cancelRequested) return false;
|
||||
// Do not begin another device once this wave has already failed.
|
||||
if (rollout.halt_on_failure && failedInWave) return false;
|
||||
const ok = await this.upgradeDevice(rollout, item);
|
||||
if (!ok) failedInWave = true;
|
||||
return ok;
|
||||
});
|
||||
|
||||
if (rollout.halt_on_failure && results.some((ok) => !ok)) {
|
||||
halted = true;
|
||||
|
||||
Reference in New Issue
Block a user