fix(firmware): halt-on-failure stopped waiting for the wave to finish (v0.24.5-beta)
CI / Deployment files in sync (push) Canceled after 0s
CI / Frontend — Lint, Type-check, Test & Build (push) Canceled after 0s
CI / Backend — Lint, Type-check & Build (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CodeQL Security Analysis / Analyze (javascript-typescript) (push) Canceled after 0s
Documentation / Build (check only) (push) Canceled after 0s
Documentation / Publish versioned site (push) Canceled after 0s
Publish to GHCR / Build & Push Backend (push) Canceled after 0s
Publish to GHCR / Build & Push Nginx (Frontend) (push) Canceled after 0s
Release / release (push) Canceled after 0s
CI / Deployment files in sync (push) Canceled after 0s
CI / Frontend — Lint, Type-check, Test & Build (push) Canceled after 0s
CI / Backend — Lint, Type-check & Build (push) Canceled after 0s
CI / Security Audit (push) Canceled after 0s
CodeQL Security Analysis / Analyze (javascript-typescript) (push) Canceled after 0s
Documentation / Build (check only) (push) Canceled after 0s
Documentation / Publish versioned site (push) Canceled after 0s
Publish to GHCR / Build & Push Backend (push) Canceled after 0s
Publish to GHCR / Build & Push Nginx (Frontend) (push) Canceled after 0s
Release / release (push) Canceled after 0s
A regression I introduced in v0.24.3, and described inaccurately in its own commit message. Adding concurrency moved the halt check to the end of a wave. I wrote that "at the default of 1 those are the same moment". That is only true of a wave holding one device. With several devices in one wave and the default sequential setting, the old code stopped the moment one failed; mine went on to attempt every remaining device in that wave first. Anyone relying on halt-on-failure with a wide wave 1 got a wider blast radius than before, without changing a setting. Halting now stops further devices from *starting* as soon as a failure is known, whatever the concurrency. Devices already in flight are allowed to finish, because interrupting one mid-write is how you brick it. At concurrency 1 that is precisely the previous behaviour. Above 1, only the devices genuinely running side by side can carry a bad build -- which is the unavoidable part of the trade in #135, rather than a cost imposed on rollouts that never asked for concurrency. The firmware documentation said the same wrong thing and is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
d9fa120abc
commit
1a4b635052
@@ -6,7 +6,7 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="https://img.shields.io/badge/version-0.24.4_Beta-blue" alt="Version" />
|
<img src="https://img.shields.io/badge/version-0.24.5_Beta-blue" alt="Version" />
|
||||||
<img src="https://img.shields.io/badge/license-AGPLv3-blue" alt="License" />
|
<img src="https://img.shields.io/badge/license-AGPLv3-blue" alt="License" />
|
||||||
<img src="https://img.shields.io/badge/docker-compose-2496ED?logo=docker&logoColor=white" alt="Docker" />
|
<img src="https://img.shields.io/badge/docker-compose-2496ED?logo=docker&logoColor=white" alt="Docker" />
|
||||||
<img src="https://img.shields.io/badge/TypeScript-5.3-3178C6?logo=typescript&logoColor=white" alt="TypeScript" />
|
<img src="https://img.shields.io/badge/TypeScript-5.3-3178C6?logo=typescript&logoColor=white" alt="TypeScript" />
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "mikrotik-manager-backend",
|
"name": "mikrotik-manager-backend",
|
||||||
"version": "0.24.4-beta",
|
"version": "0.24.5-beta",
|
||||||
"description": "Mikrotik Network Management Platform - Backend API",
|
"description": "Mikrotik Network Management Platform - Backend API",
|
||||||
"main": "dist/index.js",
|
"main": "dist/index.js",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|||||||
@@ -98,12 +98,16 @@ export class FirmwareOrchestrator {
|
|||||||
// Waves stay strictly sequential -- that is the decision point, and it is
|
// Waves stay strictly sequential -- that is the decision point, and it is
|
||||||
// what makes wave 1 a canary. Concurrency applies only *within* a wave.
|
// what makes wave 1 a canary. Concurrency applies only *within* a wave.
|
||||||
//
|
//
|
||||||
// At the default of 1 this is the previous behaviour with one difference
|
// Halt-on-failure stops *starting* further devices the moment a failure is
|
||||||
// worth stating: halting is evaluated at the end of a wave rather than the
|
// known, rather than at the end of the wave. Devices already in flight are
|
||||||
// instant a device fails. With a wave of one those are the same moment. With
|
// allowed to finish, because interrupting a device mid-write is how you
|
||||||
// a larger wave it is the trade the operator asked for (#135) -- more devices
|
// brick it.
|
||||||
// carry a bad build before anything stops, in exchange for not waiting out
|
//
|
||||||
// ten five-minute reboots in series.
|
// At the default concurrency of 1 that is exactly the previous behaviour:
|
||||||
|
// a failure stops the next device immediately. Only devices genuinely
|
||||||
|
// running side by side can carry a bad build, which is the unavoidable part
|
||||||
|
// of the trade in #135 -- not a wider blast radius imposed on rollouts that
|
||||||
|
// never asked for concurrency.
|
||||||
for (const wave of waves) {
|
for (const wave of waves) {
|
||||||
if (this.cancelRequested || halted) break;
|
if (this.cancelRequested || halted) break;
|
||||||
const inWave = items.filter((i) => i.wave === wave);
|
const inWave = items.filter((i) => i.wave === wave);
|
||||||
@@ -113,11 +117,15 @@ export class FirmwareOrchestrator {
|
|||||||
`${inWave.length} device(s), up to ${concurrency} at once`);
|
`${inWave.length} device(s), up to ${concurrency} at once`);
|
||||||
}
|
}
|
||||||
|
|
||||||
const results = await mapWithConcurrency(inWave, concurrency, (item) =>
|
let failedInWave = false;
|
||||||
this.cancelRequested
|
const results = await mapWithConcurrency(inWave, concurrency, async (item) => {
|
||||||
? Promise.resolve(false)
|
if (this.cancelRequested) return false;
|
||||||
: this.upgradeDevice(rollout, item)
|
// Do not begin another device once this wave has already failed.
|
||||||
);
|
if (rollout.halt_on_failure && failedInWave) return false;
|
||||||
|
const ok = await this.upgradeDevice(rollout, item);
|
||||||
|
if (!ok) failedInWave = true;
|
||||||
|
return ok;
|
||||||
|
});
|
||||||
|
|
||||||
if (rollout.halt_on_failure && results.some((ok) => !ok)) {
|
if (rollout.halt_on_failure && results.some((ok) => !ok)) {
|
||||||
halted = true;
|
halted = true;
|
||||||
|
|||||||
+5
-4
@@ -55,10 +55,11 @@ everything after it.
|
|||||||
|
|
||||||
There are two costs, and both are worth understanding before you raise it.
|
There are two costs, and both are worth understanding before you raise it.
|
||||||
|
|
||||||
**Halting becomes less precise.** With one device at a time, a failure stops the
|
**Halting becomes less precise.** A failure stops any further device from *starting*,
|
||||||
rollout immediately. Running several at once, the wave finishes first — so a wave of
|
immediately — but devices already running finish, because interrupting one mid-write is
|
||||||
five can put a bad build on five devices before anything stops. At the default of 1
|
how you brick it. So with **At once** set to five, up to five devices can carry a bad
|
||||||
these are the same moment.
|
build before anything stops. At the default of 1 nothing else has started, so a failure
|
||||||
|
stops the rollout exactly as it always has.
|
||||||
|
|
||||||
**Devices reboot together.** That is fine for independent routers. It is not fine for
|
**Devices reboot together.** That is fine for independent routers. It is not fine for
|
||||||
a switch that the others are reached *through*: rebooting it alongside them cuts the
|
a switch that the others are reached *through*: rebooting it alongside them cuts the
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "mikrotik-manager-frontend",
|
"name": "mikrotik-manager-frontend",
|
||||||
"version": "0.24.4-beta",
|
"version": "0.24.5-beta",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
export const APP_VERSION = 'v0.24.4 Beta';
|
export const APP_VERSION = 'v0.24.5 Beta';
|
||||||
|
|||||||
Reference in New Issue
Block a user