fix(firmware): download and verify before rebooting, and prove the reboot (v0.23.31-beta)

Reported as rollouts failing with "Device rebooted but still reports 7.24.1 —
the update did not apply", intermittently and on ad-hoc updates too. Three
separate faults, found by watching one rollout that succeeded on an access point
and failed on a switch in the same run.

**The device had never rebooted.** Its uptime was one week. The old sequence
issued `/system/package/update/install`, slept a fixed 25 seconds, reconnected,
read the version and — finding it unchanged — declared the upgrade failed. It
could not tell a device that restarted unchanged from one that never restarted
at all, and 25 seconds is nowhere near long enough: the access point that
*worked* took 245 seconds. So a device still busy downloading was routinely
reported as broken, and with halt-on-failure that aborted the rest of the fleet.

**`install` was the wrong command.** It bundles download and reboot into one
fire-and-forget call whose progress cannot be observed. On the CCR2216 it
accepted the command and then neither downloaded nor rebooted, reporting
nothing; an explicit `download` on the same device worked immediately. The
sequence is now download → confirm the device reports the image as downloaded →
reboot, which makes success verifiable *before* anything restarts and means a
device is never rebooted for an image it does not have.

**A successful reboot was being read as a failure.** Rebooting necessarily kills
the session that asked for it, so the transport error that follows is the
expected outcome. Unhandled, it surfaced as "Update failed before reboot:
Connection closed" on a device that had in fact rebooted and upgraded correctly.

The reboot is now proven by comparing uptime across it rather than assumed from
elapsed time, so "never restarted" is reported as itself — with the device left
running and reachable — instead of being mistaken for a failed upgrade.

Download failures now carry the device's free space, because that is the usual
cause and "did not finish in ten minutes" otherwise sends someone hunting the
network. A CRS309 with 1.06 MB free now says so instead of timing out silently.

Verified on hardware: the CCR2216 that produced the original report upgraded
7.24.1 to 7.24.2 cleanly, and the CRS309 that genuinely cannot fit the image
failed with an accurate message and was left untouched — same version, uptime
unbroken, never rebooted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2GT-Rich
2026-09-03 10:22:58 -07:00
co-authored by Claude Opus 5
parent 2eda3d47ae
commit 8474f8e789
6 changed files with 122 additions and 12 deletions
@@ -3681,6 +3681,38 @@ export class DeviceCollector {
return (result[0] as Record<string, string>) || {};
}
/**
* Download the pending update without rebooting.
*
* Preferred over `install` because it is *verifiable*: afterwards the device
* reports "Downloaded, please reboot", so the caller can confirm the image is
* actually on the device before restarting anything. `install` bundles the
* download and the reboot into one fire-and-forget call whose success cannot
* be observed — and on at least one CCR it neither downloaded nor rebooted
* while reporting nothing at all (#firmware).
*/
async downloadUpdate(): Promise<void> {
await this.client.execute('/system/package/update/download');
}
/** The `status` line from /system/package/update, e.g. "Downloaded, please reboot…". */
async getUpdateStatus(): Promise<string> {
const rows = await this.client.execute('/system/package/update/print').catch(() => []);
return (rows[0]?.['status'] || '').trim();
}
/** Seconds since the device last booted, for proving a reboot happened. */
async getUptimeSeconds(): Promise<number | null> {
const res = await this.getSystemResource().catch(() => ({} as Record<string, string>));
const raw = res['uptime'];
if (!raw) return null;
const units: Record<string, number> = { w: 604800, d: 86400, h: 3600, m: 60, s: 1 };
let total = 0;
let matched = false;
for (const [, n, u] of raw.matchAll(/(\d+)([wdhms])/g)) { total += parseInt(n, 10) * units[u]; matched = true; }
return matched ? total : null;
}
async installUpdate(): Promise<void> {
await this.client.execute('/system/package/update/install');
}
@@ -3699,8 +3731,22 @@ export class DeviceCollector {
await this.client.execute('/system/reboot');
}
/**
* Reboot the device.
*
* A reboot that works necessarily kills the session that asked for it, so the
* transport error that follows is the *expected* outcome, not a failure. Left
* unhandled it surfaced as "Update failed before reboot: Connection closed" on
* a device that had in fact rebooted and upgraded correctly.
*/
async reboot(): Promise<void> {
await this.client.execute('/system/reboot');
try {
await this.client.execute('/system/reboot');
} catch (e) {
const msg = (e as Error).message || '';
if (/closed|reset|EPIPE|not connected|timed? ?out|socket/i.test(msg)) return;
throw e;
}
}
// ─── Ethernet monitor / SFP DDM ───────────────────────────────────────────