Commit Graph
2 Commits
Author SHA1 Message Date
2GT-RichandClaude Opus 5 63b5677522 fix(firmware,ssh,logs): five reported bugs (v0.24.1-beta)
Closes #136. Also fixes #133, #137, #138, #139 (GitHub closes only the first).

None of these came from multi-site; two came from me.

#136 — waves failed on the second device. A regression from v0.23.31, the
firmware rewrite that replaced fire-and-forget `install` with `download` plus
polling. `/system/package/update/download` blocks for a minute or more, and
the client's read timeout is 30 seconds. The timeout path was the real damage:
it nulled the pending read but left the socket open, so when the reply finally
arrived processBuffer() queued it and the *next* command took it as its own
answer. Every later call on that connection returned the previous one's data,
which is why the status polls could never see "Downloaded".

  - a timed-out read now abandons the connection instead of silently
    desynchronising it, so the failure is loud rather than wrong
  - long-running commands can state their own budget; check-for-updates,
    download and /log/print now do
  - a download whose command times out is re-checked on a fresh connection
    rather than assumed failed, because the image may well have landed

#133 — the SSH terminal could not use a deployed key. It read only
ssh_password_encrypted and refused to connect without one, so retiring the
password after deploying a key -- the workflow the feature exists for -- broke
the console. It now uses the same resolver as every other SSH consumer.
preferredAuth(), written to make exactly this decision, had never run outside
its own tests, and disagreed with the resolver that did: it accepted a
'deployed' key where the live path required 'verified'. Unified on the
stricter rule and wired in.

#139 — the rollout was written before start() was attempted, so a rejected
start left the rows behind: "already running", press again, another rollout
banked each time. Devices already in an unfinished rollout are now refused by
name, before anything is persisted.

#138 — cancel worked but the API's explanation that the in-flight device is
never interrupted was discarded, so the button looked dead. It is shown.

#137 — log overfetching. /log/print returns the whole ring buffer, and the
count was incremented per attempted insert rather than per accepted row, so a
poll that stored nothing still reported "Collected 1000 new log entries" after
a thousand round trips. Inserts are batched and counted by RETURNING, and the
id watermark is the maximum over recent rows rather than the last one
inserted, which could move backwards and make the whole buffer look new.

One self-inflicted wound worth recording. The first cut of the desync guard
discarded sentences whenever `connected` was false -- but `connected` is only
set *after* login succeeds, so it threw away every login reply and took the
whole fleet offline until it was caught. "Not yet connected" and "abandoned"
are different states and now have different flags. Every existing test pre-set
connected = true, so none of them could see it; the test added for it fails
against the bad version.

Verified on hardware: devices back online, check-update 5.2s (was a 30s
failure), full sync completes, and log collection now reports 298 / 22 / 12 /
12 across successive polls -- real decreasing counts rather than a constant
1000 -- with no failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-07 19:17:52 -07:00
2GT-RichandClaude Opus 5 0e364b1558 feat(commands): run one command across a fleet, in waves (v0.23.27-beta)
Closes #118. Unblocked by #110, which is what makes SSH on sixty devices
practical.

Arbitrary RouterOS console commands over SSH, because `:put`, `:foreach` and
`/interface print` are console syntax that the binary API cannot express, and
because a curated command library cannot cover "add a channel to LACP" or
"rewrite a WireGuard endpoint" — the things people actually asked for.

**Waves are the feature, not the safety rail around it.** A shell loop already
runs a command on many hosts, costs nothing, and prints errors as they happen.
What it cannot do is stop. Devices run together within a wave, waves run strictly
in sequence, and halt-on-failure ends the run before a mistake becomes sixty
mistakes. Without that, this would be a worse bash.

Proved on hardware rather than argued: a bad command on two devices, one per
wave, failed on the first and left the second `skipped`. RouterOS reports most
errors in output text rather than an exit code, so output is read for failure
markers — otherwise a rejected command and a successful one are indistinguishable
and the run would sail on. Conservatively: only unambiguous markers count, since
misreading ordinary output would halt a run that was working.

Change Guard wraps each device by default, so one that stops answering after the
command restores itself. It is one click to turn off, because a sharp tool that
refuses to cut is not a tool — the requester's argument, and a fair one. The
combination that gets an explicit acknowledgement is a command that can sever
management *with the guard disabled*; that is the only case where nothing catches
a mistake.

A preview names devices, wave assignment, and the reasons a command looks capable
of cutting management access. It warns, never blocks. Devices with no SSH
credential are listed before the run rather than producing a wave of identical
authentication failures.

Per-device output is captured and kept, because "it failed" without the device's
own words is not actionable, and because some commands exist purely for output.

Verified end to end against real hardware: a read-only command across two
devices, halt-on-failure stopping a bad command after one device, and a genuine
write applied under Change Guard, confirmed, then reverted — with both devices
returned to their original configuration afterwards.

Also adds services/sshExec.ts, shared with the key work: it selects a verified
key or a password, and throws rather than swallowing. That last part is
deliberate — a swallowed error is exactly how the data-cap feature spent a week
doing nothing while looking healthy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 14:20:02 -07:00