feat(commands): run one command across a fleet, in waves (v0.23.27-beta)

Closes #118. Unblocked by #110, which is what makes SSH on sixty devices
practical.

Arbitrary RouterOS console commands over SSH, because `:put`, `:foreach` and
`/interface print` are console syntax that the binary API cannot express, and
because a curated command library cannot cover "add a channel to LACP" or
"rewrite a WireGuard endpoint" — the things people actually asked for.

**Waves are the feature, not the safety rail around it.** A shell loop already
runs a command on many hosts, costs nothing, and prints errors as they happen.
What it cannot do is stop. Devices run together within a wave, waves run strictly
in sequence, and halt-on-failure ends the run before a mistake becomes sixty
mistakes. Without that, this would be a worse bash.

Proved on hardware rather than argued: a bad command on two devices, one per
wave, failed on the first and left the second `skipped`. RouterOS reports most
errors in output text rather than an exit code, so output is read for failure
markers — otherwise a rejected command and a successful one are indistinguishable
and the run would sail on. Conservatively: only unambiguous markers count, since
misreading ordinary output would halt a run that was working.

Change Guard wraps each device by default, so one that stops answering after the
command restores itself. It is one click to turn off, because a sharp tool that
refuses to cut is not a tool — the requester's argument, and a fair one. The
combination that gets an explicit acknowledgement is a command that can sever
management *with the guard disabled*; that is the only case where nothing catches
a mistake.

A preview names devices, wave assignment, and the reasons a command looks capable
of cutting management access. It warns, never blocks. Devices with no SSH
credential are listed before the run rather than producing a wave of identical
authentication failures.

Per-device output is captured and kept, because "it failed" without the device's
own words is not actionable, and because some commands exist purely for output.

Verified end to end against real hardware: a read-only command across two
devices, halt-on-failure stopping a bad command after one device, and a genuine
write applied under Change Guard, confirmed, then reverted — with both devices
returned to their original configuration afterwards.

Also adds services/sshExec.ts, shared with the key work: it selects a verified
key or a password, and throws rather than swallowing. That last part is
deliberate — a swallowed error is exactly how the data-cap feature spent a week
doing nothing while looking healthy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/src/services/sshExec.ts

@@ -0,0 +1,131 @@
+/**
+ * Running a command on a device over SSH, with whichever credential works.
+ *
+ * Console syntax — `:put`, `:foreach`, `/interface print` — is not reachable over
+ * the binary API, which has its own command tree. Anything that runs what an
+ * operator would type has to go over SSH (#118).
+ *
+ * Credential choice is not a preference. RouterOS refuses password
+ * authentication for a user once a key is bound to them, so a device with an
+ * installed key can *only* be reached by that key. Password is the path for
+ * devices without one (#110).
+ */
+import { Client as SSHClient } from 'ssh2';
+import { queryOne } from '../config/database';
+import { decrypt } from '../utils/crypto';
+
+export interface SshExecDevice {
+  id: number;
+  name: string;
+  ip_address: string;
+  ssh_port?: number | null;
+  ssh_username?: string | null;
+  ssh_password_encrypted?: string | null;
+  api_username?: string | null;
+  api_password_encrypted?: string | null;
+}
+
+export interface SshExecResult {
+  /** Combined stdout and stderr, as an operator would see at a console. */
+  output: string;
+  /** Which credential actually carried the session. */
+  auth: 'key' | 'password';
+}
+
+const DEFAULT_TIMEOUT_MS = 30_000;
+
+/**
+ * Resolve the credential for a device.
+ *
+ * A key is used only when it has been *proved* — a key that was generated or
+ * pushed but never authenticated is not evidence the device will accept it.
+ */
+async function resolveAuth(device: SshExecDevice): Promise<{
+  username: string;
+  auth: { password: string } | { privateKey: string };
+  kind: 'key' | 'password';
+}> {
+  const key = await queryOne<{ private_key_encrypted: string; ssh_username: string | null }>(
+    `SELECT private_key_encrypted, ssh_username FROM device_ssh_keys
+      WHERE device_id = $1 AND status = 'verified'`,
+    [device.id]
+  );
+
+  if (key) {
+    const username = key.ssh_username || device.ssh_username || device.api_username;
+    if (!username) throw new Error('No SSH username available for key authentication');
+    return { username, auth: { privateKey: decrypt(key.private_key_encrypted) }, kind: 'key' };
+  }
+
+  const username = device.ssh_username || device.api_username;
+  if (!username) throw new Error('No SSH username configured');
+  const encrypted = device.ssh_password_encrypted || device.api_password_encrypted;
+  if (!encrypted) throw new Error('No SSH credential configured');
+  return { username, auth: { password: decrypt(encrypted) }, kind: 'password' };
+}
+
+/**
+ * Run one command and return everything it printed.
+ *
+ * Errors are thrown, never swallowed. A command runner that reports success
+ * because it discarded the failure is worse than one that does nothing — and a
+ * quietly discarded error is exactly how a shipped feature spent a week doing
+ * nothing at all.
+ */
+export async function runSshCommand(
+  device: SshExecDevice,
+  command: string,
+  timeoutMs = DEFAULT_TIMEOUT_MS,
+): Promise<SshExecResult> {
+  const { username, auth, kind } = await resolveAuth(device);
+
+  return new Promise<SshExecResult>((resolve, reject) => {
+    const conn = new SSHClient();
+    let settled = false;
+    const finish = (fn: () => void) => {
+      if (settled) return;
+      settled = true;
+      clearTimeout(timer);
+      try { conn.end(); } catch { /* already closing */ }
+      fn();
+    };
+
+    const timer = setTimeout(
+      () => finish(() => reject(new Error(`Command timed out after ${timeoutMs}ms`))),
+      timeoutMs,
+    );
+
+    conn.on('ready', () => {
+      conn.exec(command, (err, stream) => {
+        if (err) return finish(() => reject(err));
+        let output = '';
+        stream.on('data', (d: Buffer) => { output += d.toString(); });
+        stream.stderr.on('data', (d: Buffer) => { output += d.toString(); });
+        stream.on('close', () => finish(() => resolve({ output: output.trim(), auth: kind })));
+      });
+    });
+    conn.on('error', (err) => finish(() => reject(err)));
+
+    conn.connect({
+      host: device.ip_address,
+      port: device.ssh_port ?? 22,
+      username,
+      readyTimeout: Math.min(timeoutMs, 20_000),
+      ...auth,
+    });
+  });
+}
+
+/**
+ * Does RouterOS's output indicate the command failed?
+ *
+ * RouterOS reports most errors in the text rather than through an exit code, so
+ * a command that "succeeded" and one that printed `syntax error` are
+ * indistinguishable unless the output is read. Deliberately conservative: only
+ * unambiguous markers count, because misreading ordinary output as failure would
+ * halt a rollout that was working.
+ */
+export function looksLikeFailure(output: string): boolean {
+  return /(^|\n)\s*(syntax error|bad command name|expected end of command|no such item|failure:|input does not match any value)/i
+    .test(output);
+}