Closes#118. Unblocked by #110, which is what makes SSH on sixty devices
practical.
Arbitrary RouterOS console commands over SSH, because `:put`, `:foreach` and
`/interface print` are console syntax that the binary API cannot express, and
because a curated command library cannot cover "add a channel to LACP" or
"rewrite a WireGuard endpoint" — the things people actually asked for.
**Waves are the feature, not the safety rail around it.** A shell loop already
runs a command on many hosts, costs nothing, and prints errors as they happen.
What it cannot do is stop. Devices run together within a wave, waves run strictly
in sequence, and halt-on-failure ends the run before a mistake becomes sixty
mistakes. Without that, this would be a worse bash.
Proved on hardware rather than argued: a bad command on two devices, one per
wave, failed on the first and left the second `skipped`. RouterOS reports most
errors in output text rather than an exit code, so output is read for failure
markers — otherwise a rejected command and a successful one are indistinguishable
and the run would sail on. Conservatively: only unambiguous markers count, since
misreading ordinary output would halt a run that was working.
Change Guard wraps each device by default, so one that stops answering after the
command restores itself. It is one click to turn off, because a sharp tool that
refuses to cut is not a tool — the requester's argument, and a fair one. The
combination that gets an explicit acknowledgement is a command that can sever
management *with the guard disabled*; that is the only case where nothing catches
a mistake.
A preview names devices, wave assignment, and the reasons a command looks capable
of cutting management access. It warns, never blocks. Devices with no SSH
credential are listed before the run rather than producing a wave of identical
authentication failures.
Per-device output is captured and kept, because "it failed" without the device's
own words is not actionable, and because some commands exist purely for output.
Verified end to end against real hardware: a read-only command across two
devices, halt-on-failure stopping a bad command after one device, and a genuine
write applied under Change Guard, confirmed, then reverted — with both devices
returned to their original configuration afterwards.
Also adds services/sshExec.ts, shared with the key work: it selects a verified
key or a password, and throws rather than swallowing. That last part is
deliberate — a swallowed error is exactly how the data-cap feature spent a week
doing nothing while looking healthy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v0.23.17 stopped the poller creating an unbounded backlog but did nothing about
the one already there. A reporter with sixty devices upgraded and still had
752,318 jobs queued, nine stale devices, and an endpoint cheerfully reporting
1.59x headroom. Fair to call that not fixed.
Three faults, all mine.
**Retention never touched pending work.** cleanupJobHistory cleaned 'completed'
and 'failed' only. The queue of outstanding polls — the part the operator
actually feels — was out of scope, so an upgrade left it draining for days.
Stale pending jobs are now swept, at startup and every five minutes. The rule is
that a periodic poll which has waited longer than its own cadence is worthless:
running it yields a reading the next cycle would have taken anyway, while
occupying a worker the live fleet is waiting on. Discarding queued work is safe
precisely because it is periodic — the schedule re-enqueues within one interval.
**Deduplication failed in the case it existed for.** The in-flight claim was a
Redis key with a 135-second TTL. Under a large backlog a job waits hours, so the
claim lapsed long before it ran and the scheduler re-enqueued every cycle: the
guard worked only on a queue that was already short.
Replaced with a deterministic job id, which cannot lapse. While a job for a
device and kind exists in any state, adding it again is a no-op. That requires
finished jobs to be removed immediately or the id would stay taken and the
device would never be polled again — which costs nothing now that
device_poll_stats records every attempt, duration and error.
**The headroom metric lied.** It describes flow and is blind to accumulated
work, so it read "healthy" beside three-quarters of a million queued jobs. It
now reports backlog and drain ETA alongside, plus a single status verdict that
cannot say ok while a backlog exists. Verified by injecting 15,125 jobs: status
went to 'draining' with headroom still at 85x, which is exactly the case the old
metric got wrong.
Also added, because the reporter asked and was right to: an operator-triggered
drain (POST /api/system/poller/drain), and a Polling tab in Settings showing
queue depth, headroom, backlog, per-device staleness and the drain control. It
was all buried in an API response before. A dashboard insight fires when the
poller falls behind, so a starved poller announces itself rather than presenting
as a fleet of broken devices.
Two corrections found by testing rather than reasoning:
* BullMQ rejects ':' in a custom job id. `fast:2` threw on every enqueue and
killed the scheduler outright — three of four devices went unpolled for
eight minutes before it was caught. Hyphens now, and the dedup is exercised
directly: 500 enqueues of one device yield one queued job.
* Redis maxmemory-policy moves from volatile-lru to noeviction. I had reasoned
that only TTL-bearing keys are evicted and BullMQ job hashes carry none —
true, but its *control* keys do. Job locks and stalled-check markers both
expire, and evicting one causes duplicate processing or a stalled-job storm.
BullMQ warns about this on connect and was right.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses #114, where a ~60 device fleet saw devices stop reporting entirely
while others crawled, new devices never started polling, and a manual Sync
worked exactly once. All of it is one failure with two compounding causes.
**The queue grew faster than it drained.** Worker concurrency was hardcoded to 3
with no per-job timeout, and the scheduler enqueued a fast poll per device every
30 seconds regardless of whether the previous one had finished. Measured against
a live fleet a fast poll took ~1s at the median but 13s at p90, so three workers
sustained about 0.87 jobs/sec against an arrival rate of 2.0/sec at sixty
devices. The difference — roughly 98,000 jobs a day — accumulated in an unbounded
FIFO, and devices at the back of it were simply never polled. That is why Sync
appeared to work once and then stop: the Sync route runs inline and bypasses the
queue entirely, so it was the only thing still reaching those devices.
Three changes: concurrency is configurable and defaults to 12; every job runs
under a timeout so one unreachable device cannot hold a worker while the fleet
starves; and a device with a poll already queued or running is not enqueued
again. That last one is the important one — it makes a backlog impossible rather
than merely less likely. The claim is a Redis key with a TTL, so a worker that
dies frees its device on the next cycle instead of wedging it forever.
**Finished jobs were never removed.** BullMQ keeps them unless told otherwise.
A four-device development instance had accumulated 1,412,935 completed and
230,707 failed jobs across the three queues — 2,570,474 Redis keys and 1.16 GB,
against a Redis with no maxmemory configured. Left alone it grows until the host
dies. Retention is now bounded by age and count, and a background pass on
startup trims history from before the limits existed.
The two faults fed each other, which is measurable: after trimming 2.16M jobs
the mean fast poll on the same fleet fell from 3,448ms to 602ms and the slowest
from 16,095ms to 1,205ms. Multi-million-element sorted sets were slowing every
queue operation, which reduced throughput, which grew the backlog further.
**The reporter's follow-up is the real lesson** and is addressed too: there was
no way to tell whether a device had been missed, was unresponsive, or was merely
slow, and no view of queue depth or per-poll cost — so polling frequency was
being chosen by guesswork. device_poll_stats now records attempt and success
separately per device and kind, with duration and last error; their difference
is precisely the missing distinction. GET /api/system/poller reports queue
depths, worker settings, measured poll cost, and a headroom figure that states
whether the configured cadence is achievable at the current fleet size. Below
1.0 the backlog grows every cycle.
Verified on the live instance: 2,570,474 keys reduced to 2,336, memory 1.16 GB
to 32.25 MB, queues drained, per-device stats recording, no stranded in-flight
claims, and headroom reported at 149x for four devices — the same arithmetic
that would have read 0.43 for sixty devices before this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A user change can sever the manager's own path to a device with no warning and no
way back. Most recently, deleting an *unused* bond on a switch took the whole
switch offline. This adds a safety net, and fixes the bug that caused that
specific incident.
Root cause of the bond incident (DeviceCollector.deleteBond):
- Slaves were re-added to the bridge ONLY when the bond itself was a bridge
member. For an unbridged bond the restore loop never ran, so the member ports
were released from the bond and left in no bridge at all — the ports stopped
forwarding and the switch dropped off the network.
- The restore in both deleteBond and createBond was wrapped in `.catch(() => {})`,
so a failed restore was swallowed and the route still returned 200 OK.
- Restores also dropped PVID, silently returning tagged ports to VLAN 1.
Now: port settings (PVID, frame-types, ingress-filtering, edge, horizon) are
captured and carried over, and restore failures are reported instead of hidden.
Change Guard (services/changeGuard/ChangeGuard.ts):
Before a risky change the device saves a restore point and arms a scheduler that
restores it after N seconds. The change is applied, then reachability is proven
on a NEW connection; success disarms the scheduler, failure leaves the device to
restore itself and come back. Because the restore point is created before the
scheduler, a binary restore rolls back to a config containing neither the
scheduler nor the change — self-cleaning, with no revert loop.
- Works entirely over the RouterOS API. This matters: every recovery artifact
today requires SSH, so devices without SSH credentials had no rollback target
at all (cf. #86).
- Modes: `binary` (exact restore incl. deleted interfaces, reboots) or `script`
(/import, no reboot but additive). Configurable via change_guard_* settings.
- Per-device Redis lock, `device_change_guards` audit table, and startup
reconciliation for guards left pending by a manager restart.
- Degrades gracefully: if protection can't be armed the change still runs and the
response says it was unprotected.
Wired into bond create/delete, bridge vlan-filtering, and /ip/service. Disabling
the service the manager is connected through is now refused outright (409) — the
mgmt-service signal already existed but was confined to the read-only posture
endpoint.
Also: replaces the bare confirm() on "Destroy Bond" with a ChangeGuardDialog that
explains the risk and the protection, and reports the outcome (including "the
device is restoring itself"). Fixes getAppSettings omitting
config_snapshot_enabled/interval, which made the v0.17.4 snapshot toggle a no-op.
Verified on real hardware (hAP-class switch): capability probe confirmed
/system/backup/save and scheduler add/remove work over the API; a guarded change
armed, applied, verified and committed leaving zero schedulers and zero files on
the device; disabling the api service was refused with 409. 133 backend tests
pass, including 7 guard state-machine tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add OpenID Connect single sign-on so the platform can authenticate against
any standards-compliant provider (Entra ID, Okta, Google Workspace, Keycloak,
Authentik, …) alongside local accounts. Resolves the feature request in #73.
Flow (backend-driven Authorization Code + PKCE):
- GET /api/auth/oidc/login builds the authorize URL and stores the PKCE/nonce
transaction in Redis (single-use, 10m TTL); the callback validates the ID
token via openid-client (JWKS signature, iss/aud/exp/nonce), resolves the
identity to a local user, and mints the platform's normal session JWT —
handed to the SPA via URL fragment. No sessions/cookies introduced.
Identity mapping (users table gains email/oidc_subject/oidc_issuer/
auth_provider; password_hash now nullable; partial unique indexes):
- First sign-in auto-provisions a user, or links to an existing local account
when the IdP email is verified and matches.
- Roles derive from a configurable IdP group -> role map with a default
fallback; existing users are never auto-demoted on a groupless login.
Configuration (admin-only, no env/redeploy needed):
- Stored as an app_settings row with the client secret encrypted at rest
(utils/crypto); masked in responses and excluded from GET /api/settings.
- New UI: Settings -> SSO / OIDC panel (issuer/client, claim + group->role
mapping, provisioning toggles, discovery test, copyable redirect URI), a
login-page SSO button, and an SSO badge in Users & Roles.
Security: local break-glass login preserved; PKCE + single-use state; email
linking requires email_verified; same-site-only post-login redirect.
Adds openid-client@^5 (CommonJS). 17 new unit tests (claim/role mapping,
username derivation, config secret round-trip/masking); 114 backend tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the /terminal authorization fix: a full-codebase security
review surfaced several more issues, fixed here. All changes are designed
to roll out with no interruption to existing deployments — no forced
migration, no manual re-key, no boot failure.
Self-healing secret management (utils/secrets.ts):
- JWT_SECRET / ENCRYPTION_KEY auto-generate to strong values on first boot
and persist to the app_data volume (0600, outside the DB) when not set
to a strong value in the environment. Replaces the prior behavior of
running on the public repo defaults.
- decrypt() tries the current key plus every legacy/default key, so
existing device credentials keep decrypting after a key change; a
startup sweep re-encrypts stale rows forward (db/reencryptCredentials.ts).
- verifyToken() accepts the current signing secret plus prior *strong*
secrets (never the public defaults), so rotation doesn't invalidate live
sessions; sessions signed with a default secret require one re-login.
- Dockerfile pre-creates /app/data as the app user so the non-root
process can persist to a fresh named volume; app_data volume added to
both compose files; compose secret defaults changed to empty (auto-gen).
Authorization / input:
- Default Socket.IO namespace now requires a valid JWT (was unauthenticated
and broadcasting fleet-activity events); frontend sends the token and
drops the shared socket on logout.
- Webhook delivery blocks SSRF to loopback/private/link-local/CGNAT ranges
and pins the request to the validated IP (WebhookService.ts).
- Rate limit added to /auth/totp/verify; global mutation backstop on /api.
- Password policy (>=10 chars, letter+digit) enforced on set/change/create.
- system.ts used verifyToken as middleware (always 500); now requireAuth.
Hardening:
- CSP + HSTS/nosniff/frame-deny headers in nginx, scoped to allow the
OpenStreetMap tile/geocoding hosts the maps use.
Docs/tests:
- README secret section rewritten for the self-healing model.
- New unit tests for secret resolution, persistence, rotation, and
legacy-key decryption (97 tests pass).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Socket.IO /terminal namespace validated the JWT but discarded the
role, so a read-only viewer (403'd on every REST write) could connect
and open an interactive SSH shell on any managed device using the
device's stored router-admin credentials — a full bypass of the role
model (reported CVSS 9.6).
- Gate the namespace handshake by role: only admin/operator may connect;
viewers are rejected with "Console access denied for this role".
- Re-check the role at the start handler (defense in depth) and attach
the verified payload to socket.data.
- Audit-log terminal session opens (and rate-limit hits) to audit_log.
- Rate-limit shell starts per user (5 / 60s, in-memory sliding window).
- Frontend: gate the Terminal button (DeviceDetailPage) and dashboard
"Open terminal" quick action behind canWrite so viewers don't see it.
Validated on the local docker stack: viewer rejected from /terminal,
admin still connects; viewer remains 403 on POST /api/devices.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Guest WiFi (Hotspot): one-click captive-portal wizard that creates the guest
SSID (virtual AP per radio), VLAN/bridge segregation + L3 VLAN interface, pool,
DHCP, portal, rate-limited guest profile, and masquerade; vouchers with limits +
printable sheets; live guest sessions; walled garden.
Network intelligence: per-client device fingerprinting (vendor/hostname →
category) with a persisted per-client override on the detail page; rogue/neighbor
AP detection from stored AP scans (evil-twin flagging); Mist-style anomaly
insights (client-count/CPU vs same-hour 14-day baseline + error bursts) in the
Operations engine.
Firmware orchestration: staged wave-based RouterOS rollouts (canary first) with
pre-upgrade backup, reboot-ride-out, version verification, halt-on-failure, and
scheduling; live per-device progress + cancel.
Platform & automation: scoped API tokens (mtm_, read/write, hashed at rest,
Bearer auth in requireAuth); outbound webhooks (12 events, HMAC-signed) fired
from the alert pipeline + firmware orchestrator; scheduled email reports.
api-ssl: RouterOSClient negotiates TLS on port 8729 and accepts RouterOS's
default self-signed certs (rejectUnauthorized:false), so SSL-secured API devices
connect without manual cert trust. Fixes#57.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Global search:
- Clicking/Enter on a client result now opens that client's detail page
(/clients/<mac>) instead of dumping to the clients list.
Operations dashboard (new backend route /api/operations):
- GET /insights: server-side attention engine (offline, updates, CPU/memory
pressure, missing backups, recent outages, weak WiFi clients, co-channel
overlap, high TX retries, error events) + per-device capacity + merged
activity feed (config changes, audit actions, alerts; login noise filtered).
- POST /backup-all and /sync-all: fleet-wide actions over online devices.
- Quick Actions now execute: Run discovery, Backup all, Sync config, and
Open terminal (device picker -> live SSH TerminalModal); inline status.
- New cards: Capacity & health meters, Security posture rollup (reuses
per-device posture, cached), Recent activity feed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Configuration History:
- Per-device snapshots from the full RouterOS /export (.rsc) — config-only, so
counters/operational state no longer cause phantom changes; deduped by content hash
- New Config History tab: snapshot timeline, readable line diff, on-demand Capture
(with honest 'no changes' feedback), one-click rollback via the linked .rsc backup
- Snapshot and its backup are one artifact: badged 'Config Snapshot' on the Backups
page, FK ON DELETE CASCADE keeps them in lockstep; delete is gated by a warning modal
- New config_drift alert event (off by default); periodic snapshot poll with
configurable cadence/retention (config_snapshot_interval_min/_retention)
- New endpoints under /api/config-history (list/get/diff/capture/rollback/delete)
Scheduled Backups:
- Replaced the raw cron textbox in Settings with Frequency (Daily/Weekly/Monthly)
+ time + day pickers; cron is generated/parsed behind the scenes
- Fixed cronMatchesNow to evaluate day-of-month/month/day-of-week (it previously
only checked minute/hour, so weekly/monthly schedules ran every day)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend: GET /api/system/version-check fetches frontend/package.json from
the main branch on GitHub, caches result in app_settings for 24 hours,
and compares semver against the running version. Fails silently (no banner)
for air-gapped or firewalled deployments.
Frontend: dismissible blue update banner in SecurityBanners (rendered
between TopBar and main content) — only visible when a newer version is
detected. Shows target version and docker compose update command.
* fix(security): default-secret banners, login rate limiting, real-IP passthrough
- app.set('trust proxy', 1) so req.ip resolves to real client IP behind nginx
- loginRateLimit() middleware on POST /api/auth/login; limits read from
app_settings (login_rate_limit_max / login_rate_limit_window_sec) with
60-second cache; falls back to 10/60s when Redis/DB unavailable
- GET /api/auth/security-status returns warnings for default JWT_SECRET,
ENCRYPTION_KEY, and admin/admin credential — non-breaking for dev
- SecurityBanners component in AppLayout: yellow banner for default secrets,
red banner + change-password link for default admin password
- Login rate limit settings exposed in Settings > General tab
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(security): rate-limit /security-status, suppress CodeQL false positive on /login
- Add allMethods option to rateLimitRedis so it can apply to GET routes
- Apply rateLimitRedis (10 req/60s per user) to GET /auth/security-status
to prevent bcrypt-based CPU exhaustion by authenticated users
- Add lgtm suppression comment on POST /login — CodeQL does not recognize
our custom loginRateLimit() middleware; rate limiting is in place
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Denis Evers <git@evers.sh>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* devices: hide discovered neighbors that are already managed
Discovered Devices showed every unresolved MikroTik neighbor, even when
the neighbor was obviously the same device a managed node is reporting
through a different link / MAC / alias. Since we already know every
managed device interface MAC, primary IP, and system identity, we can
flag those rows.
- /api/devices/discovered now annotates each row with duplicate_of_device_id
and duplicate_of_device_name when the neighbor matches a managed device
by interface MAC, primary IP, or system identity.
- Add a Hide duplicates of managed devices toggle (default on) above the
table, with a count of how many matches were found. When enabled, the
list hides matched rows; when disabled, matches stay visible but are
dimmed and tagged Already managed so the origin is clear.
- Suppress the Add to Manager action for rows that are already a
managed device, since adding them would create a second record for the
same hardware.
Co-authored-by: Denis <denis-ev@users.noreply.github.com>
* devices: safer duplicate handling, try-all safeguards, and timezone/firmware fixes
Co-authored-by: Denis <denis-ev@users.noreply.github.com>
* Non-blocking follow-ups: parsePort util, tests, bulk-add queue, preset rate limit
Co-authored-by: Denis <denis-ev@users.noreply.github.com>
* fix(security): tighten CORS, Redis rate limits, and preset access controls
Require explicit CORS origins in production, align Socket.IO CORS with HTTP,
default Docker Compose to local HTTPS origins, and remove the in-memory
mutation limiter in favor of Redis. Also add operator-use flags for credential
presets in the UI and harden bulk-add job access and client error messages.
Co-authored-by: Denis <denis-ev@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Squash-merge of PR #28 from @denis-ev. All three review blockers addressed: encryption key management documented, string truncation replaced with skip+warn validation, and undefined return on duplicate merge fixed with proper 500. Local test stack verified against cloned prod DB — migrations, new features, and backend all clean.
2GT-RichandClaude Opus 5
