33ca7cc651c21e17b6e4b9c84dbd6517aa32722a
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  2GT-RichandClaude Opus 5
|
745a7cda93 |
fix(poller): drain stale backlogs, and stop reporting health that isn't (v0.23.19-beta)
v0.23.17 stopped the poller creating an unbounded backlog but did nothing about
the one already there. A reporter with sixty devices upgraded and still had
752,318 jobs queued, nine stale devices, and an endpoint cheerfully reporting
1.59x headroom. Fair to call that not fixed.
Three faults, all mine.
**Retention never touched pending work.** cleanupJobHistory cleaned 'completed'
and 'failed' only. The queue of outstanding polls — the part the operator
actually feels — was out of scope, so an upgrade left it draining for days.
Stale pending jobs are now swept, at startup and every five minutes. The rule is
that a periodic poll which has waited longer than its own cadence is worthless:
running it yields a reading the next cycle would have taken anyway, while
occupying a worker the live fleet is waiting on. Discarding queued work is safe
precisely because it is periodic — the schedule re-enqueues within one interval.
**Deduplication failed in the case it existed for.** The in-flight claim was a
Redis key with a 135-second TTL. Under a large backlog a job waits hours, so the
claim lapsed long before it ran and the scheduler re-enqueued every cycle: the
guard worked only on a queue that was already short.
Replaced with a deterministic job id, which cannot lapse. While a job for a
device and kind exists in any state, adding it again is a no-op. That requires
finished jobs to be removed immediately or the id would stay taken and the
device would never be polled again — which costs nothing now that
device_poll_stats records every attempt, duration and error.
**The headroom metric lied.** It describes flow and is blind to accumulated
work, so it read "healthy" beside three-quarters of a million queued jobs. It
now reports backlog and drain ETA alongside, plus a single status verdict that
cannot say ok while a backlog exists. Verified by injecting 15,125 jobs: status
went to 'draining' with headroom still at 85x, which is exactly the case the old
metric got wrong.
Also added, because the reporter asked and was right to: an operator-triggered
drain (POST /api/system/poller/drain), and a Polling tab in Settings showing
queue depth, headroom, backlog, per-device staleness and the drain control. It
was all buried in an API response before. A dashboard insight fires when the
poller falls behind, so a starved poller announces itself rather than presenting
as a fleet of broken devices.
Two corrections found by testing rather than reasoning:
* BullMQ rejects ':' in a custom job id. `fast:2` threw on every enqueue and
killed the scheduler outright — three of four devices went unpolled for
eight minutes before it was caught. Hyphens now, and the dedup is exercised
directly: 500 enqueues of one device yield one queued job.
* Redis maxmemory-policy moves from volatile-lru to noeviction. I had reasoned
that only TTL-bearing keys are evicted and BullMQ job hashes carry none —
true, but its *control* keys do. Job locks and stalled-check markers both
expire, and evicting one causes duplicate processing or a stalled-job storm.
BullMQ warns about this on connect and was right.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
2GT-RichandClaude Opus 5
|
37887d38c2 |
feat: RouterBOOT in staged rollouts, and snoozeable insights (v0.23.10-beta)
Closes #113, #102. #113 — RouterBOOT as a follow-on rollout step Firmware rollouts upgraded RouterOS and stopped, leaving the bootloader to be done by hand on every device — the exact task staged rollouts exist to remove. The data was already collected and the collector already had the two methods; the orchestrator simply never called them. An optional "Then RouterBOOT" step now runs per device after the RouterOS upgrade has been verified, rides out the second reboot, and confirms the firmware version actually moved. Ordering is deliberate: the bootloader ships inside the RouterOS package, so upgrading it first would apply the old one. It runs only once the OS half is confirmed good, and off by default, because it is a second flash and a second reboot per device. A device with nothing pending is a success rather than a failure — most of a fleet is already current, and treating "nothing to do" as an error would halt rollouts for no reason. When RouterBOOT does fail after a successful RouterOS upgrade the error says so explicitly, rather than reporting the device as simply failed and sending someone to re-run an upgrade it already has. #102 — snooze an item in "Things to handle" Insights are recomputed from live state on every request and have no id, so a dismissal is keyed on a fingerprint of category, target and title. That survives a reworded title without swallowing a different device's version of the same issue. Dismissals expire, defaulting to 24 hours. "Handled" and "not now" look identical to the code, and an item that is genuinely resolved stops being generated anyway — so a permanent dismissal could only ever hide something still true. Expired rows are cleared on read, so a lapse needs no scheduled job, and the panel reports how many items are currently hidden rather than quietly shrinking. Verified against the fleet: snoozing took the list from 8 items to 7 with dismissedCount 1, and removing the dismissal restored it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
2GT-RichandClaude Opus 5
|
93ff1fbc53 |
feat(safety): Config Health audit, verified auto-revert, VLAN cache fixes (v0.20.0-beta)
Completes the lockout-prevention work: a standing audit for configurations
RouterOS accepts but that do not work, an empirical answer on /system history
undo, and corrections to the VLAN cache the editors are built on.
Config Health (new)
Eleven rules over a live snapshot, encoding MikroTik's documented Layer2
misconfigurations plus the latent lockout cases: IP or VLAN interface on a
bridge slave port, a bridged VLAN interface, a bond slave that is also a
bridge port, MTU above L2MTU, a multi-VLAN entry with untagged ports,
bridges competing for hardware offload, a port in no VLAN on a filtering
bridge, a PVID that frame-type=admit-only-vlan-tagged makes inert, a VLAN
interface whose VLAN the bridge is not tagged in, duplicate addresses, and
management that survives only on a dynamic entry. Each finding names the
consequence, the fix, how long it has been present, and the vendor doc.
Runs hourly, surfaces on the Security tab and the dashboard.
Calibrated against hardware: RouterOS 7 relocates an address off a bridge
slave port onto the bridge (actual-interface), so the stock defconf layout
works. Reporting that as critical would have fired on a default install; it
is informational unless RouterOS did not relocate it.
Change Guard: severed-connection handling
Proven on a CRS by deleting its own management address — the switch locked
itself out and self-restored unattended in ~3 minutes, twice.
That test exposed a real defect. withSafeApply treated any exception from
the apply step as "the change failed", disarming the guard and rethrowing.
But the most dangerous changes sever the connection carrying them, so the
call times out even though the change applied. The guard was stripping the
safety net off a device that had just gone silent, and reporting HTTP 500
"Read timeout" instead of telling the user the device was restoring itself.
Only luck saved it: the disarm needed the same dead connection.
Reachability now decides, not the exception. Unreachable leaves the revert
armed and returns auto_reverting; still reachable means a genuine failure,
disarmed so nothing reboots for nothing. committed_at is no longer stamped
on failed or reverted guards; resolved_at records completion instead.
/system history undo probe
probeUndoCapability tests the device rather than assuming. On 7.23.x over
the binary API: /system/history/print works, /system/history/undo does not
exist, /undo numbers= rejects the parameter, bare /undo works. Undo is
therefore untargeted, and cannot replace auto-revert since a dead-man switch
must fire after contact is lost. Binary restore stays the default.
VLAN cache correctness
- vlans keyed on (device_id, bridge, vlan_id); the old key collapsed the
same VID on two bridges
- vlan-ids ranges expanded instead of parseInt truncating "10-20" to 10
- both VLAN tables pruned, but only when the device returned an
authoritative list, so a failed read leaves the cache stale not empty
- bridge_vlan_entries.vlan_ids written and .tagged derived rather than
hardcoded false, which had left the port editor unable to see a trunk
- removeBridgeVlan matched by expanded VIDs; the ?vlan-ids= server filter
compares raw strings, so deleting one VID of a range silently no-opped
while reporting success. Multi-VLAN entries are refused in the route,
before the guard arms, since a mid-apply failure would cost a reboot
- interfaces.vlan_filtering promoted to a column
Membership is deliberately the STATIC tagged/untagged, never current-*.
These tables back the editors and updateBridgeVlan is remove-then-add, so
whatever is shown is written back; the effective view baked RouterOS's
dynamic PVID entries into saved configuration on every save. Analysis that
needs the effective view reads the device live. Extracted to pure functions
with tests, because that path was previously untestable inline.
Verified on four devices; 195 backend tests; all CI gates green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
2GT-RichandClaude Opus 4.8
|
9dfb8a5333 |
feat: Guest WiFi, network intelligence, firmware orchestration, platform automation, api-ssl (v0.16.3-beta)
Guest WiFi (Hotspot): one-click captive-portal wizard that creates the guest SSID (virtual AP per radio), VLAN/bridge segregation + L3 VLAN interface, pool, DHCP, portal, rate-limited guest profile, and masquerade; vouchers with limits + printable sheets; live guest sessions; walled garden. Network intelligence: per-client device fingerprinting (vendor/hostname → category) with a persisted per-client override on the detail page; rogue/neighbor AP detection from stored AP scans (evil-twin flagging); Mist-style anomaly insights (client-count/CPU vs same-hour 14-day baseline + error bursts) in the Operations engine. Firmware orchestration: staged wave-based RouterOS rollouts (canary first) with pre-upgrade backup, reboot-ride-out, version verification, halt-on-failure, and scheduling; live per-device progress + cancel. Platform & automation: scoped API tokens (mtm_, read/write, hashed at rest, Bearer auth in requireAuth); outbound webhooks (12 events, HMAC-signed) fired from the alert pipeline + firmware orchestrator; scheduled email reports. api-ssl: RouterOSClient negotiates TLS on port 8729 and accepts RouterOS's default self-signed certs (rejectUnauthorized:false), so SSL-secured API devices connect without manual cert trust. Fixes #57. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
2GT-RichandClaude Opus 4.8
|
6cfca387d8 |
feat(ops): operations insights engine + working quick actions; fix global search (v0.16.2-beta)
Global search: - Clicking/Enter on a client result now opens that client's detail page (/clients/<mac>) instead of dumping to the clients list. Operations dashboard (new backend route /api/operations): - GET /insights: server-side attention engine (offline, updates, CPU/memory pressure, missing backups, recent outages, weak WiFi clients, co-channel overlap, high TX retries, error events) + per-device capacity + merged activity feed (config changes, audit actions, alerts; login noise filtered). - POST /backup-all and /sync-all: fleet-wide actions over online devices. - Quick Actions now execute: Run discovery, Backup all, Sync config, and Open terminal (device picker -> live SSH TerminalModal); inline status. - New cards: Capacity & health meters, Security posture rollup (reuses per-device posture, cached), Recent activity feed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |