Files
MIKROTIK-RMM/docker-compose.yml
T
2GT-RichandClaude Opus 5 745a7cda93 fix(poller): drain stale backlogs, and stop reporting health that isn't (v0.23.19-beta)
v0.23.17 stopped the poller creating an unbounded backlog but did nothing about
the one already there. A reporter with sixty devices upgraded and still had
752,318 jobs queued, nine stale devices, and an endpoint cheerfully reporting
1.59x headroom. Fair to call that not fixed.

Three faults, all mine.

**Retention never touched pending work.** cleanupJobHistory cleaned 'completed'
and 'failed' only. The queue of outstanding polls — the part the operator
actually feels — was out of scope, so an upgrade left it draining for days.

Stale pending jobs are now swept, at startup and every five minutes. The rule is
that a periodic poll which has waited longer than its own cadence is worthless:
running it yields a reading the next cycle would have taken anyway, while
occupying a worker the live fleet is waiting on. Discarding queued work is safe
precisely because it is periodic — the schedule re-enqueues within one interval.

**Deduplication failed in the case it existed for.** The in-flight claim was a
Redis key with a 135-second TTL. Under a large backlog a job waits hours, so the
claim lapsed long before it ran and the scheduler re-enqueued every cycle: the
guard worked only on a queue that was already short.

Replaced with a deterministic job id, which cannot lapse. While a job for a
device and kind exists in any state, adding it again is a no-op. That requires
finished jobs to be removed immediately or the id would stay taken and the
device would never be polled again — which costs nothing now that
device_poll_stats records every attempt, duration and error.

**The headroom metric lied.** It describes flow and is blind to accumulated
work, so it read "healthy" beside three-quarters of a million queued jobs. It
now reports backlog and drain ETA alongside, plus a single status verdict that
cannot say ok while a backlog exists. Verified by injecting 15,125 jobs: status
went to 'draining' with headroom still at 85x, which is exactly the case the old
metric got wrong.

Also added, because the reporter asked and was right to: an operator-triggered
drain (POST /api/system/poller/drain), and a Polling tab in Settings showing
queue depth, headroom, backlog, per-device staleness and the drain control. It
was all buried in an API response before. A dashboard insight fires when the
poller falls behind, so a starved poller announces itself rather than presenting
as a fleet of broken devices.

Two corrections found by testing rather than reasoning:

  * BullMQ rejects ':' in a custom job id. `fast:2` threw on every enqueue and
    killed the scheduler outright — three of four devices went unpolled for
    eight minutes before it was caught. Hyphens now, and the dedup is exercised
    directly: 500 enqueues of one device yield one queued job.
  * Redis maxmemory-policy moves from volatile-lru to noeviction. I had reasoned
    that only TTL-bearing keys are evicted and BullMQ job hashes carry none —
    true, but its *control* keys do. Job locks and stalled-check markers both
    expire, and evicting one causes duplicate processing or a stalled-job storm.
    BullMQ warns about this on connect and was right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-01 08:07:49 -07:00

146 lines
4.3 KiB
YAML

version: '3.8'
services:
nginx:
build:
context: .
dockerfile: nginx/Dockerfile
ports:
- "${HTTP_PORT:-80}:80"
- "${HTTPS_PORT:-443}:443"
volumes:
- certs_data:/certs
depends_on:
backend:
condition: service_healthy
restart: unless-stopped
networks:
- app-network
backend:
build:
context: ./backend
dockerfile: Dockerfile
environment:
NODE_ENV: production
TZ: ${TZ:-UTC}
DATABASE_URL: postgresql://mikrotik:${DB_PASSWORD:-mikrotik_secure_pw}@postgres:5432/mikrotik_manager
REDIS_URL: redis://redis:6379
INFLUXDB_URL: http://influxdb:8086
INFLUXDB_TOKEN: ${INFLUXDB_TOKEN:-mytoken123456789}
INFLUXDB_ORG: ${INFLUXDB_ORG:-mikrotik-manager}
INFLUXDB_BUCKET: ${INFLUXDB_BUCKET:-metrics}
# Leave unset to auto-generate & persist strong secrets on first boot
# (stored in the app_data volume). Set them to pin your own values.
JWT_SECRET: ${JWT_SECRET:-}
ENCRYPTION_KEY: ${ENCRYPTION_KEY:-}
CORS_ORIGIN: ${CORS_ORIGIN:-https://localhost,http://localhost,https://127.0.0.1,http://127.0.0.1}
PORT: 3001
NETFLOW_BIND_PORT: 2055
ports:
# NetFlow/IPFIX collector — UDP goes straight to the backend (nginx not involved).
# Host port must match the netflow_collector_port app setting (default 2055).
- "${NETFLOW_PORT:-2055}:2055/udp"
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
influxdb:
condition: service_healthy
volumes:
- backups_data:/app/backups
- certs_data:/certs
- app_data:/app/data
cap_add:
- NET_RAW
restart: unless-stopped
networks:
- app-network
healthcheck:
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3001/api/health"]
interval: 15s
timeout: 10s
retries: 5
start_period: 30s
postgres:
image: postgres:15-alpine
environment:
TZ: ${TZ:-UTC}
POSTGRES_DB: mikrotik_manager
POSTGRES_USER: mikrotik
POSTGRES_PASSWORD: ${DB_PASSWORD:-mikrotik_secure_pw}
volumes:
- postgres_data:/var/lib/postgresql/data
restart: unless-stopped
networks:
- app-network
healthcheck:
test: ["CMD-SHELL", "pg_isready -U mikrotik -d mikrotik_manager"]
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
redis:
image: redis:7-alpine
# A memory ceiling so a runaway producer can never take the host down, with
# eviction disabled.
#
# volatile-lru looks tempting — it would evict only keys with an expiry, and
# BullMQ's job hashes have none. But its *control* keys do: job locks and
# per-queue stalled-check markers both carry TTLs, and losing one causes
# duplicate processing or a stalled-job storm. BullMQ warns about this on
# connect, and it is right. noeviction means Redis refuses writes when full
# rather than quietly corrupting queue state: the host stays up, and the
# failure is loud. Retention limits and the stale-pending sweep are what keep
# us away from the ceiling in the first place.
command: >
redis-server --appendonly yes
--maxmemory ${REDIS_MAXMEMORY:-512mb}
--maxmemory-policy noeviction
volumes:
- redis_data:/data
restart: unless-stopped
networks:
- app-network
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
influxdb:
image: influxdb:2.7
environment:
DOCKER_INFLUXDB_INIT_MODE: setup
DOCKER_INFLUXDB_INIT_USERNAME: admin
DOCKER_INFLUXDB_INIT_PASSWORD: ${INFLUXDB_ADMIN_PASSWORD:-admin_password_123}
DOCKER_INFLUXDB_INIT_ORG: ${INFLUXDB_ORG:-mikrotik-manager}
DOCKER_INFLUXDB_INIT_BUCKET: ${INFLUXDB_BUCKET:-metrics}
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: ${INFLUXDB_TOKEN:-mytoken123456789}
volumes:
- influxdb_data:/var/lib/influxdb2
restart: unless-stopped
networks:
- app-network
healthcheck:
test: ["CMD", "influx", "ping"]
interval: 15s
timeout: 10s
retries: 10
start_period: 30s
volumes:
postgres_data:
redis_data:
influxdb_data:
backups_data:
certs_data:
app_data:
networks:
app-network:
driver: bridge