From 636ed5228b46d3bb19ac03e9d3e4e5f5a1040fc5 Mon Sep 17 00:00:00 2001 From: gti Date: Tue, 22 Sep 2026 15:08:26 -0300 Subject: [PATCH] docs: add control panel v1 implementation plan --- .../2026-09-22-gti-fiscal-control-panel-v1.md | 198 ++++++++++++++++++ 1 file changed, 198 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-22-gti-fiscal-control-panel-v1.md diff --git a/docs/superpowers/plans/2026-09-22-gti-fiscal-control-panel-v1.md b/docs/superpowers/plans/2026-09-22-gti-fiscal-control-panel-v1.md new file mode 100644 index 0000000..7dcfcee --- /dev/null +++ b/docs/superpowers/plans/2026-09-22-gti-fiscal-control-panel-v1.md @@ -0,0 +1,198 @@ +# GTI Fiscal Control Panel V1 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a tested FastAPI/Jinja2 control panel that identifies every Agent installation, exposes deterministic health and coverage, centralizes redacted events, and supports safe authenticated operations. + +**Architecture:** Keep the existing Hub as the only application boundary. Add focused SQLAlchemy models and query/service modules for installations, snapshots, events, human sessions, and audits. Keep Agent Bearer authentication separate from human cookie sessions, and render the operator UI with Jinja2, CSS, and small JavaScript. + +**Tech Stack:** FastAPI, SQLAlchemy 2, Alembic, PostgreSQL, Jinja2, standard-library PBKDF2-HMAC password hashing, HttpOnly cookie sessions, server-rendered HTML, CSS, vanilla JavaScript. + +## Global Constraints + +- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` remain unchanged. +- No fiscal decision logic changes, production deployment, real-client installation, raw log upload, SQL execution, shell execution, or remote Windows restart. +- Agent authentication remains Bearer/token based; human authentication uses a separate expiring cookie session. +- `installation_id` is a persistent UUID and IP is never a primary identity. +- `SAFE_BLOCKED` does not independently make an Agent degraded or failed. +- All human mutations require authentication, CSRF validation, explicit confirmation, and an audit event. +- Dashboard queries use snapshots and bounded aggregates; list endpoints are paginated with bounded page sizes. +- Existing `tests`, `central/tests`, and `agent/tests` must remain passing. + +--- + +## File Map + +### Foundation + +- Create `central/alembic/versions/0017_control_plane_foundation.py`: installation, snapshot, event, admin/session/audit tables and indexes. +- Modify `central/app/models.py`: focused models for the new tables. +- Modify `central/app/config.py`: heartbeat/offline, retention, session, cookie, and version settings. +- Create `central/app/installation_identity.py`: UUID generation, masked CNPJ/path, metadata allowlist, health/version functions. +- Create `central/app/events.py`: event type validation, redaction, size bounds, ingestion and retention. +- Create `central/app/control_plane.py`: snapshot aggregation, summary queries, filters, and pagination services. +- Create `central/app/human_auth.py`: password hashing, admin lookup, session and CSRF helpers. +- Create `central/app/audit.py`: immutable human action audit helper. +- Modify `central/app/api.py`: enriched Agent registration/heartbeat and Agent event/snapshot ingestion routes; preserve existing routes. +- Modify `central/app/main.py`: mount foundation/control-plane routes and templates/static files. +- Modify `central/app/auth.py`: registration lookup by installation identity without weakening Bearer auth. +- Modify `agent/src/gti_fiscal_agent/state.py`: persist installation UUID and local registration metadata. +- Modify `agent/src/gti_fiscal_agent/client.py`: registration, enriched heartbeat, event sending methods. +- Modify `agent/scripts/run_agent_service.py`: load stable installation identity and send structured lifecycle/heartbeat metadata. +- Create `central/scripts/create_admin.py`: interactive secure admin creation CLI. +- Create `central/scripts/maintain_events.py`: retention maintenance command. + +### APIs And Web + +- Create `central/app/control_api.py`: human-authenticated JSON dashboard, agents, blockers, jobs, and events endpoints. +- Create `central/app/web.py`: login/logout, dashboard/list/detail pages, safe action form handlers, CSRF and confirmation checks. +- Create `central/templates/base.html`, `login.html`, `dashboard.html`, `agents.html`, `agent_detail.html`, `blockers.html`, `jobs.html`, `events.html`. +- Create `central/static/control-panel.css`, `central/static/control-panel.js`. +- Create `central/tests/test_control_plane.py`, `test_human_auth.py`, `test_control_api.py`, `test_web_panel.py`, `test_event_security.py`. +- Create `central/scripts/generate_control_panel_fixture.py`: explicit non-production 100-Agent fixture generator. +- Create `central/tests/test_control_panel_fixture.py`: fixture distribution and bounded query checks. +- Create `docs/CONTROL_PANEL_V1.md`: architecture, auth, identity, health, events, retention, actions, endpoints, screens, operations. +- Modify `docs/AI_PROJECT_STATE.md`: record panel readiness and security invariants. + +## Tasks + +### Task 1: Add Persistent Control-Plane Schema + +**Files:** migration and model files above. + +**Interfaces:** Models expose `Installation`, `AgentSnapshot`, `AgentEvent`, `AdminUser`, `WebSession`, and `AuditEvent`; later services use `installation_id`, indexed event timestamps, snapshot counts, and session hashes. + +- [ ] Write model tests asserting UUID uniqueness, one installation per Agent, snapshot upsert shape, event indexes, admin/session uniqueness, and audit append-only fields. +- [ ] Add the models with nullable metadata for backward-compatible existing Agents. Store `installation_id` as a 36-character UUID string with a unique index; store secrets only as hashes. +- [ ] Add migration `0017_control_plane_foundation` with foreign keys to `tenants`, `agents`, `jobs`, and `blocked_products` where applicable. Add indexes on tenant/installation, health/last_seen, snapshot update time, event installation/time/severity/type, job agent/status/time, and blocker tenant/type/resolution. +- [ ] Run `python -m pytest central/tests/test_control_plane.py -q` and apply migration only to the configured development PostgreSQL database when available. +- [ ] Commit `feat: add control plane foundation schema`. + +### Task 2: Implement Installation Identity And Agent Metadata + +**Files:** identity service, existing Agent auth/API, Agent state/client/service, tests. + +**Interfaces:** `ensure_installation_id(state_path: str) -> str`, `sanitize_registration(payload: dict) -> dict`, `mask_cnpj(value: str | None) -> str | None`, `sanitize_database_path(value: str | None) -> str | None`, and `agent_registration(payload, bearer) -> dict`. + +- [ ] Write failing tests proving the same local state path returns the same UUID, a new explicit installation creates a new UUID, hostname/IP changes do not change identity, and registration never stores password/token/API-key/connection-string values. +- [ ] Add an atomic local `installation_id` record to AgentState and include it in registration/heartbeat payloads. Keep the existing `agent_id` field separate. +- [ ] Extend registration/heartbeat metadata with display name, customer code, notes, OS/version, machine fingerprint, masked company CNPJ, UF/CRT, Firebird version/ODS, sanitized database identifier, capabilities, fiscal engine, datasets, started time, and last job data. +- [ ] Upsert the Hub installation record by `(agent_id, installation_id)` and reject conflicting installation IDs rather than silently reassigning an existing installation. +- [ ] Run `python -m pytest central/tests/test_control_plane.py agent/tests/test_state.py agent/tests/test_run_agent_service.py -q`. +- [ ] Commit `feat: persist agent installation identity and metadata`. + +### Task 3: Implement Health, Version Drift, And Snapshots + +**Files:** `central/app/installation_identity.py`, `central/app/control_plane.py`, config, heartbeat/job completion routes, tests. + +**Interfaces:** `compute_agent_health(snapshot, now) -> str`, `compute_version_drift(agent_version, server_version) -> str`, `upsert_agent_snapshot(session, installation, payload) -> AgentSnapshot`, `dashboard_summary(session, filters) -> dict`, and `paginate(query, page, page_size) -> dict`. + +- [ ] Write tests for `UNKNOWN`, recent healthy `ONLINE`, recent technical failure `DEGRADED`, persistent technical failure `FAILED`, stale `OFFLINE`, and safe-blocked-only `ONLINE`. +- [ ] Implement configurable `AGENT_HEARTBEAT_INTERVAL_SECONDS`, `AGENT_OFFLINE_AFTER_SECONDS` with a three-interval default, `CURRENT_SERVER_VERSION`, and `AGENT_EVENT_RETENTION_DAYS=90`. +- [ ] Persist product counts, coverage, failed products, blocker aggregates, last/last-successful job, heartbeat, health, and version drift in one upserted snapshot. +- [ ] Add dashboard aggregate queries that only read snapshot rows plus bounded grouped counts. Normalize datetimes as UTC for SQLite and PostgreSQL tests. +- [ ] Run health/snapshot tests and existing job tests; verify `SAFE_BLOCKED` does not degrade health. +- [ ] Commit `feat: add deterministic agent health and snapshots`. + +### Task 4: Add Structured Events, Redaction, Retention, And Audit + +**Files:** `central/app/events.py`, `central/app/audit.py`, models/migration if needed, API routes, maintenance CLI, tests. + +**Interfaces:** `sanitize_event(message, context) -> tuple[str, dict]`, `record_event(session, *, installation_id, severity, event_type, message, context, job_id=None, product_code=None, blocker_id=None) -> AgentEvent`, `purge_expired_events(session, now, retention_days) -> int`, and `record_audit(session, actor, action, target, context) -> AuditEvent`. + +- [ ] Write tests covering case-insensitive redaction of passwords, tokens, API keys, connection strings and secret env names in both messages and nested JSON; assert event message/context maximums. +- [ ] Implement allowlisted event types and severities from the mission. Store structured context only, truncate safely, and never persist raw log files. +- [ ] Record Agent lifecycle, heartbeat, preflight, job, blocker, dataset/schema, connection, retry, version, and write-audit events at the relevant existing route boundaries. +- [ ] Implement retention deletion for non-audit events older than `AGENT_EVENT_RETENTION_DAYS`; preserve audit events, blocker history, write audit, and critical events. +- [ ] Add `python -m central.scripts.maintain_events --dry-run` and a write mode that only deletes eligible events. +- [ ] Run event security and retention tests, then commit `feat: add redacted operational events and retention`. + +### Task 5: Implement Human Admin Authentication And Session Security + +**Files:** `central/app/human_auth.py`, `central/app/web.py`, config/main, CLI, tests. + +**Interfaces:** `hash_password(password) -> str`, `verify_password(password, encoded_hash) -> bool`, `create_admin(session, username, password) -> AdminUser`, `create_web_session(session, admin, user_agent, ip) -> tuple[str, str]`, `require_human(request) -> AdminUser`, and `require_csrf(request, submitted_token) -> None`. + +- [ ] Write tests for PBKDF2 password storage, wrong-password rejection, inactive admin rejection, session expiry, cookie flags, session rotation, logout invalidation, and CSRF mismatch/replay rejection. +- [ ] Add PBKDF2-HMAC-SHA256 hashes with random salts and constant-time verification using only the standard library. Store only session token hashes and CSRF token hashes server-side. +- [ ] Add `gti_admin_session` HttpOnly cookie with `Secure` according to production config, `SameSite=Lax`, bounded max age, and no Agent token reuse. +- [ ] Add `python -m central.scripts.create_admin` using `getpass`, refusing empty/mismatched passwords and duplicate usernames. +- [ ] Run auth tests and commit `feat: add separate human admin sessions`. + +### Task 6: Add Control-Plane APIs And Safe Actions + +**Files:** `central/app/control_api.py`, API integration, audit/events, tests. + +**Interfaces:** + +```text +GET /v1/dashboard/summary +GET /v1/agents +GET /v1/agents/{agent_id} +GET /v1/agents/{agent_id}/health +GET /v1/agents/{agent_id}/coverage +GET /v1/agents/{agent_id}/blockers +GET /v1/agents/{agent_id}/jobs +GET /v1/agents/{agent_id}/events +GET /v1/blockers/global +GET /v1/jobs/global +POST /v1/agents/{agent_id}/actions/reprocess +POST /v1/blockers/{blocker_id}/resolve-human +``` + +- [ ] Write tests for authentication, client/health/version/last-seen/blocker/severity/date filters, bounded pagination, empty pages, and cross-tenant access denial. +- [ ] Implement query services with `page`, `page_size`, `total`, `items`, max page size 100, explicit sort allowlists, and no relationship N+1 queries. +- [ ] Implement safe actions only: create idempotent reprocess jobs, resolve only with explicit resolution context, and record actor/action/target/context audit events. +- [ ] Reject requests that attempt Firebird writes, AI flags, credentials, database paths, CRT changes, SQL, shell, restart, or arbitrary action names. +- [ ] Run API tests and commit `feat: expose authenticated control plane APIs`. + +### Task 7: Build Server-Rendered Operational UI + +**Files:** templates, CSS/JS, `central/app/web.py`, main, tests. + +**Interfaces:** browser routes `/login`, `/logout`, `/`, `/agents`, `/agents/{agent_id}`, `/blockers`, `/jobs`, and `/events`; all mutations use POST forms with hidden CSRF and a visible confirmation field. + +- [ ] Write template tests asserting dashboard cards, client/display-name/hostname identity, health indicator, blocker/job/event pagination, technical IDs only in detail views, and no secret values in rendered HTML. +- [ ] Implement dashboard cards for Agent health, product coverage, technical failures, recent failed jobs, stale heartbeats, version drift, and top blocker types. +- [ ] Implement responsive Agents/Servers table with search/sort/filter/page controls and links to detail tabs: overview, fiscal, blockers, jobs, events, capabilities, technical info. +- [ ] Implement global blockers/jobs/event pages and per-installation timelines with bounded pagination and filters. +- [ ] Add plain CSS responsive layout, recognizable health badges, compact tables, confirmation dialogs, and minimal JavaScript for filters/confirmations only. +- [ ] Run `python -m pytest central/tests/test_web_panel.py -q` and commit `feat: add server-rendered control panel`. + +### Task 8: Add Fixture Generator And Performance Checks + +**Files:** fixture script, tests, optional fixture helpers. + +**Interfaces:** `generate_fixture(session, agent_count=100, seed=...) -> dict`; must be explicit and never run during app startup or migration. + +- [ ] Write tests asserting exactly 100 installations with distributed `ONLINE`, `DEGRADED`, `OFFLINE`, and `FAILED` states, blockers, jobs, and events. +- [ ] Implement deterministic fixture generation behind an explicit `--database-url` and `--confirm-non-production` guard; refuse production APP_ENV and refuse empty confirmation. +- [ ] Add query-count/performance checks that dashboard uses snapshot rows and event/job pages issue bounded queries for 100 Agents and large event counts. +- [ ] Run fixture tests without seeding the configured production database and commit `test: add control panel fixture and query checks`. + +### Task 9: Document, Run Full Verification, And Consolidate + +**Files:** `docs/CONTROL_PANEL_V1.md`, `docs/AI_PROJECT_STATE.md`, final tests. + +- [ ] Document architecture, identity lifecycle, metadata allowlist, health thresholds, event taxonomy/redaction, retention command, auth/session/CSRF, safe actions, endpoints, screens, and fixture usage. +- [ ] Update project state with control-plane readiness and explicit `AI_WRITE_ALLOWED=false`, `FIREBIRD_AI_WRITES=0`, `DEPLOY_PERFORMED=false`. +- [ ] Run `python -m pytest tests -q`. +- [ ] Run `python -m pytest central/tests -q`. +- [ ] Run `python -m pytest agent/tests -q`. +- [ ] Run `git diff --check`, inspect `git status`, and verify no secrets or generated fixture database are tracked. +- [ ] Commit `docs: document control panel v1 operations`. + +## Verification Matrix + +| Requirement | Plan task | Evidence | +|---|---:|---| +| Persistent installation identity and metadata | 1-2 | identity tests, registration tests, migration | +| Deterministic health/offline/version drift | 3 | health/snapshot tests | +| Coverage and blocker summaries | 3, 6-7 | snapshot/API/template tests | +| Structured redacted events | 4 | redaction, size, ingestion tests | +| Retention and audit preservation | 4 | maintenance tests | +| Human auth, cookie session, CSRF | 5 | auth/session tests | +| Paginated filtered APIs and safe actions | 6 | API and audit tests | +| Dashboard and operational screens | 7 | template and route tests | +| 100-Agent simulation/performance | 8 | fixture/query checks | +| Documentation and regression | 9 | docs and three suites |