docs: add control panel v1 implementation plan
This commit is contained in:
gti
@@ -0,0 +1,198 @@
|
|||||||
|
# GTI Fiscal Control Panel V1 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Build a tested FastAPI/Jinja2 control panel that identifies every Agent installation, exposes deterministic health and coverage, centralizes redacted events, and supports safe authenticated operations.
|
||||||
|
|
||||||
|
**Architecture:** Keep the existing Hub as the only application boundary. Add focused SQLAlchemy models and query/service modules for installations, snapshots, events, human sessions, and audits. Keep Agent Bearer authentication separate from human cookie sessions, and render the operator UI with Jinja2, CSS, and small JavaScript.
|
||||||
|
|
||||||
|
**Tech Stack:** FastAPI, SQLAlchemy 2, Alembic, PostgreSQL, Jinja2, standard-library PBKDF2-HMAC password hashing, HttpOnly cookie sessions, server-rendered HTML, CSS, vanilla JavaScript.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` remain unchanged.
|
||||||
|
- No fiscal decision logic changes, production deployment, real-client installation, raw log upload, SQL execution, shell execution, or remote Windows restart.
|
||||||
|
- Agent authentication remains Bearer/token based; human authentication uses a separate expiring cookie session.
|
||||||
|
- `installation_id` is a persistent UUID and IP is never a primary identity.
|
||||||
|
- `SAFE_BLOCKED` does not independently make an Agent degraded or failed.
|
||||||
|
- All human mutations require authentication, CSRF validation, explicit confirmation, and an audit event.
|
||||||
|
- Dashboard queries use snapshots and bounded aggregates; list endpoints are paginated with bounded page sizes.
|
||||||
|
- Existing `tests`, `central/tests`, and `agent/tests` must remain passing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File Map
|
||||||
|
|
||||||
|
### Foundation
|
||||||
|
|
||||||
|
- Create `central/alembic/versions/0017_control_plane_foundation.py`: installation, snapshot, event, admin/session/audit tables and indexes.
|
||||||
|
- Modify `central/app/models.py`: focused models for the new tables.
|
||||||
|
- Modify `central/app/config.py`: heartbeat/offline, retention, session, cookie, and version settings.
|
||||||
|
- Create `central/app/installation_identity.py`: UUID generation, masked CNPJ/path, metadata allowlist, health/version functions.
|
||||||
|
- Create `central/app/events.py`: event type validation, redaction, size bounds, ingestion and retention.
|
||||||
|
- Create `central/app/control_plane.py`: snapshot aggregation, summary queries, filters, and pagination services.
|
||||||
|
- Create `central/app/human_auth.py`: password hashing, admin lookup, session and CSRF helpers.
|
||||||
|
- Create `central/app/audit.py`: immutable human action audit helper.
|
||||||
|
- Modify `central/app/api.py`: enriched Agent registration/heartbeat and Agent event/snapshot ingestion routes; preserve existing routes.
|
||||||
|
- Modify `central/app/main.py`: mount foundation/control-plane routes and templates/static files.
|
||||||
|
- Modify `central/app/auth.py`: registration lookup by installation identity without weakening Bearer auth.
|
||||||
|
- Modify `agent/src/gti_fiscal_agent/state.py`: persist installation UUID and local registration metadata.
|
||||||
|
- Modify `agent/src/gti_fiscal_agent/client.py`: registration, enriched heartbeat, event sending methods.
|
||||||
|
- Modify `agent/scripts/run_agent_service.py`: load stable installation identity and send structured lifecycle/heartbeat metadata.
|
||||||
|
- Create `central/scripts/create_admin.py`: interactive secure admin creation CLI.
|
||||||
|
- Create `central/scripts/maintain_events.py`: retention maintenance command.
|
||||||
|
|
||||||
|
### APIs And Web
|
||||||
|
|
||||||
|
- Create `central/app/control_api.py`: human-authenticated JSON dashboard, agents, blockers, jobs, and events endpoints.
|
||||||
|
- Create `central/app/web.py`: login/logout, dashboard/list/detail pages, safe action form handlers, CSRF and confirmation checks.
|
||||||
|
- Create `central/templates/base.html`, `login.html`, `dashboard.html`, `agents.html`, `agent_detail.html`, `blockers.html`, `jobs.html`, `events.html`.
|
||||||
|
- Create `central/static/control-panel.css`, `central/static/control-panel.js`.
|
||||||
|
- Create `central/tests/test_control_plane.py`, `test_human_auth.py`, `test_control_api.py`, `test_web_panel.py`, `test_event_security.py`.
|
||||||
|
- Create `central/scripts/generate_control_panel_fixture.py`: explicit non-production 100-Agent fixture generator.
|
||||||
|
- Create `central/tests/test_control_panel_fixture.py`: fixture distribution and bounded query checks.
|
||||||
|
- Create `docs/CONTROL_PANEL_V1.md`: architecture, auth, identity, health, events, retention, actions, endpoints, screens, operations.
|
||||||
|
- Modify `docs/AI_PROJECT_STATE.md`: record panel readiness and security invariants.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
### Task 1: Add Persistent Control-Plane Schema
|
||||||
|
|
||||||
|
**Files:** migration and model files above.
|
||||||
|
|
||||||
|
**Interfaces:** Models expose `Installation`, `AgentSnapshot`, `AgentEvent`, `AdminUser`, `WebSession`, and `AuditEvent`; later services use `installation_id`, indexed event timestamps, snapshot counts, and session hashes.
|
||||||
|
|
||||||
|
- [ ] Write model tests asserting UUID uniqueness, one installation per Agent, snapshot upsert shape, event indexes, admin/session uniqueness, and audit append-only fields.
|
||||||
|
- [ ] Add the models with nullable metadata for backward-compatible existing Agents. Store `installation_id` as a 36-character UUID string with a unique index; store secrets only as hashes.
|
||||||
|
- [ ] Add migration `0017_control_plane_foundation` with foreign keys to `tenants`, `agents`, `jobs`, and `blocked_products` where applicable. Add indexes on tenant/installation, health/last_seen, snapshot update time, event installation/time/severity/type, job agent/status/time, and blocker tenant/type/resolution.
|
||||||
|
- [ ] Run `python -m pytest central/tests/test_control_plane.py -q` and apply migration only to the configured development PostgreSQL database when available.
|
||||||
|
- [ ] Commit `feat: add control plane foundation schema`.
|
||||||
|
|
||||||
|
### Task 2: Implement Installation Identity And Agent Metadata
|
||||||
|
|
||||||
|
**Files:** identity service, existing Agent auth/API, Agent state/client/service, tests.
|
||||||
|
|
||||||
|
**Interfaces:** `ensure_installation_id(state_path: str) -> str`, `sanitize_registration(payload: dict) -> dict`, `mask_cnpj(value: str | None) -> str | None`, `sanitize_database_path(value: str | None) -> str | None`, and `agent_registration(payload, bearer) -> dict`.
|
||||||
|
|
||||||
|
- [ ] Write failing tests proving the same local state path returns the same UUID, a new explicit installation creates a new UUID, hostname/IP changes do not change identity, and registration never stores password/token/API-key/connection-string values.
|
||||||
|
- [ ] Add an atomic local `installation_id` record to AgentState and include it in registration/heartbeat payloads. Keep the existing `agent_id` field separate.
|
||||||
|
- [ ] Extend registration/heartbeat metadata with display name, customer code, notes, OS/version, machine fingerprint, masked company CNPJ, UF/CRT, Firebird version/ODS, sanitized database identifier, capabilities, fiscal engine, datasets, started time, and last job data.
|
||||||
|
- [ ] Upsert the Hub installation record by `(agent_id, installation_id)` and reject conflicting installation IDs rather than silently reassigning an existing installation.
|
||||||
|
- [ ] Run `python -m pytest central/tests/test_control_plane.py agent/tests/test_state.py agent/tests/test_run_agent_service.py -q`.
|
||||||
|
- [ ] Commit `feat: persist agent installation identity and metadata`.
|
||||||
|
|
||||||
|
### Task 3: Implement Health, Version Drift, And Snapshots
|
||||||
|
|
||||||
|
**Files:** `central/app/installation_identity.py`, `central/app/control_plane.py`, config, heartbeat/job completion routes, tests.
|
||||||
|
|
||||||
|
**Interfaces:** `compute_agent_health(snapshot, now) -> str`, `compute_version_drift(agent_version, server_version) -> str`, `upsert_agent_snapshot(session, installation, payload) -> AgentSnapshot`, `dashboard_summary(session, filters) -> dict`, and `paginate(query, page, page_size) -> dict`.
|
||||||
|
|
||||||
|
- [ ] Write tests for `UNKNOWN`, recent healthy `ONLINE`, recent technical failure `DEGRADED`, persistent technical failure `FAILED`, stale `OFFLINE`, and safe-blocked-only `ONLINE`.
|
||||||
|
- [ ] Implement configurable `AGENT_HEARTBEAT_INTERVAL_SECONDS`, `AGENT_OFFLINE_AFTER_SECONDS` with a three-interval default, `CURRENT_SERVER_VERSION`, and `AGENT_EVENT_RETENTION_DAYS=90`.
|
||||||
|
- [ ] Persist product counts, coverage, failed products, blocker aggregates, last/last-successful job, heartbeat, health, and version drift in one upserted snapshot.
|
||||||
|
- [ ] Add dashboard aggregate queries that only read snapshot rows plus bounded grouped counts. Normalize datetimes as UTC for SQLite and PostgreSQL tests.
|
||||||
|
- [ ] Run health/snapshot tests and existing job tests; verify `SAFE_BLOCKED` does not degrade health.
|
||||||
|
- [ ] Commit `feat: add deterministic agent health and snapshots`.
|
||||||
|
|
||||||
|
### Task 4: Add Structured Events, Redaction, Retention, And Audit
|
||||||
|
|
||||||
|
**Files:** `central/app/events.py`, `central/app/audit.py`, models/migration if needed, API routes, maintenance CLI, tests.
|
||||||
|
|
||||||
|
**Interfaces:** `sanitize_event(message, context) -> tuple[str, dict]`, `record_event(session, *, installation_id, severity, event_type, message, context, job_id=None, product_code=None, blocker_id=None) -> AgentEvent`, `purge_expired_events(session, now, retention_days) -> int`, and `record_audit(session, actor, action, target, context) -> AuditEvent`.
|
||||||
|
|
||||||
|
- [ ] Write tests covering case-insensitive redaction of passwords, tokens, API keys, connection strings and secret env names in both messages and nested JSON; assert event message/context maximums.
|
||||||
|
- [ ] Implement allowlisted event types and severities from the mission. Store structured context only, truncate safely, and never persist raw log files.
|
||||||
|
- [ ] Record Agent lifecycle, heartbeat, preflight, job, blocker, dataset/schema, connection, retry, version, and write-audit events at the relevant existing route boundaries.
|
||||||
|
- [ ] Implement retention deletion for non-audit events older than `AGENT_EVENT_RETENTION_DAYS`; preserve audit events, blocker history, write audit, and critical events.
|
||||||
|
- [ ] Add `python -m central.scripts.maintain_events --dry-run` and a write mode that only deletes eligible events.
|
||||||
|
- [ ] Run event security and retention tests, then commit `feat: add redacted operational events and retention`.
|
||||||
|
|
||||||
|
### Task 5: Implement Human Admin Authentication And Session Security
|
||||||
|
|
||||||
|
**Files:** `central/app/human_auth.py`, `central/app/web.py`, config/main, CLI, tests.
|
||||||
|
|
||||||
|
**Interfaces:** `hash_password(password) -> str`, `verify_password(password, encoded_hash) -> bool`, `create_admin(session, username, password) -> AdminUser`, `create_web_session(session, admin, user_agent, ip) -> tuple[str, str]`, `require_human(request) -> AdminUser`, and `require_csrf(request, submitted_token) -> None`.
|
||||||
|
|
||||||
|
- [ ] Write tests for PBKDF2 password storage, wrong-password rejection, inactive admin rejection, session expiry, cookie flags, session rotation, logout invalidation, and CSRF mismatch/replay rejection.
|
||||||
|
- [ ] Add PBKDF2-HMAC-SHA256 hashes with random salts and constant-time verification using only the standard library. Store only session token hashes and CSRF token hashes server-side.
|
||||||
|
- [ ] Add `gti_admin_session` HttpOnly cookie with `Secure` according to production config, `SameSite=Lax`, bounded max age, and no Agent token reuse.
|
||||||
|
- [ ] Add `python -m central.scripts.create_admin` using `getpass`, refusing empty/mismatched passwords and duplicate usernames.
|
||||||
|
- [ ] Run auth tests and commit `feat: add separate human admin sessions`.
|
||||||
|
|
||||||
|
### Task 6: Add Control-Plane APIs And Safe Actions
|
||||||
|
|
||||||
|
**Files:** `central/app/control_api.py`, API integration, audit/events, tests.
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
|
||||||
|
```text
|
||||||
|
GET /v1/dashboard/summary
|
||||||
|
GET /v1/agents
|
||||||
|
GET /v1/agents/{agent_id}
|
||||||
|
GET /v1/agents/{agent_id}/health
|
||||||
|
GET /v1/agents/{agent_id}/coverage
|
||||||
|
GET /v1/agents/{agent_id}/blockers
|
||||||
|
GET /v1/agents/{agent_id}/jobs
|
||||||
|
GET /v1/agents/{agent_id}/events
|
||||||
|
GET /v1/blockers/global
|
||||||
|
GET /v1/jobs/global
|
||||||
|
POST /v1/agents/{agent_id}/actions/reprocess
|
||||||
|
POST /v1/blockers/{blocker_id}/resolve-human
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] Write tests for authentication, client/health/version/last-seen/blocker/severity/date filters, bounded pagination, empty pages, and cross-tenant access denial.
|
||||||
|
- [ ] Implement query services with `page`, `page_size`, `total`, `items`, max page size 100, explicit sort allowlists, and no relationship N+1 queries.
|
||||||
|
- [ ] Implement safe actions only: create idempotent reprocess jobs, resolve only with explicit resolution context, and record actor/action/target/context audit events.
|
||||||
|
- [ ] Reject requests that attempt Firebird writes, AI flags, credentials, database paths, CRT changes, SQL, shell, restart, or arbitrary action names.
|
||||||
|
- [ ] Run API tests and commit `feat: expose authenticated control plane APIs`.
|
||||||
|
|
||||||
|
### Task 7: Build Server-Rendered Operational UI
|
||||||
|
|
||||||
|
**Files:** templates, CSS/JS, `central/app/web.py`, main, tests.
|
||||||
|
|
||||||
|
**Interfaces:** browser routes `/login`, `/logout`, `/`, `/agents`, `/agents/{agent_id}`, `/blockers`, `/jobs`, and `/events`; all mutations use POST forms with hidden CSRF and a visible confirmation field.
|
||||||
|
|
||||||
|
- [ ] Write template tests asserting dashboard cards, client/display-name/hostname identity, health indicator, blocker/job/event pagination, technical IDs only in detail views, and no secret values in rendered HTML.
|
||||||
|
- [ ] Implement dashboard cards for Agent health, product coverage, technical failures, recent failed jobs, stale heartbeats, version drift, and top blocker types.
|
||||||
|
- [ ] Implement responsive Agents/Servers table with search/sort/filter/page controls and links to detail tabs: overview, fiscal, blockers, jobs, events, capabilities, technical info.
|
||||||
|
- [ ] Implement global blockers/jobs/event pages and per-installation timelines with bounded pagination and filters.
|
||||||
|
- [ ] Add plain CSS responsive layout, recognizable health badges, compact tables, confirmation dialogs, and minimal JavaScript for filters/confirmations only.
|
||||||
|
- [ ] Run `python -m pytest central/tests/test_web_panel.py -q` and commit `feat: add server-rendered control panel`.
|
||||||
|
|
||||||
|
### Task 8: Add Fixture Generator And Performance Checks
|
||||||
|
|
||||||
|
**Files:** fixture script, tests, optional fixture helpers.
|
||||||
|
|
||||||
|
**Interfaces:** `generate_fixture(session, agent_count=100, seed=...) -> dict`; must be explicit and never run during app startup or migration.
|
||||||
|
|
||||||
|
- [ ] Write tests asserting exactly 100 installations with distributed `ONLINE`, `DEGRADED`, `OFFLINE`, and `FAILED` states, blockers, jobs, and events.
|
||||||
|
- [ ] Implement deterministic fixture generation behind an explicit `--database-url` and `--confirm-non-production` guard; refuse production APP_ENV and refuse empty confirmation.
|
||||||
|
- [ ] Add query-count/performance checks that dashboard uses snapshot rows and event/job pages issue bounded queries for 100 Agents and large event counts.
|
||||||
|
- [ ] Run fixture tests without seeding the configured production database and commit `test: add control panel fixture and query checks`.
|
||||||
|
|
||||||
|
### Task 9: Document, Run Full Verification, And Consolidate
|
||||||
|
|
||||||
|
**Files:** `docs/CONTROL_PANEL_V1.md`, `docs/AI_PROJECT_STATE.md`, final tests.
|
||||||
|
|
||||||
|
- [ ] Document architecture, identity lifecycle, metadata allowlist, health thresholds, event taxonomy/redaction, retention command, auth/session/CSRF, safe actions, endpoints, screens, and fixture usage.
|
||||||
|
- [ ] Update project state with control-plane readiness and explicit `AI_WRITE_ALLOWED=false`, `FIREBIRD_AI_WRITES=0`, `DEPLOY_PERFORMED=false`.
|
||||||
|
- [ ] Run `python -m pytest tests -q`.
|
||||||
|
- [ ] Run `python -m pytest central/tests -q`.
|
||||||
|
- [ ] Run `python -m pytest agent/tests -q`.
|
||||||
|
- [ ] Run `git diff --check`, inspect `git status`, and verify no secrets or generated fixture database are tracked.
|
||||||
|
- [ ] Commit `docs: document control panel v1 operations`.
|
||||||
|
|
||||||
|
## Verification Matrix
|
||||||
|
|
||||||
|
| Requirement | Plan task | Evidence |
|
||||||
|
|---|---:|---|
|
||||||
|
| Persistent installation identity and metadata | 1-2 | identity tests, registration tests, migration |
|
||||||
|
| Deterministic health/offline/version drift | 3 | health/snapshot tests |
|
||||||
|
| Coverage and blocker summaries | 3, 6-7 | snapshot/API/template tests |
|
||||||
|
| Structured redacted events | 4 | redaction, size, ingestion tests |
|
||||||
|
| Retention and audit preservation | 4 | maintenance tests |
|
||||||
|
| Human auth, cookie session, CSRF | 5 | auth/session tests |
|
||||||
|
| Paginated filtered APIs and safe actions | 6 | API and audit tests |
|
||||||
|
| Dashboard and operational screens | 7 | template and route tests |
|
||||||
|
| 100-Agent simulation/performance | 8 | fixture/query checks |
|
||||||
|
| Documentation and regression | 9 | docs and three suites |
|
||||||
Reference in New Issue
Block a user