Add 'fiscal/' from commit '102ba84500dbd9655a0471288c3e278652593ca9'

git-subtree-dir: fiscal
git-subtree-mainline: d95165fc8f
git-subtree-split: 102ba84500
This commit is contained in:
gti
2026-09-29 14:43:23 -03:00
868 changed files with 566394 additions and 0 deletions
@@ -0,0 +1,183 @@
# Firebird Discovery Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Executar discovery reproduzível e estritamente read-only do banco Firebird disponível, documentando ambiente, schema, produto, tributação, imagens, incrementalidade, riscos, mapping e fingerprint.
**Architecture:** Um coletor local terá uma camada de validação que aceita somente consultas `SELECT`/metadados e um adaptador de conexão que não expõe operações de escrita. O relatório será gerado a partir de resultados do MCP Firebird e, quando possível, do coletor local; nenhuma tabela auxiliar, migration ou alteração será feita no banco.
**Tech Stack:** Python 3, driver Firebird disponível no ambiente somente se já instalado, `pytest`, SQL de `RDB$`/`INFORMATION_SCHEMA` compatível com Firebird.
## Global Constraints
- O banco deve ser tratado como read-only.
- São proibidos `INSERT`, `UPDATE`, `DELETE`, `ALTER`, `CREATE`, `DROP`, `TRUNCATE`, migrations, triggers, procedures, alteração de índices/constraints, criação de usuário e modificação de permissões.
- Não executar testes destrutivos contra o banco real.
- Não armazenar senhas ou secrets em arquivos, logs ou documentação.
- Não iniciar API fiscal, API de imagens, serviço Windows, atualização de produtos ou scheduler de produção.
- Achados devem ser classificados como `CONFIRMADO`, `INFERIDO` ou `NÃO CONFIRMADO`.
---
### Task 1: Preparar base e contrato read-only
**Files:**
- Create: `src/gti_discovery/__init__.py`
- Create: `src/gti_discovery/read_only.py`
- Create: `tests/test_read_only.py`
- Create: `pyproject.toml`
- Create: `.env.example`
- Create: `CLAUDE.md`
**Interfaces:**
- Produces `validate_read_only(sql: str) -> str`, que retorna SQL normalizado ou lança `ValueError`.
- Produces `ReadOnlyConnection`, com método `execute(sql: str, params: tuple = ())` e sem métodos de escrita.
- [ ] **Step 1: Write the failing tests**
```python
import pytest
from gti_discovery.read_only import validate_read_only
def test_accepts_select_and_with_queries():
assert validate_read_only(" SELECT * FROM RDB$DATABASE ") == "SELECT * FROM RDB$DATABASE"
assert validate_read_only("WITH x AS (SELECT 1 AS n FROM RDB$DATABASE) SELECT n FROM x")
@pytest.mark.parametrize("sql", ["INSERT INTO X VALUES (1)", "UPDATE X SET A=1", "DELETE FROM X", "CREATE TABLE X(A INT)", "ALTER TABLE X ADD A INT", "DROP TABLE X", "TRUNCATE TABLE X", "EXECUTE PROCEDURE P"])
def test_rejects_write_or_side_effect_sql(sql):
with pytest.raises(ValueError):
validate_read_only(sql)
```
- [ ] **Step 2: Run test to verify it fails**
Run: `python -m pytest tests/test_read_only.py -q`
Expected: FAIL because the package and validator do not exist.
- [ ] **Step 3: Implement the minimal guard**
Normalize whitespace, remove one trailing semicolon, reject multiple statements, comments and any first keyword other than `SELECT` or `WITH`. Reject side-effect keywords anywhere in the statement, including procedure execution. `ReadOnlyConnection.execute` must call the validator before delegating to the driver connection.
- [ ] **Step 4: Run test to verify it passes**
Run: `python -m pytest tests/test_read_only.py -q`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add src tests pyproject.toml .env.example CLAUDE.md
git commit -m "feat: add read-only discovery foundation"
```
If Git is still unavailable, record the exact blocker and do not initialize or commit without explicit authorization.
### Task 2: Coletar metadados e fingerprint
**Files:**
- Create: `src/gti_discovery/collector.py`
- Create: `src/gti_discovery/fingerprint.py`
- Create: `tests/test_fingerprint.py`
**Interfaces:**
- `collect_metadata(connection) -> dict` executes only predefined read-only queries.
- `build_fingerprint(metadata: dict) -> str` returns a stable SHA-256 digest over sorted relevant metadata.
- [ ] **Step 1: Write tests for stable fingerprints and changes**
```python
from gti_discovery.fingerprint import build_fingerprint
def test_fingerprint_is_order_independent():
a = {"tables": [{"name": "A", "columns": ["ID", "NAME"]}]}
b = {"tables": [{"columns": ["ID", "NAME"], "name": "A"}]}
assert build_fingerprint(a) == build_fingerprint(b)
def test_relevant_schema_change_changes_fingerprint():
a = {"tables": [{"name": "A", "columns": ["ID"]}]}
b = {"tables": [{"name": "A", "columns": ["ID", "NCM"]}]}
assert build_fingerprint(a) != build_fingerprint(b)
```
- [ ] **Step 2: Run tests and verify failure**
Run: `python -m pytest tests/test_fingerprint.py -q`
Expected: FAIL because fingerprint functions do not exist.
- [ ] **Step 3: Implement collection**
Use Firebird system metadata for database info, relations, fields, keys, indexes, foreign keys, triggers, procedures/packages/events where exposed. Include table counts and approximate row counts only when safely available. Redact values and credentials from serialized output. Query a minimal masked sample only for candidate product tables.
- [ ] **Step 4: Implement canonical fingerprinting**
Canonicalize JSON recursively with sorted object keys and deterministic list ordering for tables, columns, keys, indexes, relations and relevant engine/type metadata; hash UTF-8 canonical JSON with SHA-256. Expose classification helper returning `SCHEMA_SUPPORTED`, `SCHEMA_CHANGED` or `SCHEMA_UNKNOWN` based on an allowlisted expected fingerprint and missing/ambiguous product mapping.
- [ ] **Step 5: Run tests**
Run: `python -m pytest tests -q`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add src/gti_discovery/collector.py src/gti_discovery/fingerprint.py tests
git commit -m "feat: collect Firebird metadata and fingerprints"
```
### Task 3: Execute discovery against the available database
**Files:**
- Create: `artifacts/` output files only if explicitly safe and non-sensitive
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
**Interfaces:**
- Uses the Firebird MCP read-only tools and collector contracts from Tasks 1-2.
- [ ] **Step 1: Capture environment metadata**
Collect Firebird version, database identity without secrets, charset/collation, timezone when detectable, approximate size, relation count, engines, and relevant configuration.
- [ ] **Step 2: Enumerate schema relationships**
Inspect all candidate tables/columns for product, stock, barcode, fiscal, company/branch, image/file/attachment and audit concepts; document PKs, FKs, indexes, nullability, defaults and apparent relationships.
- [ ] **Step 3: Confirm product and fiscal semantics with minimal samples**
Select only masked, non-sensitive columns and a very small sample. Never select credentials, customer data, free-form secrets or full BLOB values. Mark names without direct evidence as `INFERIDO` or `NÃO CONFIRMADO`.
- [ ] **Step 4: Inspect operational objects**
List and describe relevant triggers, procedures, packages and events. Check replication/cache indicators only where readable; explicitly document unavailable evidence.
- [ ] **Step 5: Write the two discovery documents**
Record actual findings and uncertainties, including no-write evidence, images alternatives A/B/C, incremental processing options, risks and future decisions. Do not propose fiscal automation yet.
- [ ] **Step 6: Run offline safety tests and review outputs**
Run: `python -m pytest tests -q`
Expected: PASS; manually verify no password, connection string secret, raw BLOB or unnecessary personal data appears in docs/artifacts.
### Task 4: Update project state and final verification
**Files:**
- Create: `docs/AI_PROJECT_STATE.md`
- Modify: `CLAUDE.md` if needed
- [ ] **Step 1: Record current state**
State that this workspace contains the Firebird discovery target, that no ERP writes were performed, which artifacts exist, and what remains blocked or undecided. Keep it current rather than diary-like.
- [ ] **Step 2: Verify prohibited SQL is absent from executable discovery paths**
Run: `rg -n --glob '!docs/superpowers/plans/**' '\\b(INSERT|UPDATE|DELETE|ALTER|CREATE|DROP|TRUNCATE)\\b' src tests`
Expected: only intentional rejection tests/guard literals, with no executable write path.
- [ ] **Step 3: Run final tests and inspect repository status**
Run: `python -m pytest tests -q`; then `git status --short --branch`, `git diff --check`, and `git log -5 --oneline` when Git is available.
- [ ] **Step 4: Create one final commit**
Stage only mission files and create one clear final commit if the repository exists. If no Git repository exists, report `commit: não criado, diretório sem .git` and do not claim a clean Git status.
@@ -0,0 +1,294 @@
# Firebird Mission 00 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Connect read-only to the installed Firebird environment, inspect only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`, perform evidence-backed discovery, and document the result without application writes.
**Architecture:** A read-only connection module will validate the absolute target path and wrap a compatible Firebird connection configured read-only. A capability-aware collector will query catalog metadata and return sanitized evidence, while separate mapping and fingerprint functions classify findings conservatively. External ERP files are inventory inputs only and are never modified.
**Tech Stack:** Python 3.10+, `fdb` already available in the environment, installed Firebird server/client and native tools when present, pytest, existing `gti_discovery` modules.
## Global Constraints
- Files outside `D:\PROJETOS\GTISISTEMAS\Projeto` are `READ_ONLY_EXTERNAL_SYSTEM`.
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened or queried.
- `OTHER_DATABASES_ACCESSED = 0`.
- `APPLICATION_DDL_DML_WRITES = 0`.
- Do not install an arbitrary Firebird version or change service state.
- Do not execute procedures, generators, DML, DDL, migrations, or metadata changes.
- Never store passwords, connection secrets, raw BLOBs, full routine source, customer data, or unnecessary commercial data.
- Use installed Firebird compatibility evidence before selecting a client or driver.
- Preserve unrelated staged and unstaged work already present in the repository.
- Do not commit unless Git identity is already configured; otherwise report `GIT_IDENTITY_NOT_CONFIGURED`.
---
### Task 1: Inventory the installed Firebird environment
**Files:**
- Read-only external paths under `D:\PROJETOS\GTISISTEMAS\`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-5-report.md`
**Interfaces:**
- Produces a sanitized inventory of services, processes, executable/DLL versions, tools, and relevant ERP configuration paths.
- Does not open any `.FDB` during inventory; only the explicitly authorized target may be opened in a later task.
- [ ] **Step 1: Capture read-only Windows service and process evidence**
Run PowerShell commands that only query state:
```powershell
Get-CimInstance Win32_Service |
Where-Object { $_.Name -match 'Firebird|InterBase' -or $_.DisplayName -match 'Firebird|InterBase' } |
Select-Object Name,DisplayName,State,StartMode,PathName
Get-CimInstance Win32_Process |
Where-Object { $_.Name -match 'firebird|fbserver|fb_inet_server|ibserver' } |
Select-Object Name,ExecutablePath,ProcessId
```
Record names, state, executable path, and architecture indicators only. Do not start, stop, restart, or reconfigure anything.
- [ ] **Step 2: Locate client DLLs, server configuration, and tools without opening databases**
Search only names and file properties under `D:\PROJETOS\GTISISTEMAS\` and standard program directories for `fbclient.dll`, `firebird.conf`, `databases.conf`, `isql.exe`, `gstat.exe`, `gbak.exe`, `fbserver.exe`, and `fb_inet_server.exe`. Use file version metadata, not file replacement or loading against another database.
- [ ] **Step 3: Read ERP configuration files as sanitized text**
Read only `.ini`, `.xml`, `.cfg`, `.conf`, and `.txt` candidates that mention Firebird or database connectivity. Extract host, port, path pattern, client library, charset, and dialect when present. Replace every password, token, user secret, and credential value with `<redacted>` before recording evidence. Do not record the ERP's original database as a target.
- [ ] **Step 4: Run offline regression tests**
Run `python -m pytest tests -q`. Expected: all existing tests pass and no external file is changed.
- [ ] **Step 5: Record sanitized inventory**
Write only the resulting facts and unavailable checks to `task-5-report.md`; include `ERP_FILES_MODIFIED = 0`, no target database access yet, and any version/architecture uncertainty.
- [ ] **Step 6: Commit only if identity is configured**
Check `git config user.name` and `git config user.email`. If both are non-empty, stage only the new report and commit:
```text
git add docs/superpowers/sdd/2026-09-16-firebird-discovery/task-5-report.md
git commit -m "chore: record Firebird installation inventory"
```
If either is empty, do not configure Git and record `GIT_IDENTITY_NOT_CONFIGURED`.
### Task 2: Add an authorized-target connection boundary
**Files:**
- Modify: `src/gti_discovery/read_only.py`
- Create or modify: `src/gti_discovery/firebird_connection.py`
- Test: `tests/test_firebird_connection.py`
**Interfaces:**
- Produces `AUTHORIZED_DATABASE_PATH: Path` and `validate_authorized_database_path(path) -> Path`.
- Produces `connect_authorized_read_only(path, driver, client_library=None)`, which rejects every path except the resolved authorized target and returns a connection adapter with `read_only is True`.
- [ ] **Step 1: Write failing path-guard tests**
```python
def test_only_authorized_target_is_accepted(tmp_path):
target = Path(r"D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB")
assert validate_authorized_database_path(target) == target.resolve()
def test_other_database_is_rejected(tmp_path):
with pytest.raises(ValueError, match="authorized"):
validate_authorized_database_path(tmp_path / "other.FDB")
```
Also test case/relative path normalization, missing target rejection, and rejection of a configured ERP-original path.
- [ ] **Step 2: Run the focused tests and verify failure**
Run `python -m pytest tests/test_firebird_connection.py -q`. Expected: FAIL because the path guard module does not yet exist.
- [ ] **Step 3: Implement the smallest path guard and adapter**
Resolve both requested and authorized paths, compare them using Windows case-insensitive semantics, require the target to be a regular file, and reject any other `.FDB`. Wrap the installed driver connection behind an adapter exposing only `execute`, `close`, and an explicit `read_only = True` marker. Configure a read-only transaction through the driver API when supported; if it cannot be proven, fail closed rather than claim read-only.
- [ ] **Step 4: Add a minimal identity-query contract**
Expose a function that executes only `SELECT MON$DATABASE_NAME FROM MON$DATABASE` through `ReadOnlyConnection`, with no write probe. Assert the adapter is constructed only after the target guard and read-only configuration succeed.
- [ ] **Step 5: Run focused and full tests**
Run `python -m pytest tests/test_firebird_connection.py tests/test_read_only.py -q`, then `python -m pytest tests -q`. Expected: PASS.
- [ ] **Step 6: Commit the boundary when Git identity permits**
```text
git add src/gti_discovery/read_only.py src/gti_discovery/firebird_connection.py tests/test_firebird_connection.py
git commit -m "feat: guard authorized Firebird read-only target"
```
### Task 3: Establish real read-only connectivity
**Files:**
- Create: `scripts/run_firebird_discovery.py`
- Create: `.gitignore` update only if required for sanitized local outputs
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-6-report.md`
**Interfaces:**
- Consumes the installed service/client inventory and `connect_authorized_read_only`.
- Produces sanitized connection facts: driver version, client path/version, architecture, engine version, dialect, ODS, charset, page size, and engine-reported database name.
- [ ] **Step 1: Add a no-write script entry point test**
Test that the script's target argument is fixed to `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`, that an alternate path exits before driver invocation, and that the first SQL is exactly the minimal identity query.
- [ ] **Step 2: Implement installed-client selection**
Use only the version and architecture found in Task 1. Prefer the Firebird server connection form and the matching `fbclient.dll`; do not download or install another client. If no compatible client can be proven, stop with `MISSION_00_BLOCKED` and document the exact reason.
- [ ] **Step 3: Connect and execute the first query**
Connect only to the authorized absolute path, begin a read-only transaction, run the minimal identity query, and retrieve engine information using read-only metadata APIs. Never query or log credentials.
- [ ] **Step 4: Verify no-write boundaries**
Inspect the executed SQL log in memory and assert every statement is validated `SELECT`/`WITH`; assert no call to `commit` for application writes, `rollback` as a probe, procedure execution, or generator modification occurs.
- [ ] **Step 5: Run the script once and record sanitized output**
Write `task-6-report.md` with connection outcome, sanitized strategy, errors, and explicit safety values. Do not write raw query dumps or full database rows.
- [ ] **Step 6: Run offline tests again**
Run `python -m pytest tests -q`. Expected: PASS. If connection fails, preserve the failure as evidence and continue only with offline capability tests.
### Task 4: Expand capability-aware catalog collection
**Files:**
- Modify: `src/gti_discovery/collector.py`
- Modify: `src/gti_discovery/fingerprint.py`
- Test: `tests/test_collector.py`
- Test: `tests/test_fingerprint.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-7-report.md`
**Interfaces:**
- Produces `collect_metadata(connection) -> dict` with database facts, relations, fields, keys, indexes, foreign keys, generators, dependencies, triggers, procedures, parameters, packages, and optional monitoring capability results.
- Produces sanitized rows and explicit `errors`, `unavailable`, and `incomplete` fields.
- [ ] **Step 1: Add failing fixture tests for optional catalogs**
Use in-memory fake read-only readers to verify a missing `RDB$PACKAGES`, `RDB$DEPENDENCIES`, `RDB$GENERATORS`, or monitoring relation becomes an unavailable/optional result, while permission/query failures remain `query_error`.
- [ ] **Step 2: Add all catalog SELECT statements**
Add only catalog queries for the required system relations. Do not project trigger/procedure/package source, defaults, BLOB contents, secrets, or application routine execution. Validate every query through `ReadOnlyConnection`.
- [ ] **Step 3: Implement conservative row sanitization**
Redact binary values, sensitive key names and values, oversized strings, tuple-like raw DB-API rows where necessary, and routine source fields. Preserve only metadata needed for schema mapping and fingerprinting.
- [ ] **Step 4: Implement capability detection and collection**
Execute each optional query independently, record Firebird error type only, and continue with available metadata. Never infer absence from an unavailable catalog.
- [ ] **Step 5: Update fingerprint context**
Include sanitized engine context, relevant catalog structure, product mapping evidence, fiscal fields, image structure, and critical trigger metadata. Preserve ordering for positional fields and index segments; sort unordered entities deterministically.
- [ ] **Step 6: Run tests and record metadata outcome**
Run `python -m pytest tests/test_collector.py tests/test_fingerprint.py tests/test_read_only.py -q`, then run the real collector only against the authorized target. Save sanitized outcome in `task-7-report.md`.
### Task 5: Identify products and map fiscal/image/incremental evidence
**Files:**
- Create: `src/gti_discovery/mapping.py`
- Test: `tests/test_mapping.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-8-report.md`
**Interfaces:**
- Produces `identify_product_candidates(metadata, samples) -> list[dict]`.
- Produces `build_schema_mapping(metadata, samples) -> dict` with each mapping classified as `CONFIRMED`, `INFERRED`, or `UNKNOWN`.
- Produces no write operation and never returns raw BLOB or credential contents.
- [ ] **Step 1: Write failing mapping tests**
Create fixtures with explicit PK/FK/index metadata and masked sample rows. Assert that product identity is `CONFIRMED` only when structural and data evidence agree; name-only candidates are `INFERRED`; absent evidence is `UNKNOWN`.
- [ ] **Step 2: Implement candidate scoring from catalog evidence**
Use relation/field names, data types, PK/FK/index information, approximate counts when available, and minimal permitted scalar samples. Do not copy full rows or treat conventional names as proof.
- [ ] **Step 3: Implement product and stock mapping**
Map only fields directly supported by evidence: PK, internal code, descriptions, barcode, reference, unit, group, brand/manufacturer, status, timestamps, stock relation, and relevant constraints.
- [ ] **Step 4: Implement fiscal inventory mapping**
Inventory NCM, CEST, CST, CSOSN, CFOP, origin, ICMS, PIS, COFINS, IPI, FCP, ST, rates, reductions, nature/operation, company, branch, regime, and UF relationships. Describe representation only; do not implement tax rules.
- [ ] **Step 5: Implement image and incremental mapping**
Identify image/file/path/URL/BLOB candidates with subtype and relationship metadata, excluding binary contents. Identify timestamps, audit tables, generators, version fields, and triggers that can support new/changed product detection. If evidence is absent, return `UNKNOWN` and document the gap.
- [ ] **Step 6: Analyze trigger and routine impact as metadata**
Record relevant trigger timing/events and textual dependencies/source only after sanitization. Identify possible secondary tables and generators from metadata without executing any routine. State that future direct UPDATE safety remains unresolved unless evidence proves semantics.
- [ ] **Step 7: Run mapping tests and produce sanitized evidence**
Run `python -m pytest tests/test_mapping.py tests -q`; generate the mapping report with only masked samples and classifications.
### Task 6: Update current-state documentation and final verification
**Files:**
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
- Create: `docs/MISSION_00_REPORT.md`
- Create or modify: `.superpowers/sdd/2026-09-16-firebird-discovery/task-9-report.md`
**Interfaces:**
- Consumes sanitized reports from Tasks 1 through 5 and the generated fingerprint.
- Produces the mandatory final report with a single verdict and explicit database/filesystem safety assertions.
- [ ] **Step 1: Replace stale connectivity claims with current evidence**
Update the three current-state documents to distinguish confirmed connection facts from unavailable catalogs and inferred mappings. Do not state a schema finding without direct evidence.
- [ ] **Step 2: Write the mandatory final report sections**
Include verdict, filesystem safety, authorized database, other database count, application write count, installed Firebird details, connectivity, database facts, products, fiscal, images, incremental processing, triggers/procedures, fingerprint, tests, Git, gaps, and the next-step prohibition list.
- [ ] **Step 3: Scan for secrets and unauthorized targets**
Search project text for password-like keys, connection strings, raw BLOB markers, ERP-original paths, and other `.FDB` access claims. Remove only newly introduced sensitive content; do not alter unrelated user changes.
- [ ] **Step 4: Run final verification commands**
Run:
```text
python -m pytest tests -q
git diff --check
git status --short --branch
git config user.name
git config user.email
```
Verify `TESTESOFICIAIS.FDB` remains ignored, no external path was modified, no DDL/DML exists in executable discovery paths, and no other `.FDB` was opened.
- [ ] **Step 5: Commit only the mission changes when identity is valid**
If Git identity is configured, stage only the mission files and create one commit:
```text
git add src tests docs .superpowers/sdd/2026-09-16-firebird-discovery
git commit -m "feat: complete read-only Firebird discovery"
```
If identity is not configured, leave the working tree uncommitted and report `GIT_IDENTITY_NOT_CONFIGURED`; never change global Git configuration.
## Plan Self-Review
- The approved design's connection, path guard, sanitization, capability detection, mapping, fingerprint, testing, documentation, and Git requirements are covered by Tasks 1-6.
- Every database operation is constrained to the authorized target and read-only SQL; no task permits service lifecycle changes or external file writes.
- Later interfaces use the exact names and return shapes defined in earlier tasks.
- No task depends on an unproved Firebird version; compatibility is established from installed evidence before connection.
- A failed connection remains a documented `MISSION_00_BLOCKED` outcome rather than a reason to use another database or install an arbitrary client.
@@ -0,0 +1,231 @@
# Mission 00D Structured Firebird Boundary Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace arbitrary SQL execution with a structured read-only Firebird API, pass the pre-connection security gate, then connect only to the authorized database and complete discovery.
**Architecture:** A private executor selects static catalog queries by operation key and is the only layer allowed to call the Firebird cursor. `StructuredFirebirdReader` exposes fixed catalog operations and a catalog-issued identity for safe sampling. The old SQL parser remains offline-only or is removed from runtime authority; the live path never accepts consumer SQL.
**Tech Stack:** Python 3.10+, existing `fdb` 2.0.4 and installed Firebird 2.5.9 client, pytest, existing collector/fingerprint/mapping modules.
## Global Constraints
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened or queried.
- Files outside `Projeto` are `READ_ONLY_EXTERNAL_SYSTEM` and must not be modified.
- `OTHER_DATABASES_ACCESSED = 0`.
- `APPLICATION_DDL_DML_WRITES = 0`.
- The public adapter must not expose `execute(sql)`, `query(sql)`, `raw_execute(sql)`, `execute_statement(sql)`, or equivalents.
- Procedures and UDFs may be inventoried as metadata but never executed.
- Credentials, raw BLOBs, full rows, customer data, original ERP database paths, and secrets must not enter reports or source.
- No Firebird installation, DLL, service, registry, permission, or configuration changes.
- No real database access before the security gate in Task 4 passes.
- Git identity must not be invented or configured; report `GIT_IDENTITY_NOT_CONFIGURED` when absent.
---
### Task 1: Replace the raw execution boundary with a private query executor
**Files:**
- Create: `src/gti_discovery/firebird_queries.py`
- Modify: `src/gti_discovery/read_only.py`
- Modify: `src/gti_discovery/__init__.py`
- Test: `tests/test_read_only_boundary.py`
**Interfaces:**
- Produces private `_execute_catalog(adapter, operation_key, params=()) -> Any`.
- Produces immutable internal query mapping keyed by approved operation names, never by caller-provided SQL.
- Public package exports no raw SQL execution function or `ReadOnlyConnection.execute` method.
- [ ] **Step 1: Write failing API-boundary tests**
```python
def test_public_connection_has_no_sql_execute_method():
connection = make_structured_connection()
assert not hasattr(connection, "execute")
assert not hasattr(connection, "query")
assert not hasattr(connection, "raw_execute")
def test_unknown_operation_key_never_reaches_driver():
adapter = RecordingAdapter()
with pytest.raises(ValueError):
_execute_catalog(adapter, "SELECT * FROM X")
assert adapter.calls == []
```
- [ ] **Step 2: Run the focused tests and verify failure**
Run `python -m pytest tests/test_read_only_boundary.py -q`. Expected: FAIL because the current public wrapper exposes `execute` and the private operation executor does not exist.
- [ ] **Step 3: Define static internal query operations**
Move catalog SQL into `firebird_queries.py` under fixed keys such as
`database`, `relations`, `fields`, `keys`, `indexes`, `foreign_keys`,
`triggers`, `procedures`, `procedure_parameters`, `packages`, `generators`,
`dependencies`, and `monitoring`. The executor accepts only one of these
keys, binds values supplied by the operation implementation, and calls the
private driver cursor. It must not accept SQL text, fragments, clauses, or
callable expressions.
- [ ] **Step 4: Remove runtime authority from `ReadOnlyConnection.execute`**
Replace the public wrapper with a private connection implementation that is
constructed only by the structured reader. Keep read-only transaction setup,
adapter capability checks, close behavior, and fail-closed cleanup. Remove
`validate_read_only` and raw execution from package exports if no runtime
consumer depends on them; retain the parser only as an explicitly offline
diagnostic utility if existing tests need it.
- [ ] **Step 5: Run boundary and existing safety tests**
Run `python -m pytest tests/test_read_only_boundary.py tests/test_read_only.py -q`. Expected: PASS after updating tests to assert the new architecture rather than raw SQL execution.
- [ ] **Step 6: Record the root cause**
Add a report stating that quoted identifiers bypassed lexical callable detection and that the incomplete allowlist rejected `CAST`/aggregates. State that the parser is no longer a runtime security boundary.
### Task 2: Implement the structured catalog reader
**Files:**
- Modify: `src/gti_discovery/collector.py`
- Modify: `src/gti_discovery/firebird_connection.py`
- Test: `tests/test_collector.py`
- Test: `tests/test_firebird_connection.py`
**Interfaces:**
- Produces `StructuredFirebirdReader.get_database_info()`.
- Produces `list_relations()`, `get_relation_columns(relation)`, `get_primary_keys(relation)`, `get_foreign_keys(relation)`, `get_indices(relation)`, `get_triggers()`, `get_procedures()`, `get_generators()`, `get_dependencies()`, and optional monitoring methods.
- `collect_metadata(reader)` accepts only the approved reader runtime type and calls structured methods, never SQL.
- [ ] **Step 1: Write failing structured-reader tests**
Use a fake private executor keyed by operation name and assert each public method requests the expected key. Assert an arbitrary adapter, fake object with matching method names, and raw SQL string are rejected by `collect_metadata`.
- [ ] **Step 2: Implement fixed catalog methods**
Each method calls the private operation executor with a fixed key. Metadata queries may contain `CAST`, `COUNT`, `SUM`, `MIN`, `MAX`, `COALESCE`, joins, and grouping. No query is assembled from caller text. Optional catalog failures become `unavailable`; permission failures remain `query_error`.
- [ ] **Step 3: Preserve sanitized metadata**
Keep field names, relation names, types, nullability, positions, keys, indexes, trigger timing/type, procedure metadata, generator names, and dependency names. Redact binary/source/default/sensitive values and never select routine source or BLOB contents.
- [ ] **Step 4: Update collection and fingerprint fixtures**
Adapt collector tests to the structured reader and assert catalog-shaped rows retain safe fields. Assert fingerprint ordering, positional fields, optional capabilities, and `SCHEMA_UNKNOWN` for incomplete metadata remain correct.
- [ ] **Step 5: Run the focused collector suite**
Run `python -m pytest tests/test_collector.py tests/test_fingerprint.py tests/test_read_only_boundary.py -q`. Expected: PASS with no database access.
### Task 3: Add catalog-issued safe sampling
**Files:**
- Modify: `src/gti_discovery/collector.py`
- Create: `src/gti_discovery/identifiers.py`
- Test: `tests/test_sampling.py`
**Interfaces:**
- Produces `quote_firebird_identifier(name) -> str` for validated identifiers.
- Produces `StructuredFirebirdReader.sample_relation_rows(relation, columns, limit) -> list[dict]`.
- Sampling requires relation and columns previously issued by `list_relations`/`get_relation_columns`; it accepts no SQL, WHERE, ORDER BY, expression, procedure, or UDF.
- [ ] **Step 1: Write failing sampling tests**
Cover valid catalog-issued relation/column identities, quoted names, injection strings, unknown relation/column, procedure-like relation, expression input, negative/oversized/non-integer limit, and raw BLOB redaction.
- [ ] **Step 2: Implement identifier validation and quoting**
Require exact membership in the reader’s catalog snapshot, reject control characters, dots, parentheses, semicolons, comments, and expressions, and quote embedded double quotes using Firebird identifier rules. Preserve the original catalog identity for comparison; never normalize arbitrary text into an accepted identifier.
- [ ] **Step 3: Implement bounded sampling**
Generate the internal sample operation from validated identifiers and a validated limit capped at a small constant. The private executor receives an operation key plus structured identifier parameters; it never receives a complete SQL string from the caller.
- [ ] **Step 4: Sanitize sample output**
Return only permitted scalar metadata evidence. Redact binary values, sensitive names, oversized values, and customer/commercial values not needed for mapping. Never execute selectable procedures or functions.
- [ ] **Step 5: Run sampling tests**
Run `python -m pytest tests/test_sampling.py tests/test_collector.py -q`. Expected: PASS with all injection and quoted-identifier cases rejected structurally.
### Task 4: Pass the pre-connection security gate
**Files:**
- Modify: `tests/test_read_only.py` or replace with offline parser tests
- Modify: `tests/test_firebird_discovery_script.py`
- Modify: `docs/MISSION_00_REPORT.md`
- Create: `.superpowers/sdd/2026-09-16-firebird-mission-00d/task-4-report.md`
**Interfaces:**
- Produces a reproducible gate result proving no public arbitrary SQL reaches the driver.
- Produces no database access.
- [ ] **Step 1: Add regression tests for parser non-authority**
Keep offline tests for quoted identifiers, single/double quotes, comments, semicolons, `EXECUTE PROCEDURE`, selectable procedures, UDFs, `UNION`, subqueries, malicious aliases, reserved words in strings, valid quoted names, and valid relation/column names. Assert these tests exercise only the offline helper and cannot invoke the private executor.
- [ ] **Step 2: Add internal Firebird expression fixtures**
Test static internal query text containing `CAST`, `COUNT`, `SUM`, `MIN`, `MAX`, `COALESCE`, and `GROUP BY` through the operation-key executor fixture. Assert no blacklist parser is called and the driver receives only known internal operations.
- [ ] **Step 3: Run the gate commands**
Run:
```text
python -m pytest tests/test_read_only_boundary.py tests/test_sampling.py tests/test_collector.py tests/test_firebird_connection.py tests/test_firebird_discovery_script.py -q
python -m pytest tests -q
```
Expected: all pass, with no database connection attempt. If any gate condition fails, stop with `MISSION_00_BLOCKED`.
- [ ] **Step 4: Update the implementation report**
Document the actual public methods, private executor, transaction read-only behavior, parser role, and gate result. Do not claim live connectivity yet.
### Task 5: Connect and complete real discovery only after the gate
**Files:**
- Modify: `scripts/run_firebird_discovery.py`
- Modify: `src/gti_discovery/mapping.py`
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
- Modify: `docs/MISSION_00_REPORT.md`
- Create: `.superpowers/sdd/2026-09-16-firebird-mission-00d/task-5-report.md`
**Interfaces:**
- Consumes the structured reader and safe sampling API from Tasks 1-4.
- Produces sanitized engine facts, catalog metadata, UI-to-column mappings, image/incremental/trigger evidence, and real fingerprint only if connection succeeds.
- [ ] **Step 1: Revalidate installed Firebird read-only**
Inspect services, processes, client DLL, tools, and configuration under external ERP paths without changing them. Use the already identified Firebird 2.5.9 client only; do not install or start anything.
- [ ] **Step 2: Connect only to the authorized target**
Require the pre-connection gate result, validate the absolute target, require credentials from the process environment without logging them, configure Firebird read-only isolation, issue the minimal identity query first, and compare the returned identity with the authorized target. On any failure, stop with `MISSION_00_BLOCKED`.
- [ ] **Step 3: Execute structured discovery**
Collect database facts and catalogs through fixed operations. Use only catalog-issued sampling to investigate candidate products, stock, fiscal concepts from the ERP UI (`NCM`, `CFOP`, `CST ICMS`, `ICMS rate`, `CEST`, `CSOSN`, PIS/COFINS, IPI, origin, IAT, incidence, IPPT), image/photo tab structures, timestamps/audit/generators, and trigger/procedure dependencies. Mark each mapping `CONFIRMED`, `INFERRED`, or `UNKNOWN`.
- [ ] **Step 4: Generate fingerprint V1**
Hash sanitized engine context, product structure, fiscal fields, relationships, image structure, trigger metadata, and relevant positional catalog data. Treat incomplete or ambiguous collection as `SCHEMA_UNKNOWN`.
- [ ] **Step 5: Update final documents**
Replace stale blocked claims only when live evidence exists. Otherwise preserve `MISSION_00_BLOCKED` and record the exact blocker. Include all safety invariants and no future write/API/service/scheduler proposal.
- [ ] **Step 6: Run final verification**
Run `python -m pytest tests -q`, `git diff --check`, `git status --short --branch`, and Git identity checks. Confirm no other `.FDB` was accessed, no external file changed, no secret exists in project text, and no DDL/DML path exists.
## Plan Self-Review
- The root-cause, structured boundary, static query ownership, safe sampling, procedure/UDF exclusion, parser demotion, defense in depth, gate, live connection, discovery, fingerprint, documentation, and Git requirements from the specification map to Tasks 1-5.
- No task permits database access before Task 4 passes.
- Later interfaces use exact names introduced by earlier tasks.
- The old parser is never used to authorize internal Firebird query syntax.
- No placeholder, unbounded sampling, arbitrary SQL input, or external file modification is permitted.
@@ -0,0 +1,155 @@
# Mission 00E Provenance and Real Discovery Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans (recommended). Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Formalize the trusted-process boundary, prove the imported `fdb` provenance and Firebird compatibility, then execute complete read-only discovery only against the authorized FDB.
**Architecture:** The existing structured reader and private operation registry remain the runtime boundary; Python same-process introspection is explicitly outside the threat model. A provenance module will inspect module origin, distribution metadata, import paths, and local shadowing before the live driver is imported or used. The final workflow will require this gate, the installed Firebird client, target identity, and complete sanitized discovery evidence.
**Tech Stack:** Python 3.10+, installed `fdb` 2.0.4, installed Firebird 2.5.9.27139 client, `importlib.metadata`, pytest, existing structured catalog/mapping/fingerprint modules.
## Global Constraints
- `TRUSTED_PROCESS = TRUE`; malicious same-process introspection is outside this component boundary.
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened or queried.
- External ERP files under `D:\PROJETOS\GTISISTEMAS\` outside `Projeto` are `READ_ONLY_EXTERNAL_SYSTEM`.
- `APPLICATION_DDL_DML_WRITES = 0`, `OTHER_DATABASES_ACCESSED = 0`, and `ERP_FILES_MODIFIED = 0`.
- No public SQL execution or registry mutation API; procedures/UDFs are metadata-only.
- No dependency update, service/configuration change, arbitrary Firebird installation, or guessed credential.
- No database access before the provenance and safety gates pass.
- Never store credentials, original ERP database paths, raw BLOBs, full rows, or unnecessary personal/commercial data.
- Git identity must not be invented or configured; report `GIT_IDENTITY_NOT_CONFIGURED` when absent.
---
### Task 1: Formalize threat model and registry hardening
**Files:**
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/MISSION_00_REPORT.md`
- Modify: `src/gti_discovery/firebird_queries.py`
- Modify: `src/gti_discovery/catalog.py`
- Test: `tests/test_catalog.py`
- Test: `tests/test_read_only_boundary.py`
**Interfaces:**
- Preserves the structured public reader and private immutable operation registry.
- Produces no public operation-registration mutator and no raw SQL API.
- [ ] **Step 1: Write failing threat-model/registry tests**
Assert `TRUSTED_PROCESS = TRUE` is documented, package exports contain no raw execution, registry maps are immutable, no public registration function exists, and normal callers cannot mutate approved operation types. Do not test deliberate access to Python internals as a security requirement.
- [ ] **Step 2: Implement minimal registry hardening**
Keep operation registration module-private and centralized; use immutable mappings and exact approved reader/adapter types. Remove only accidental public aliases or mutators. Do not add cryptographic or sandbox-like mechanisms.
- [ ] **Step 3: Document actual boundary**
State which protections are runtime, which parser code is offline-only, and that a process already running arbitrary Python is outside scope. Remove claims that imply cryptographic isolation.
- [ ] **Step 4: Run focused/full tests**
Run `python -m pytest tests/test_catalog.py tests/test_read_only_boundary.py -q` and `python -m pytest tests -q`. Expected: PASS without database access.
### Task 2: Prove `fdb` module provenance and shadowing absence
**Files:**
- Create: `src/gti_discovery/provenance.py`
- Create: `tests/test_provenance.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-mission-00e/task-2-report.md`
**Interfaces:**
- Produces `inspect_fdb_provenance() -> dict` with `module_file`, `module_package`, module version, distribution name/version/location, interpreter, working directory, relevant `sys.path`/`PYTHONPATH`, and shadowing findings.
- Produces `validate_fdb_provenance(expected_version="2.0.4") -> dict` and raises a sanitized error on ambiguity/mismatch.
- [ ] **Step 1: Write failing provenance tests**
Test installed module metadata correlation, module file under expected `site-packages`, distribution/version agreement, detection of local `fdb.py`/`fdb` directory and unexpected `.pth`, and rejection of a shadowed/mismatched module. Use monkeypatched fixtures; never import or connect to another database.
- [ ] **Step 2: Implement provenance inspection**
Use `importlib.util.find_spec`, `importlib.metadata.distribution/version`, `fdb.__file__`, `fdb.__package__`, `sys.executable`, `Path.cwd()`, `sys.path`, and `PYTHONPATH`. Resolve the actual file and distribution location. Do not print credential values or scan external databases.
- [ ] **Step 3: Implement local shadowing scan**
Search only `Projeto` for exact `fdb.py`, `fdb` directory, and `.pth` candidates. Treat ambiguous duplicate locations as blocked. Record paths only for project-local shadowing evidence, not ERP database paths.
- [ ] **Step 4: Run provenance tests and write report**
Run `python -m pytest tests/test_provenance.py tests/test_read_only_boundary.py -q` and `python -m pytest tests -q`. Write sanitized provenance results and `GIT_IDENTITY_NOT_CONFIGURED` status to the task report.
### Task 3: Pin and validate the proven dependency/client pair
**Files:**
- Modify: `pyproject.toml`
- Modify: `scripts/run_firebird_discovery.py`
- Create: `tests/test_dependency_compatibility.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-mission-00e/task-3-report.md`
**Interfaces:**
- Produces `validate_firebird_runtime() -> dict` that requires the proven `fdb` distribution/version and installed Firebird 2.5.9 client evidence before driver use.
- Produces sanitized client path/version/architecture evidence.
- [ ] **Step 1: Write failing compatibility tests**
Test wrong distribution version, module/distribution mismatch, missing client, wrong client version, shadowing, unsupported read-only TPB, and valid installed `fdb 2.0.4` plus Firebird `2.5.9.27139` fixtures. Tests must fail before any driver connection.
- [ ] **Step 2: Record the existing dependency without unnecessary installation**
Add the appropriate exact `fdb` 2.0.4 dependency declaration only if the project dependency mechanism requires it; do not download or update packages. Document why this version matches the installed Firebird 2.5.9 client.
- [ ] **Step 3: Implement pre-import compatibility gate**
Validate distribution metadata, module provenance, client DLL existence/version, architecture evidence, and exact installed read-only isolation API before importing/using `fdb` for connection. Fail closed on ambiguity.
- [ ] **Step 4: Run tests and record evidence**
Run `python -m pytest tests/test_dependency_compatibility.py tests/test_provenance.py -q` and `python -m pytest tests -q`. Do not connect to the database.
### Task 4: Execute gated real discovery and finalize documentation
**Files:**
- Modify: `scripts/run_firebird_discovery.py`
- Modify: `src/gti_discovery/mapping.py`
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
- Modify: `docs/MISSION_00_REPORT.md`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-mission-00e/task-4-report.md`
**Interfaces:**
- Consumes provenance/runtime validation and structured reader/sampling APIs from Tasks 1-3.
- Produces sanitized connection facts, full catalog discovery, product/UI fiscal mapping, image/incremental/trigger evidence, and a real fingerprint only when complete.
- [ ] **Step 1: Run the complete pre-connection gate**
Run all focused tests and `python -m pytest tests -q`. Confirm no public SQL API, immutable registry, provenance match, no shadowing, compatible client, and green suite. If any fails, write `MISSION_00_BLOCKED` and do not connect.
- [ ] **Step 2: Inspect installed Firebird evidence read-only**
Use services/processes/DLL/tools/configuration only as read-only evidence. Do not start/stop/restart services or modify external ERP files. Keep credentials and original database paths out of reports.
- [ ] **Step 3: Connect to the sole authorized target**
Require process credentials without logging them, validate the absolute allowlist path, configure Firebird transaction READ ONLY, issue the minimal identity query first, and compare returned identity. Never use another `.FDB`, write probe, procedure, UDF, generator, or DDL/DML.
- [ ] **Step 4: Run structured discovery**
Collect all available Firebird catalogs through operation keys, use catalog-issued bounded samples, and map product fields plus every UI fiscal field (`NCM`, CFOP/CST entrada/saída, ICMS, CEST, CSOSN, PIS/COFINS, IPI, origem, IAT, incidência, IPPT). Investigate Foto/image/BLOB subtype/relationship, incremental evidence, and trigger/procedure dependencies without execution.
- [ ] **Step 5: Produce fingerprint and current documents**
Generate a sanitized real fingerprint only for complete evidence and classify it. Update current-state documents with the threat model, provenance, client, connection, schema, mappings, gaps, and exact safety counters. Preserve `MISSION_00_BLOCKED` if any live prerequisite or discovery result is unavailable.
- [ ] **Step 6: Final verification and Git status**
Run `python -m pytest tests -q`, `git diff --check`, `git status --short --branch`, `git config user.name`, and `git config user.email`. Verify no external file changes and no other database access. Do not configure Git or commit if identity is absent.
## Plan Self-Review
- The specification's threat model, registry hardening, provenance, shadowing, dependency/client validation, pre-import gate, real connection, discovery, UI mapping, image/incremental/trigger review, documentation, tests, and Git requirements are covered by Tasks 1-4.
- No database access is permitted before Task 4 Step 1 passes.
- `TRUSTED_PROCESS = TRUE` avoids pretending Python private objects are cryptographic isolation while retaining accidental-use defenses.
- All runtime interfaces use structured operations; the old parser cannot authorize internal SQL.
- No placeholder instructions, arbitrary sampling, dependency download, service mutation, or external file write is permitted.
@@ -0,0 +1,64 @@
# Fiscal Data Foundation V1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add isolated, versioned official fiscal-data ingestion without product tax determination.
**Architecture:** A shared dataset lifecycle stores auditable metadata and retains active versions. Each domain parser/importer validates its own official payload and fails independently.
**Tech Stack:** Python, SQLAlchemy, Alembic, FastAPI, pytest.
## Global Constraints
- Use only official primary sources.
- Preserve existing NCM and CEST imports.
- Never write to Firebird or classify products.
- Failed imports cannot supersede active datasets.
---
### Task 1: Dataset Version Metadata
**Files:**
- Modify: `central/app/models.py`
- Create: `central/alembic/versions/0003_dataset_foundation_metadata.py`
- Test: `central/tests/test_dataset_lifecycle.py`
- [ ] Write a failing test asserting `parser_version`, effective dates, raw reference, and failure reason persist.
- [ ] Run `python -m pytest central/tests/test_dataset_lifecycle.py -q` and verify failure.
- [ ] Add the model fields and migration.
- [ ] Re-run the focused test and verify pass.
### Task 2: Isolated Lifecycle Service
**Files:**
- Modify: `central/app/datasets/service.py`
- Test: `central/tests/test_dataset_lifecycle.py`
- [ ] Write failing tests for a failed replacement retaining its prior active dataset and for independent domains.
- [ ] Run focused tests and verify failure.
- [ ] Implement pending, failed, and active promotion methods scoped by domain.
- [ ] Re-run focused tests and verify pass.
### Task 3: Official Domain Importers
**Files:**
- Create: `central/app/datasets/foundation.py`
- Create: `central/scripts/import_foundation.py`
- Test: `central/tests/test_foundation_parsers.py`
- [ ] Write failing parser tests for official structured payload validation and unsupported structures.
- [ ] Run focused tests and verify failure.
- [ ] Implement versioned fail-closed adapters for BA_ST_MVA, CFOP, CST_CSOSN, PIS_COFINS, TIPI_IPI, CCLASSTRIB_IBSCBS, FCP, and MONOFASICO_ICMS.
- [ ] Re-run focused tests and verify pass.
### Task 4: Active Dataset Smoke Matrix
**Files:**
- Modify: `central/app/api.py`
- Test: `central/tests/test_dataset_status.py`
- [ ] Write failing test for active-only dataset status with lifecycle metadata.
- [ ] Run focused test and verify failure.
- [ ] Add the active dataset status/smoke response.
- [ ] Run `python -m pytest tests central/tests agent/tests -q` and verify pass.
@@ -0,0 +1,523 @@
# GTI Fiscal Hub V1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build and exercise a Docker-hosted FastAPI fiscal hub, official local NCM registry, and read-only Windows-ready Firebird agent for 20 real products.
**Architecture:** A modular FastAPI monolith owns versioned datasets, classification, determination placeholders, cache, and audit in PostgreSQL. A separate agent uses the existing Firebird read-only boundary plus SQLite outbox state to submit product batches and create comparison artifacts, without any ERP writes.
**Tech Stack:** Python 3.12, FastAPI, Pydantic, SQLAlchemy, Alembic, psycopg, PostgreSQL 16, httpx, Docker Compose, SQLite, pytest, `fdb==2.0.4`.
## Global Constraints
- The only canonical NCM source is `https://portalunico.siscomex.gov.br/classif/api/publico/nomenclatura/download/json`.
- Verify the downloaded JSON root has `Data_Ultima_Atualizacao_NCM`, `Ato`, and `Nomenclaturas`; each nomenclature must contain `Codigo`, `Descricao`, `Data_Inicio`, and `Data_Fim`.
- Store raw-payload SHA-256, provenance, record count, and version metadata; preserve prior dataset versions.
- Promote a dataset atomically from `READY` to `ACTIVE` while changing the former active version to `SUPERSEDED` in one transaction.
- No paid APIs, LLMs, photos, BLOB reads, Firebird DML/DDL, ERP schema changes, or Firebird writes.
- Central PostgreSQL is Hub-only; the agent communicates only through authenticated HTTP.
- Every response contains `engine_version`, `dataset_versions`, and `decision_version`.
- `NCM_VALIDATED` does not authorize fiscal-field writes; `can_auto_apply` is always `false` in Mission 02.
- Use the reference fingerprint `1198f89897532b643439f06b729e93b189018f56e7d478e95095f8f56977005b` as `REFERENCE_SCHEMA_V1`.
- Preserve existing worktree changes. Do not configure Git identity or create a commit unless an identity already exists.
## Planned File Structure
- `central/app/config.py`: environment-backed Hub settings.
- `central/app/db.py`: SQLAlchemy engine/session lifecycle.
- `central/app/models.py`: PostgreSQL ORM records.
- `central/app/schemas.py`: HTTP request and response contracts.
- `central/app/datasets/ncm.py`: verified source parsing, import validation, and normalization.
- `central/app/datasets/service.py`: dataset lifecycle and atomic promotion.
- `central/app/classification.py`: exact NCM validation and deterministic candidate ranking.
- `central/app/providers.py`: fiscal integration statuses and RTC provider interface.
- `central/app/auth.py`: bearer token verification and tenant/agent binding.
- `central/app/api.py`: versioned routes and dependencies.
- `central/app/main.py`: application construction and health checks.
- `central/alembic/`: migration environment and initial schema migration.
- `central/scripts/import_ncm.py`: Hub NCM update command.
- `agent/src/gti_fiscal_agent/`: Firebird reader, local state/outbox, client, and runner.
- `agent/scripts/run_agent.py`: CLI for the 20-product run.
- `central/tests/` and `agent/tests/`: focused unit/integration tests.
- `docker-compose.yml`, `central/Dockerfile`, `.env.example`: development operation.
- `docs/MISSION_02_PRODUCT_RESULTS.json`, `docs/MISSION_02_REPORT.md`, and `docs/FISCAL_DATASETS.md`: proof and source documentation.
---
### Task 1: Create Hub, Agent, and Docker Foundations
**Files:**
- Create: `central/pyproject.toml`
- Create: `central/__init__.py`
- Create: `central/app/__init__.py`
- Create: `central/app/config.py`
- Create: `central/app/main.py`
- Create: `central/Dockerfile`
- Create: `docker-compose.yml`
- Modify: `.env.example`
- Test: `central/tests/test_health.py`
**Interfaces:**
- Produces `Settings`, `create_app() -> FastAPI`, and a Compose service named `fiscal-hub`.
- Consumes no new internal components.
- [ ] **Step 1: Write the failing health test**
```python
from fastapi.testclient import TestClient
from central.app.main import create_app
def test_health_reports_database_dependency():
client = TestClient(create_app(database_probe=lambda: True))
assert client.get("/health").json() == {"status": "ok"}
```
- [ ] **Step 2: Run the focused test to verify it fails**
Run: `python -m pytest central/tests/test_health.py -q`
Expected: import failure because `central.app.main` does not exist.
- [ ] **Step 3: Implement the minimal configuration and health application**
```python
def create_app(database_probe: Callable[[], bool] = database_is_available) -> FastAPI:
app = FastAPI(title="GTI Fiscal Hub", version="0.1.0")
@app.get("/health")
def health():
if not database_probe():
raise HTTPException(status_code=503, detail="postgres unavailable")
return {"status": "ok"}
return app
```
Create Compose services for PostgreSQL 16 and `fiscal-hub`, each with a healthcheck. Configure `DATABASE_URL`, `HUB_PORT`, `AGENT_DEV_TOKEN`, `NCM_DATASET_URL`, `DATASET_STORAGE_PATH`, `LOG_LEVEL`, and `HTTP_TIMEOUT_SECONDS` in `.env.example` without values that are real secrets.
- [ ] **Step 4: Run focused tests and Compose validation**
Run: `python -m pytest central/tests/test_health.py -q`
Expected: PASS.
Run: `docker compose config`
Expected: rendered `postgres` and `fiscal-hub` services with healthchecks.
- [ ] **Step 5: Commit if Git identity exists**
Run: `git config user.name`
If it exits nonzero, record `GIT_IDENTITY_NOT_CONFIGURED` and do not commit. Otherwise: `git add central docker-compose.yml .env.example && git commit -m "feat: scaffold fiscal hub services"`.
### Task 2: Add PostgreSQL Schema and Dataset Lifecycle
**Files:**
- Create: `central/app/db.py`
- Create: `central/app/models.py`
- Create: `central/alembic.ini`
- Create: `central/alembic/env.py`
- Create: `central/alembic/versions/0001_initial.py`
- Create: `central/app/datasets/service.py`
- Test: `central/tests/test_dataset_lifecycle.py`
**Interfaces:**
- Produces `DatasetService.create_pending()`, `mark_ready()`, and `promote(dataset_id)`.
- Produces ORM tables `datasets`, `ncm_entries`, `tenants`, `agents`, `classification_requests`, `classification_results`, and `classification_cache`.
- `promote()` is consumed by the importer in Task 3.
- [ ] **Step 1: Write lifecycle tests**
```python
def test_promotion_supersedes_previous_active_in_one_commit(session):
old = datasets.create_ready(session, version="old", sha256="a" * 64)
datasets.promote(session, old.id)
new = datasets.create_ready(session, version="new", sha256="b" * 64)
datasets.promote(session, new.id)
assert datasets.status(session, old.id) == "SUPERSEDED"
assert datasets.status(session, new.id) == "ACTIVE"
```
- [ ] **Step 2: Run lifecycle test to verify failure**
Run: `python -m pytest central/tests/test_dataset_lifecycle.py -q`
Expected: import failure for `DatasetService`.
- [ ] **Step 3: Implement models, migration, and transaction-scoped promotion**
Define a unique dataset identity on `(provider, sha256)`. Scope `ncm_entries` to `dataset_id` with an index on `(dataset_id, code)`. Store only `token_hash` in `agents`. In `promote`, lock current active NCM dataset rows with `SELECT ... FOR UPDATE`, verify target is `READY`, update old active rows to `SUPERSEDED`, update target to `ACTIVE`, and commit once. Roll back on every exception.
- [ ] **Step 4: Run migration and tests against PostgreSQL**
Run: `docker compose up -d postgres`
Run: `docker compose exec fiscal-hub alembic upgrade head`
Run: `python -m pytest central/tests/test_dataset_lifecycle.py -q`
Expected: migration succeeds; lifecycle test passes.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: add fiscal hub persistence"`.
### Task 3: Implement Official NCM Download, Validation, and Import
**Files:**
- Create: `central/app/datasets/ncm.py`
- Create: `central/scripts/import_ncm.py`
- Create: `central/tests/fixtures/ncm_official_sample.json`
- Test: `central/tests/test_ncm_import.py`
**Interfaces:**
- Produces `parse_ncm_payload(payload: bytes) -> NcmPayload`, `import_ncm(session, payload, source_url, downloaded_at) -> Dataset`.
- `NcmPayload` contains `version_label`, `act`, and `entries: list[NcmEntryInput]`.
- Consumed by the import CLI and API dataset status route.
- [ ] **Step 1: Write parser and idempotency tests using the verified source shape**
```python
def test_parser_accepts_verified_siscomex_shape():
parsed = parse_ncm_payload(b'{"Data_Ultima_Atualizacao_NCM":"Vigente em 17/09/2026","Ato":"Resolução Gecex nº 926/2026","Nomenclaturas":[{"Codigo":"08043000","Descricao":"Abacaxis.","Data_Inicio":"01/04/2022","Data_Fim":"31/12/9999"}]}')
assert parsed.entries[0].code == "08043000"
def test_import_same_sha_returns_existing_dataset(session):
first = import_ncm(session, payload, SOURCE_URL, now)
second = import_ncm(session, payload, SOURCE_URL, now)
assert first.id == second.id
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest central/tests/test_ncm_import.py -q`
Expected: import failure for `parse_ncm_payload`.
- [ ] **Step 3: Implement strict parser and staged import**
Calculate `hashlib.sha256(payload).hexdigest()` before parsing. Require an object root with the known `Data_Ultima_Atualizacao_NCM`, `Ato`, and nonempty `Nomenclaturas` members; tolerate additional metadata but reject missing or incorrectly typed known members. Require string `Codigo`, `Descricao`, `Data_Inicio`, and `Data_Fim` per entry. Normalize code by removing non-digits; retain all hierarchy rows but validate product NCMs only against eight-digit entries. Reject duplicate normalized `(code, valid_from, valid_to)` values and implausible totals below 1,000 entries. Insert the dataset as `PENDING`, bulk-insert all entries, validate inserted count equals parsed count, mark `READY`, then call Task 2 promotion. Never delete rows from a prior dataset.
The CLI fetches with `httpx`, requires a successful response, writes the raw payload under the configured dataset storage path using its SHA-256 filename, then invokes the transactional importer.
- [ ] **Step 4: Run importer tests and actual source import**
Run: `python -m pytest central/tests/test_ncm_import.py -q`
Expected: PASS.
Run: `docker compose exec fiscal-hub python -m central.scripts.import_ncm`
Expected: one `ACTIVE` NCM dataset with recorded source URL, SHA-256, update label, act, downloaded timestamp, and count.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: import versioned official NCM data"`.
### Task 4: Add Deterministic Product Classification and Provider Boundaries
**Files:**
- Create: `central/app/classification.py`
- Create: `central/app/providers.py`
- Test: `central/tests/test_classification.py`
- Test: `central/tests/test_providers.py`
**Interfaces:**
- Produces `normalize_code(value) -> str`, `normalize_description(value) -> str`, and `Classifier.classify(request) -> ClassificationDecision`.
- Produces `RtcCalculatorProvider.determine(context) -> ProviderResult` and `NotConfiguredRtcCalculator`.
- Consumed by API routes in Task 5.
- [ ] **Step 1: Write exact-validation and candidate-ranking tests**
```python
def test_valid_existing_ncm_is_validated_without_reclassification(repository):
decision = Classifier(repository).classify(product(current_ncm="0804.30.00", description="anything"))
assert decision.status == "NCM_VALIDATED"
assert decision.classification.ncm == "08043000"
assert decision.can_auto_apply is False
def test_unmatched_description_is_not_invented(repository):
decision = Classifier(repository).classify(product(current_ncm=None, description="RISOLE DE CAMARAO"))
assert decision.status in {"HIGH_CONFIDENCE_CANDIDATE", "AMBIGUOUS", "NOT_FOUND"}
assert decision.can_auto_apply is False
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest central/tests/test_classification.py central/tests/test_providers.py -q`
Expected: import failure for `Classifier`.
- [ ] **Step 3: Implement normalizers, ranking, and NOT_CONFIGURED RTC adapter**
Normalize descriptions with Unicode decomposition, accent removal, uppercase conversion, punctuation-to-space conversion, and collapsed whitespace. Rank candidates with token overlap between normalized description tokens and active NCM descriptions; only return a candidate when its score is at least `0.90` and exceeds the second candidate by `0.15`. Otherwise return `AMBIGUOUS` or `NOT_FOUND`. Leave CEST and fiscal determination fields `null` with reason codes such as `FISCAL_DETERMINATION_NOT_CONFIGURED`.
Implement the RTC protocol to return `{status: "NOT_CONFIGURED", values: {}}`; do not call an external service.
- [ ] **Step 4: Run focused tests**
Run: `python -m pytest central/tests/test_classification.py central/tests/test_providers.py -q`
Expected: PASS.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: add deterministic fiscal classification"`.
### Task 5: Implement Authenticated, Cached `/v1` API
**Files:**
- Create: `central/app/schemas.py`
- Create: `central/app/auth.py`
- Create: `central/app/api.py`
- Modify: `central/app/main.py`
- Test: `central/tests/test_api.py`
**Interfaces:**
- Produces `POST /v1/validate/ncm`, `POST /v1/classify`, `POST /v1/classify/batch`, and `GET /v1/datasets/status`.
- Consumes `Classifier`, `DatasetService`, and a bearer-token `AgentPrincipal`.
- Returns `ClassificationResponse` containing provenance versions and `can_auto_apply=False`.
- [ ] **Step 1: Write API auth, isolation, cache, and idempotency tests**
```python
def test_classify_requires_agent_token(client):
assert client.post("/v1/classify", json=payload()).status_code == 401
def test_same_authenticated_request_uses_cache(client, token):
first = client.post("/v1/classify", headers={"Authorization": f"Bearer {token}"}, json=payload())
second = client.post("/v1/classify", headers={"Authorization": f"Bearer {token}"}, json=payload())
assert first.json()["request_fingerprint"] == second.json()["request_fingerprint"]
assert second.json()["provenance"]["cache_hit"] is True
```
- [ ] **Step 2: Run API tests to verify failure**
Run: `python -m pytest central/tests/test_api.py -q`
Expected: import failure for API routes.
- [ ] **Step 3: Implement models and route behavior**
Hash configured bootstrap tokens with SHA-256 at provisioning and compare bearer tokens with `hmac.compare_digest`. Resolve `tenant_id` and `agent_id` from the token; reject a payload whose supplied IDs differ. Build cache and request fingerprints from resolved tenant, normalized product/context, `ENGINE_VERSION`, `DECISION_VERSION`, and active dataset versions. Persist one request/result audit pair per new fingerprint and return cache metadata for repeats. Batch must preserve input order and return one decision per product. Health returns 503 if the database probe fails.
- [ ] **Step 4: Run API tests and smoke test**
Run: `python -m pytest central/tests/test_api.py -q`
Expected: PASS.
Run: `curl http://localhost:8000/health`
Expected: `{"status":"ok"}` after Compose is up.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: expose authenticated fiscal API"`.
### Task 6: Build Agent State, Outbox, and HTTP Client
**Files:**
- Create: `agent/pyproject.toml`
- Create: `agent/src/gti_fiscal_agent/__init__.py`
- Create: `agent/src/gti_fiscal_agent/state.py`
- Create: `agent/src/gti_fiscal_agent/client.py`
- Create: `agent/src/gti_fiscal_agent/fingerprints.py`
- Test: `agent/tests/test_state.py`
- Test: `agent/tests/test_client.py`
**Interfaces:**
- Produces `AgentState.enqueue(product)`, `pending_batch()`, `mark_sent()`, `mark_result_received()`, and `product_fingerprint(product) -> str`.
- Produces `HubClient.classify_batch(products) -> list[dict]`.
- Consumed by the Firebird runner in Task 7.
- [ ] **Step 1: Write outbox and offline-client tests**
```python
def test_unchanged_product_with_current_versions_is_not_enqueued(state):
state.mark_result_received("000276", source_fingerprint="a", dataset_version="v1", engine_version="0.1.0")
assert state.enqueue("000276", "a", "v1", "0.1.0") is False
def test_offline_client_leaves_row_pending(state, offline_client):
state.enqueue("000276", "a", "v1", "0.1.0")
send_pending(state, offline_client)
assert state.status("000276") == "PENDING_API"
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest agent/tests/test_state.py agent/tests/test_client.py -q`
Expected: import failure for `AgentState`.
- [ ] **Step 3: Implement SQLite state machine and client**
Create a SQLite `product_state` table keyed by product ID and an `outbox` table keyed by product ID plus request fingerprint. Enforce transitions `DISCOVERED -> PENDING_API -> SENT -> RESULT_RECEIVED`; define but do not use `APPLIED`. Persist retry count and next-attempt timestamp. Build product fingerprints from description, GTIN, NCM, `CLASSIFICACAO_FISCAL`, CFOP, CST, aliquota, CEST, and CSOSN. On connection error or non-success HTTP status, increment retry metadata and restore `PENDING_API`; never raise a Firebird write path.
- [ ] **Step 4: Run focused tests**
Run: `python -m pytest agent/tests/test_state.py agent/tests/test_client.py -q`
Expected: PASS.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add agent && git commit -m "feat: add fiscal agent outbox"`.
### Task 7: Integrate Read-Only Firebird Agent and Generate the 20-Product Proof
**Files:**
- Create: `agent/src/gti_fiscal_agent/firebird_reader.py`
- Create: `agent/src/gti_fiscal_agent/runner.py`
- Create: `agent/scripts/run_agent.py`
- Test: `agent/tests/test_firebird_reader.py`
- Test: `agent/tests/test_runner.py`
- Create: `docs/MISSION_02_PRODUCT_RESULTS.json`
- Create: `docs/MISSION_02_REPORT.md`
**Interfaces:**
- Produces `read_products(catalog, codes) -> list[AgentProduct]` and `run_once() -> RunSummary`.
- Consumes `gti_discovery` closed read-only catalog APIs, `AgentState`, and `HubClient`.
- [ ] **Step 1: Write fingerprint gate and payload-mapping tests**
```python
def test_unknown_schema_does_not_read_or_send(reader, client):
result = run_once(reader.with_fingerprint("other"), client)
assert result.status == "SCHEMA_UNKNOWN"
assert client.calls == []
def test_payload_uses_classificacao_fiscal_when_ncm_is_empty(reader):
product = reader.product("000276")
assert to_hub_product(product).current_ncm == "08043000"
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest agent/tests/test_firebird_reader.py agent/tests/test_runner.py -q`
Expected: import failure for `run_once`.
- [ ] **Step 3: Implement SELECT-only reader and runner**
Reuse `connect_authorized_read_only`, catalog data methods, and existing fingerprint functions; do not create a raw SQL escape hatch. Verify the computed schema fingerprint equals the reference constant before selecting the 20 codes from `SAMPLE_PRODUCTS_V1.json`. Select only mapped fiscal scalar fields, never `IMG_PRODUTO` or `FOTO`. Map `current_ncm` from `NCM` when nonempty, otherwise `CLASSIFICACAO_FISCAL`. Send batch requests with the configured tenant, agent, and company context. Save exactly 20 comparison entries containing current fields, returned classification, proposed fields, provenance, reason codes, and divergences. Mark results received only after a valid response.
- [ ] **Step 4: Run focused tests and the real end-to-end CLI**
Run: `python -m pytest agent/tests/test_firebird_reader.py agent/tests/test_runner.py -q`
Expected: PASS.
Run: `python agent/scripts/run_agent.py --sample docs/SAMPLE_PRODUCTS_V1.json --output docs/MISSION_02_PRODUCT_RESULTS.json`
Expected: 20 results, zero Firebird writes, and a persisted SQLite outbox state.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add agent docs/MISSION_02_PRODUCT_RESULTS.json && git commit -m "feat: process products through fiscal hub"`.
### Task 8: Document Dataset and Provider Investigation Results
**Files:**
- Create: `docs/FISCAL_DATASETS.md`
- Modify: `docs/MISSION_02_REPORT.md`
- Test: `central/tests/test_dataset_status_documentation.py`
**Interfaces:**
- Produces a factual provider status registry consumed only by the Mission 02 report.
- [ ] **Step 1: Write a report consistency test**
```python
def test_report_contains_required_cost_and_safety_counters():
report = Path("docs/MISSION_02_REPORT.md").read_text(encoding="utf-8")
assert "PAID_API_CALLS=0" in report
assert "PAID_API_COST=R$0.00" in report
assert "APPLICATION_DDL_DML_WRITES=0" in report
```
- [ ] **Step 2: Run test to verify failure**
Run: `python -m pytest central/tests/test_dataset_status_documentation.py -q`
Expected: FAIL because the Mission 02 report does not exist.
- [ ] **Step 3: Document exact source and integrations without invented claims**
Record NCM as `IMPLEMENTED` with actual imported source URL, download timestamp, SHA-256, update label, act, and row count. Research primary/free sources for CEST, CFOP, CST/CSOSN, cClassTrib, IBS/CBS, and the official RTC calculator; assign only `IMPLEMENTED`, `PARTIAL`, `DISCOVERED`, or `NOT_AVAILABLE` supported by evidence. Include RTC installation/API/license/hosting/input/output evidence, or record the concrete absence/blocker and `NOT_CONFIGURED` adapter. Report the real 20-product metrics, timing, cache hits, and every required safety and cost counter.
- [ ] **Step 4: Run documentation test**
Run: `python -m pytest central/tests/test_dataset_status_documentation.py -q`
Expected: PASS.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add docs central/tests/test_dataset_status_documentation.py && git commit -m "docs: report fiscal hub POC coverage"`.
### Task 9: Verify Complete POC and Operational Resilience
**Files:**
- Modify: `docs/MISSION_02_REPORT.md`
- Test: `central/tests/test_health.py`
- Test: `central/tests/test_api.py`
- Test: `agent/tests/test_runner.py`
**Interfaces:**
- Consumes the complete Hub, importer, agent, report, and Docker Compose configuration.
- [ ] **Step 1: Add failure-mode regression tests**
```python
def test_health_is_503_when_postgres_is_down():
client = TestClient(create_app(database_probe=lambda: False))
assert client.get("/health").status_code == 503
def test_second_agent_run_skips_unchanged_products_and_counts_cache(agent):
first = agent.run_once()
second = agent.run_once()
assert first.total == 20
assert second.api_requests == 0
```
- [ ] **Step 2: Run these tests to verify their expected initial behavior**
Run: `python -m pytest central/tests/test_health.py agent/tests/test_runner.py -q`
Expected: PASS after Tasks 1-7; otherwise fix the implementation before continuing.
- [ ] **Step 3: Execute Compose and live proof**
Run: `docker compose up -d --build`
Run: `docker compose ps`
Expected: `postgres` and `fiscal-hub` report healthy.
Run: `docker compose exec fiscal-hub python -m central.scripts.import_ncm`
Run: `python agent/scripts/run_agent.py --sample docs/SAMPLE_PRODUCTS_V1.json --output docs/MISSION_02_PRODUCT_RESULTS.json`
Run the same agent command again and record API call count, cache behavior, total timing, per-product timing, and cache-hit timing in the report.
- [ ] **Step 4: Run full verification**
Run: `python -m pytest tests central/tests agent/tests -q`
Expected: all tests pass.
Run: `python -m json.tool docs/MISSION_02_PRODUCT_RESULTS.json > $null`
Expected: valid JSON with 20 entries.
Run: `git diff --check`
Expected: no whitespace errors; line-ending warnings may be reported without diff failures.
- [ ] **Step 5: Record final state without forcing Git identity**
Run: `git status --short`
Run: `git log --oneline -10`
Add branch, status, commit state if any, health evidence, test output, performance, and real gaps to `docs/MISSION_02_REPORT.md`. If `git config user.name` is unavailable, report `GIT_IDENTITY_NOT_CONFIGURED` and do not configure it.
@@ -0,0 +1,187 @@
# PRE-AI Local Completion V3 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Complete and verify the local semantic pipeline, knowledge preparation, and operator tooling without external AI or uncertain Firebird writes.
**Architecture:** Extend the existing `central.app.ai` contracts with a deterministic scenario provider and an ID-correlated batch harness. Add conservative, versioned KB/retrieval preparation artifacts and CLI-first operational commands that compose existing read-only mechanisms.
**Tech Stack:** Python 3.11+, SQLAlchemy, PostgreSQL/SQLite test engines, pytest, existing Alembic models, JSON/CSV artifacts, Docker only for existing local test execution.
## Global Constraints
- Do not call paid/external AI APIs.
- Keep `DETERMINISTIC_GATE_CLOSED=true`.
- Keep `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0`.
- Never use fake answers as semantic decisions.
- Never write uncertain semantic values to Firebird.
- Preserve tenant-specific fiscal context outside global KB identity.
- Preserve all conflicts and historical evidence.
- Do not add arbitrary fiscal rules.
---
### Task 1: Fake Provider Contract
**Files:**
- Create: `central/app/ai/fake_provider.py`
- Modify: `central/app/ai/provider.py`
- Test: `central/tests/test_fake_provider.py`
**Interfaces:**
- `DeterministicFakeSemanticProvider.select_batch(items, scenario=None) -> list[dict]`
- Scenarios: `CORRECT`, `WRONG`, `ABSTAIN`, `INVALID_JSON`, `MISSING_ITEM`, `DUPLICATE_ITEM`, `OUT_OF_CANDIDATE`, `TIMEOUT`, `429`, `500`.
- The provider returns only test responses and never persists them.
- [ ] Write tests asserting every scenario and deterministic call counters.
- [ ] Run `python -m pytest central/tests/test_fake_provider.py -q` and observe failure.
- [ ] Implement the provider using the existing `LlmEvalItem`/validator contract.
- [ ] Ensure failures use typed transport/contract errors and do not mutate input items.
- [ ] Run the focused tests and confirm pass.
### Task 2: ID-Correlated Batch Harness
**Files:**
- Create: `central/app/ai/batch_runner.py`
- Test: `central/tests/test_batch_runner.py`
**Interfaces:**
- `run_batches(items, provider, batch_size, max_retries=2) -> BatchRunResult`.
- `BatchRunResult` exposes `results_by_work_item_id`, `errors`, `provider_calls`, `retries`, and `usage`.
- [ ] Add failing tests for batch sizes 5, 10, and 20, shuffled responses, missing IDs, duplicates, invalid JSON, timeout, 429, 500, and partial failure.
- [ ] Implement correlation by `work_item_id`, never array position.
- [ ] Retry only transport failures and preserve failed item IDs.
- [ ] Shrink/retry only according to explicit policy; never silently drop items.
- [ ] Run focused tests.
### Task 3: Budget, Cache, and Dataset Drift Tests
**Files:**
- Modify: `central/app/ai/provider.py`
- Modify: `central/app/ai/llm_provider.py`
- Create: `central/app/ai/cache_policy.py`
- Test: `central/tests/test_ai_controls_v3.py`
**Interfaces:**
- `cache_key(input_fingerprint, candidate_set_hash, model, prompt_version, dataset_versions) -> str`.
- `classify_knowledge_freshness(record_versions, active_versions) -> VALID | NEEDS_REVALIDATION | CONFLICT`.
- [ ] Test sufficient and exceeded budgets, including zero provider calls on pre-call rejection.
- [ ] Test input/output/cached token accounting and unknown prices.
- [ ] Test cache hit/miss for each key component with `provider_calls=0` on hit.
- [ ] Test dataset drift classification without deleting history.
- [ ] Implement minimal helpers and connect them to existing cache/provider paths.
- [ ] Run focused tests.
### Task 4: Offline E2E Simulation
**Files:**
- Modify: `central/scripts/run_real_llm_ncm_eval.py`
- Create: `central/scripts/run_offline_ai_simulation.py`
- Test: `central/tests/test_offline_ai_simulation.py`
**Interfaces:**
- `run_offline_ai_simulation(output_dir, scenario, batch_sizes=(5,10,20)) -> dict`.
- Output explicitly marks fake results as plumbing-only and `FIREBIRD_AI_WRITES=0`.
- [ ] Test GOLD -> gate -> SILVER/NOISY -> residual plan sequencing.
- [ ] Test that fake selections never enter Firebird or KB semantic truth tables.
- [ ] Implement the command using final existing work-item/candidate contracts.
- [ ] Emit sanitized metrics and zero-write evidence.
- [ ] Run the command and focused tests.
### Task 5: Canonical KB and Conflict Audit
**Files:**
- Modify: `central/app/models.py`
- Create: `central/alembic/versions/0015_kb_canonical_identity.py`
- Create: `central/app/ai/kb_consolidation.py`
- Create: `central/scripts/audit_kb_conflicts.py`
- Create: `central/tests/test_kb_consolidation.py`
- Generate: `docs/KB_CONFLICT_AUDIT_V1.json`
**Interfaces:**
- `canonical_identity(observation) -> str` using GTIN first, then conservative fingerprint fields.
- `classify_conflict(observations) -> str` using the six requested classifications.
- `audit_conflicts(observations) -> dict` with counts, decisions, and evidence.
- [ ] Test GTIN-equivalent abbreviation merges and incompatible same-GTIN conflicts.
- [ ] Test packaging variations remain distinct or explicitly classified.
- [ ] Add additive canonical identity without promoting tenant fiscal fields.
- [ ] Audit all available KB conflicts and generate the JSON artifact.
- [ ] Run focused tests and validate artifact counts.
### Task 6: Retrieval V3 Evaluation and Candidate Preparation
**Files:**
- Modify: `central/app/ai/retrieval.py`
- Create: `central/app/ai/retrieval_v3.py`
- Create: `central/scripts/run_retrieval_eval_v3.py`
- Create: `central/scripts/prepare_ai_candidates.py`
- Test: `central/tests/test_retrieval_v3.py`
- Generate: `docs/NCM_RETRIEVAL_EVALUATION_V3.json`
**Interfaces:**
- `retrieval_v3_candidates(corpus, index, description, metadata, k=20) -> list`.
- `prepare_candidate_set(work_item, active_datasets) -> dict` with normalized product, top-20, scores, official descriptions, hierarchy, neighbors, GTIN evidence, and input fingerprint.
- [ ] Add holdout tests for BM25/TF-IDF/character/rarity/hierarchy/KB/GTIN/group evidence.
- [ ] Compare V3 to V2 and activate only if GOLD recall@20 is at least 0.954 and no required metric regresses.
- [ ] Prepare NCM invalid, CEST, EX_TIPI, and PRODUCT_ROLE payloads without making semantic decisions.
- [ ] Persist versioned candidate artifacts idempotently.
- [ ] Run retrieval evaluation and focused tests.
### Task 7: Payload Optimization and Final Phase-1 Rerun
**Files:**
- Create: `central/app/ai/payloads.py`
- Modify: `central/scripts/run_offline_ai_simulation.py`
- Test: `central/tests/test_payloads.py`
- Generate: `docs/AI_PAYLOAD_OPTIMIZATION_V1.json`
- [ ] Test compaction removes only redundant repeated evidence and preserves candidate recall.
- [ ] Measure before/after token estimates for all 511 selection items.
- [ ] Rerun Phase 1 simulation against final work items and candidate sets.
- [ ] Confirm fake outputs remain non-authoritative and zero-write.
### Task 8: Guided Company and Group Questionnaires
**Files:**
- Create: `scripts/answer_company_context.py`
- Create: `scripts/answer_group_questions.py`
- Create: `central/app/ai/questionnaire.py`
- Test: `central/tests/test_questionnaire.py`
- [ ] Test validation, review-before-import, cancel-without-write, JSON output, and CSV output.
- [ ] Implement one-question-at-a-time Portuguese prompts with affected products/cells/options.
- [ ] Group questions by company, group, subgroup, NCM, question, and consequence; show examples.
- [ ] Enforce `ENTRY_SCOPE` values and document `MIXED_OR_DYNAMIC` behavior.
- [ ] Implement `STORE_IPI_PROFILE` as explicit choice plus preview/rollback plan only.
### Task 9: Rollback Tooling and Full Local Validation
**Files:**
- Modify: `scripts/rollback_field.py`
- Create: `scripts/rollback_run.py`
- Create: `scripts/run_local_validation.py`
- Test: `tests/test_rollback_run.py`
- Test: `tests/test_local_validation.py`
- [ ] Test preview fields: run_id, database, field, product, before, current, target, affected rows.
- [ ] Enforce `current == expected_after`; emit `CONFLICT` and never overwrite drift.
- [ ] Compose preflight, sources, company context, KB, deterministic engine, residual classification, AI-ready preparation, and reports.
- [ ] Make repeated runs idempotent and assert no semantic Firebird writes.
### Task 10: Health Report and Final Verification
**Files:**
- Create: `scripts/generate_local_health.py`
- Generate: `docs/LOCAL_PRODUCT_HEALTH_V1.json`
- Modify: `docs/AI_PROJECT_STATE.md`
- [ ] Generate all requested health fields with measured values, not placeholders.
- [ ] Run Phase-1 E2E again using final Phase-2 work items/candidates.
- [ ] Run `python -m pytest tests -q` and `python -m pytest central/tests -q`.
- [ ] Run `git diff --check` and verify no API key or token appears in tracked artifacts.
- [ ] Report remaining blockers precisely: external Z.ai balance/credential, missing source files, unresolved human decisions, and any measurable failures.
@@ -0,0 +1,68 @@
# PRE-ZAI Blocker Closure Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Audit historical KB conflicts and produce an honest, operator-ready pre-Z.ai state without external AI or semantic Firebird writes.
**Architecture:** Add pure evidence reconstruction over the existing seed payload, a guided COMPANY session wrapper around the existing context validator, and deterministic final-state/report generators. Preserve current retrieval and conflict policy.
**Tech Stack:** Python, JSON/CSV, existing context validation, pytest.
## Global Constraints
- Do not call external AI APIs.
- Do not deploy.
- Keep `FIREBIRD_AI_WRITES=0` and `AI_WRITE_ALLOWED=false`.
- Never resolve NCM by majority voting.
- Never auto-answer human context questions.
- Preserve all 90 current NCM conflicts unless objective evidence changes an individual row.
---
### Task 1: Historical Conflict Reconstruction
**Files:**
- Create: `central/app/ai/historical_conflicts.py`
- Create: `central/scripts/audit_historical_kb_conflicts.py`
- Create: `central/tests/test_historical_conflicts.py`
- Generate: `docs/KB_HISTORICAL_CONFLICT_AUDIT_V1.json`
- [ ] Test each classification and the no-majority rule.
- [ ] Join payload rows by GTIN/fingerprint and retain all evidence fields.
- [ ] Classify 73 historical rows; emit `INSUFFICIENT_EVIDENCE` when source evidence is absent.
- [ ] Assert current 90 conflicts are unchanged in the new artifact.
### Task 2: Guided COMPANY Session and Impact Preparation
**Files:**
- Create: `scripts/run_company_decision_session.py`
- Modify: `central/app/ai/questionnaire.py`
- Create: `central/tests/test_company_decision_session.py`
- Generate: `docs/COMPANY_DECISION_SESSION_V1.json`
- [ ] Test COMPANY-only first ordering, Portuguese labels, cancellation, and review-only behavior.
- [ ] Add actual local IPI impact fields from available coverage artifacts; do not invent unavailable counts.
- [ ] Validate `INTRA`, `INTER`, `MIXED_OR_DYNAMIC`, and both IPI profile options.
- [ ] Require explicit confirmation before writing the official context artifact.
### Task 3: Final Regeneration and Health State
**Files:**
- Create: `scripts/generate_pre_zai_state.py`
- Modify: `scripts/generate_local_health.py`
- Create: `central/tests/test_pre_zai_state.py`
- Generate: `docs/PRE_ZAI_FINAL_STATE_V1.json`
- Update: `docs/LOCAL_PRODUCT_HEALTH_V1.json`
- [ ] Reuse the regenerated residual counts and historical audit counts.
- [ ] Keep retrieval V2 active and record V3 as not activated without holdout gain.
- [ ] Emit all requested health fields and explicit human/external blockers.
- [ ] Assert zero AI writes and no source/pipeline gaps except evidence-unavailable historical rows.
### Task 4: Verification
- [ ] Run `python -m pytest tests -q`.
- [ ] Run `python -m pytest central/tests -q`.
- [ ] Run the historical audit, company session preview, regeneration, and health commands.
- [ ] Run `git diff --check` and a tracked-secret scan.
- [ ] Document final metrics in `docs/PRE_AI_LOCAL_COMPLETION_V3_REPORT.md` or the new final artifact.
@@ -0,0 +1,182 @@
# Autonomous Fiscal Completion V1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Reduce deterministic fiscal residuals across the five allowlisted databases and produce a measured autonomy report without AI calls or human-answer inference.
**Architecture:** Add pure, data-driven requiredness and completeness modules first. Feed them sanitized read-only ERP evidence and conservative provenance-aware aggregation. Keep dry-run/write planning separate from guarded execution, and add provider/cache/budget primitives without enabling any external provider.
**Tech Stack:** Python 3.12, pytest, Firebird read-only discovery through existing `firebird_reader`, existing guarded executor, JSON/Markdown artifacts, SQLAlchemy/PostgreSQL models only for central AI ledger.
## Global Constraints
- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` throughout.
- No Z.ai, OpenAI, or any AI/provider network call.
- Human COMPANY/GROUP/PRODUCT decisions remain unanswered and unpersisted.
- `NCM_CONFLICT` rows and the 73 historical `INSUFFICIENT_EVIDENCE` conflicts remain blocked.
- Firebird access is read-only unless an exact allowlisted path passes every existing guard and the deterministic dry-run proves a write.
- Never infer fiscal origin, ST withholding, regime facts, or tenant policy from weak/default evidence.
- `RETRIEVAL_V2` remains active; retrieval V3 remains unpromoted.
- Every new artifact reports provenance, unknowns, counts, and zero-call/write measurements.
## File Map
- Create `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json`: field/context requiredness rules.
- Create `central/app/fiscal_completeness.py`: pure field/product evaluation API.
- Create `central/tests/test_fiscal_completeness.py`: matrix and evaluator tests.
- Create `docs/ERP_INBOUND_FISCAL_EVIDENCE_MAP_V1.json`: discovered tables, relations, evidence capabilities, and counts.
- Create `scripts/discover_inbound_fiscal_evidence.py`: read-only schema/evidence discovery with sanitized output.
- Create `central/app/evidence_aggregation.py`: supplier/document observations and conservative consensus metrics.
- Create `central/tests/test_evidence_aggregation.py`: conflict and threshold tests.
- Create `docs/ORIGIN_EVIDENCE_RESOLUTION_V1.json`: origin statuses and provenance metrics.
- Create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json`: per-database completeness/autonomy metrics.
- Create `docs/LOCAL_PRODUCT_HEALTH_V2.json`: updated health summary.
- Create `docs/EXTERNAL_PRODUCT_DATA_PROVIDER_EVALUATION_V1.md`: researched provider comparison without purchase/query.
- Create `central/app/product_data_provider.py`: provider protocol and cache normalization types.
- Create `central/app/ai/budget_ledger.py`: persistent budget decision interface and atomic pre-call guard.
- Create `central/tests/test_product_data_provider.py` and `central/tests/test_ai_budget_ledger.py`.
- Modify `central/app/models.py` and migrations only if the existing migration/test pattern supports the ledger/cache tables.
- Modify `docs/AI_PROJECT_STATE.md` and create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.md` only after measured outputs exist.
---
### Task 1: Requiredness Matrix Contract
**Files:** Create `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json`; test in `central/tests/test_fiscal_completeness.py`.
**Interfaces:** Matrix JSON has `artifact`, `version`, `fields`, `contexts`, and each field rule has `classification`, `when`, `validity`, `source`, and `human_decision_required`. The five classifications are exact strings: `REQUIRED`, `CONDITIONAL`, `OPTIONAL_REFERENCE`, `NOT_APPLICABLE`, `ERP_NOT_USED`.
- [ ] Write tests that load the matrix, require all 23 contract fields exactly once, reject unknown classifications, and verify `NOT_APPLICABLE` does not imply missing data.
- [ ] Run `python -m pytest central/tests/test_fiscal_completeness.py -q`; expect failure because the matrix is absent.
- [ ] Create the matrix from `FISCAL_FIELD_CONTRACT_V1.json`, `ERP_OPERATION_PROFILES_V1.json`, and known company context keys. Mark unresolved PIS/COFINS and ICMS operator facts conditional, IPI reference-only for non-contributors, service/input-only fields as conditional, and unsupported ERP fields as `ERP_NOT_USED`.
- [ ] Run the focused tests and confirm they pass.
### Task 2: Completeness Evaluator
**Files:** Create `central/app/fiscal_completeness.py`; modify `central/app/field_contract.py` only if a shared field-name export is needed; test `central/tests/test_fiscal_completeness.py`.
**Interfaces:** Implement `evaluate_field(field, value, context, matrix) -> dict` and `evaluate_product(product, context, matrix) -> dict`. Field statuses are `REQUIRED_AND_VALID`, `REQUIRED_MISSING`, `CONDITIONAL_AND_VALID`, `CONDITIONAL_NOT_TRIGGERED`, `OPTIONAL_VALID`, `OPTIONAL_EMPTY`, `NOT_APPLICABLE`, `ERP_NOT_SUPPORTED`; product output includes `FISCALLY_COMPLETE` and `fields`.
- [ ] Add tests for required valid/missing, conditional triggered/untriggered, optional empty, not applicable empty, ERP unsupported, and invalid values.
- [ ] Run focused tests to confirm the new API fails before implementation.
- [ ] Implement pure evaluation with no database/network dependency; use existing contract validators where available and preserve explicit `None`/empty values.
- [ ] Add tests proving a blank `NOT_APPLICABLE` field does not make a product incomplete and a human-required company fact remains missing.
- [ ] Run focused tests and then `python -m pytest central/tests/test_field_contract.py central/tests/test_fiscal_completeness.py -q`.
### Task 3: Read-Only Inbound Discovery
**Files:** Create `scripts/discover_inbound_fiscal_evidence.py`, `docs/ERP_INBOUND_FISCAL_EVIDENCE_MAP_V1.json`; tests in `tests/test_inbound_evidence_discovery.py`.
**Interfaces:** `discover_database(path, connection_factory) -> dict` must use read-only isolation and return sanitized table/column/relationship/evidence metadata. `classify_evidence_source(table, columns) -> set[str]` returns categories such as `HEADER`, `ITEM`, `SUPPLIER`, `KEY`, `XML`, `FISCAL_ITEM`, and `UNKNOWN`.
- [ ] Write fake-cursor tests proving table names are discovered from system metadata, XML/BLOB columns are detected by type/name, and no DML/DDL SQL is issued.
- [ ] Run the focused tests and confirm failure before implementation.
- [ ] Implement metadata discovery using the existing Firebird reader/connection conventions and parameter-free system SELECTs only; do not assume NF-e table names.
- [ ] Run discovery against the five explicit paths in read-only mode, record unavailable databases/evidence as `UNAVAILABLE`, and sanitize credentials, CNPJ, XML payloads, and product descriptions from the artifact.
- [ ] Validate the JSON artifact contains database/path policy, discovered tables, candidate relationships, XML availability, and row counts.
### Task 4: Inbound Observation Extraction
**Files:** Modify `scripts/discover_inbound_fiscal_evidence.py`; create `central/app/evidence_aggregation.py`, `central/tests/test_evidence_aggregation.py`; create `docs/INBOUND_SUPPLIER_OBSERVATIONS_V1.json`.
**Interfaces:** `Observation` contains product identity, supplier/document identity hash, observed fiscal values, UF/date, and provenance. `aggregate_observations(observations) -> dict` returns `observation_count`, `supplier_count`, `agreement_rate`, `latest_observation`, and `conflict_count` per product/attribute.
- [ ] Add tests for unanimous multi-supplier evidence, split evidence, latest-date ordering, missing values, and supplier/document de-duplication.
- [ ] Run focused tests to verify failure before implementation.
- [ ] Implement conservative aggregation: agreement is strong only when at least two observations from at least two suppliers agree and no conflicting observation exists; never use simple majority as truth.
- [ ] Extract only available inbound facts; classify every observation as `SUPPLIER_OBSERVATION` and retain source table/document provenance.
- [ ] Produce counts for inbound items analyzed, XML documents available, and evidence-bearing products.
### Task 5: Deterministic Origin Resolution
**Files:** Create `central/app/origin_resolution.py`, `central/tests/test_origin_resolution.py`, `docs/ORIGIN_EVIDENCE_RESOLUTION_V1.json`.
**Interfaces:** `resolve_origin(product_id, observations, kb_facts) -> dict` returns `status` in `ORIGIN_CONFIRMED`, `ORIGIN_CONFLICT`, `ORIGIN_NEEDS_EVIDENCE`, plus value, metrics, and provenance.
- [ ] Test that explicit consistent inbound origin can confirm, disagreement creates conflict, one weak observation remains pending, and CST digits never determine origin.
- [ ] Implement only the conservative threshold from Task 4; do not globalize a lot/acquisition-specific fact without compatible identity evidence.
- [ ] Reconcile against the existing 20,995-style pending origin counts without changing company decision artifacts or current conflicts.
- [ ] Record before/resolved/conflict/after counts and exact reasons.
### Task 6: Product Role, ST/CEST, and Dynamic CFOP
**Files:** Create `central/app/deterministic_resolution.py`, `central/tests/test_deterministic_resolution.py`; update `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json` and `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json` through generated outputs.
**Interfaces:** `resolve_product_role(signals)`, `resolve_st_status(signals)`, and `resolve_entry_scope(observations, company_uf)` return a value only with strong multi-signal evidence; otherwise return a typed pending result with reasons. Entry scope accepts only `INTRA`, `INTER`, `MIXED_OR_DYNAMIC`.
- [ ] Test conflicting role signals, recipe/production evidence, ST subject versus withheld, CEST corroboration, and mixed supplier UFs.
- [ ] Implement evidence stacks using group, recipe, stock, supplier, purchases, sales, production, flags, and history; no single weak flag may resolve role.
- [ ] Keep `ST_WITHHELD` pending when purchase evidence does not prove retention; do not copy company-specific CFOP/CST/CSOSN across tenants.
- [ ] Determine CFOP entry behavior as operational default/last operation/preferred default only if ERP evidence proves the behavior; otherwise report dynamic scope.
- [ ] Re-run deterministic dry-run planning for all five databases and classify each candidate without writing.
### Task 7: Guarded Deterministic Write Reprocessing
**Files:** Modify only existing dry-run/write orchestration files required by tests; add `tests/test_autonomous_write_gate.py`; create `docs/AUTONOMOUS_WRITE_AUDIT_V1.json`.
**Interfaces:** `eligible_write(candidate, guards) -> bool` and `reprocess_allowlisted(databases, dry_run) -> dict` must expose `writes_attempted`, `writes_verified`, `second_pass_writes`, skipped reasons, and changed columns.
- [ ] Add tests proving non-allowlisted paths, missing provenance, human-required facts, conflicts, failed schema/type guards, rowcount mismatch, and post-write mismatch all fail closed.
- [ ] Run tests against mocked executor before changing orchestration.
- [ ] Wire only deterministic candidates that already satisfy the existing guarded executor contract; do not broaden field allowlists or AI flags.
- [ ] Execute first pass only after the dry-run artifact reports eligible changes; verify exact before/after and unchanged columns.
- [ ] Reprocess all five databases and require `SECOND_PASS_WRITES=0`; if any guard or verification fails, record pending and do not retry destructively.
### Task 8: Completeness and Autonomy Metrics
**Files:** Create `scripts/generate_autonomous_completion.py`, `central/tests/test_autonomy_metrics.py`, `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json`, `docs/LOCAL_PRODUCT_HEALTH_V2.json`.
**Interfaces:** `build_database_metrics(database_snapshot, evaluator, evidence) -> dict` returns `TOTAL_PRODUCTS`, complete/incomplete counts, reason buckets, required instances, valid/missing counts, autonomous rate, KB reuse, and AI residual counts.
- [ ] Add fixture tests for all incompleteness buckets: `MISSING_COMPANY_FACT`, `MISSING_PRODUCT_FACT`, `MISSING_EXTERNAL_PRODUCT_DATA`, `AI_SEMANTIC_CANDIDATE`, `DATA_CONFLICT`, `ERP_SCHEMA_LIMITATION`.
- [ ] Implement metrics from evaluator outputs and measured artifacts, not estimates or prior report text.
- [ ] Generate five-bank metrics, `AI_READY_BEFORE=511`, measured `AI_READY_AFTER`, GTIN split, and autonomous/external/AI projected rates without calling AI.
- [ ] Add a new-install simulation that excludes tenant-specific answers and reports `MANUAL_CONFIGURATION_REQUIRED_ON_INSTALL`.
### Task 9: External Provider Adapter and Evaluation
**Files:** Create `central/app/product_data_provider.py`, `central/tests/test_product_data_provider.py`, `docs/EXTERNAL_PRODUCT_DATA_PROVIDER_EVALUATION_V1.md`.
**Interfaces:** Define `ProductDataProvider` protocol with `lookup_gtin(gtin) -> ProviderRecord | None`; `ProviderRecord` contains provider/version/retrieved_at/raw_hash/normalized_data/license metadata. Define `LocalBulkDatasetProvider` as an offline-only implementation; no network provider is enabled.
- [ ] Test normalization, raw hashing, provider versioning, cache-key construction, and provider-unavailable behavior.
- [ ] Implement the protocol and offline cache boundary without adding a paid dependency.
- [ ] Document GS1, official/free sources, commercial API/dataset options, terms, cache/redistribution constraints, and cost scenarios for 35k/70-client, 100k, and 500k products. Mark unknown prices as `NOT_VERIFIED` rather than inventing them.
- [ ] Set recommendation to authorization-required unless an existing licensed local source is proven.
### Task 10: Persistent AI Budget and Idempotency Controls
**Files:** Create `central/app/ai/budget_ledger.py`, `central/tests/test_ai_budget_ledger.py`, `central/tests/test_ai_idempotency.py`; modify `central/app/models.py` and add a migration only if the repository migration conventions support it.
**Interfaces:** `BudgetLimits`, `UsageEntry`, `BudgetLedger.check_and_reserve(request) -> BudgetDecision`, and `result_cache_key(product, candidates, datasets, prompt, model) -> str`.
- [ ] Test per-run, daily, monthly, request-count, and new-product limits; boundary equality is allowed, exceeding a limit returns `AI_BUDGET_EXHAUSTED` before provider invocation.
- [ ] Test atomic reservation semantics with two concurrent reservations against the same budget window.
- [ ] Implement ledger persistence using the existing central database abstraction; keep all credentials out of the ledger.
- [ ] Test identical product/candidate/dataset/prompt/model inputs produce a cache hit and zero second call; changed model or dataset version produces a new key.
- [ ] Run these tests entirely offline with fake providers and assert no network calls.
### Task 11: Final Artifacts and Project State
**Files:** Create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.md`; modify `docs/AI_PROJECT_STATE.md`.
- [ ] Generate the consolidated report from measured JSON artifacts, including per-bank autonomy, requiredness, inbound reduction, KB reuse, provider candidates/cost/licensing, AI residual, human residual, writes, second pass, tests, and blockers.
- [ ] Update project state with the actual mission status, preserving `AI_WRITE_ALLOWED=false`, `FIREBIRD_AI_WRITES=0`, retrieval V2, and all conflict counts.
- [ ] Add tests that load the final JSON and assert required keys and safety invariants.
- [ ] Run `python -m pytest tests -q` and `python -m pytest central/tests -q`.
- [ ] Run `git diff --check`, artifact invariant validation, and tracked-secret scan.
- [ ] Commit implementation in coherent milestone commits; do not tag or push.
## Final Verification
Run exactly:
```text
python -m pytest tests -q
python -m pytest central/tests -q
git diff --check
```
The final report must state measured values, not expected values. A failed guard,
unavailable inbound evidence, or unavailable external provider is a documented
residual, never an inferred fiscal answer.
@@ -0,0 +1,81 @@
# Control Panel Final Pre-Deploy Fixes V1 Implementation Plan
> **For agentic workers:** Execute inline; the operator mission authorizes autonomous completion. Steps use checkbox syntax for tracking.
**Goal:** Reconcile fiscal coverage counts, correct the fixture gap, and apply the narrowly scoped login/Agent-list/Agent-detail presentation fixes before deploy freeze.
**Architecture:** Preserve the FastAPI/Jinja2/CSS/vanilla-JS panel and fiscal/Agent behavior. Add a derived, explicitly named `unclassified_products` residual from snapshot coverage, expose a consistency flag for over-counted categories, correct only fixture counts that are known synthetic gaps, and make requested UI-only changes.
**Tech Stack:** Python 3.12, FastAPI, Jinja2, SQLAlchemy, CSS, pytest, Docker Compose local runtime.
## Global Constraints
- Do not deploy, alter fiscal engine/requiredness, call AI, enable Firebird writes, change architecture, add SPA, modify Proxmox, or integrate GTI Gerenciador.
- A product coverage state is counted at installation/snapshot grain; existing distinct blocker registry totals remain separately labeled.
- Define `UNCLASSIFIED = TOTAL - COMPLETE - SAFE_BLOCKED - FAILED_PRODUCTS` only when nonnegative; show an inconsistency warning and fail `coverage_invariant_pass` when the explicit categories exceed total.
- Do not use an unexplained generic “other” label; the residual is “Não classificados (sem estado informado)” and documentation says precisely that snapshot telemetry did not assign a known state.
- Keep Agent percentage and technical IDs in detail/API; remove UUID only from the main Agents table.
- Do not modify Firebird databases or run fixture generation against production/client data.
---
### Task 1: Prove coverage gap and add view-model/UI tests
**Files:**
- Modify `central/tests/test_operational_view_models.py`
- Modify `central/tests/test_control_panel_fixture.py`
- Modify `central/tests/test_control_panel_pages.py`
- Add focused assertions to `central/tests/test_control_plane.py` only if needed.
- [ ] Test coverage partition arithmetic for normal, residual, and over-counted snapshots.
- [ ] Test dashboard sums total/complete/snapshot-safe-blocked/failed and explicit residual; require nonnegative invariant pass only when components do not exceed total.
- [ ] Test Agent detail/list expose explicit residual and the fiscal percentage.
- [ ] Test fixture total equals complete + safe-blocked + failed + residual; assert fixture residual zero after correction.
- [ ] Test login form labels/container, Agent list does not contain technical UUID, detail shows missing fields as `Não informado`/`Desconhecida`, and no empty green anchor/button exists in company card.
- [ ] Run targeted tests and capture expected failures before implementation.
### Task 2: Implement reconciled coverage view-models and fixture
**Files:**
- Modify `central/app/control_plane.py`
- Modify `central/scripts/generate_control_panel_fixture.py`
- Modify dashboard, Agent list, and Agent detail templates.
**Interfaces:**
- Add a single helper returning `complete`, `safe_blocked`, `failed`, `unclassified`, `total`, `consistent`, and `overcount` for a snapshot.
- Agent view adds `unclassified_products`, `coverage_invariant_pass`, `coverage_overcount_products`.
- Dashboard returns `safe_blocked_coverage_products`, `unclassified_products`, `coverage_invariant_pass`, and `coverage_overcount_products`; retain registry `safe_blocked_products` as a separately labeled blocker metric if displayed.
- [ ] Implement arithmetic with explicit consistency behavior; do not clamp or silently hide over-count.
- [ ] Dashboard SQL aggregate sums snapshot fields and computes residual at presentation/query aggregation without loading products.
- [ ] Fixture sets complete to `total - safe_blocked - failed_products`; set coverage percentage from complete/total rounded one decimal.
- [ ] Add template rows/cards for safe-blocked coverage, failed technical products, and unclassified products; display inconsistency warning when needed.
- [ ] Agent list explains the remaining percentage beside complete/blocked/failed/unclassified counts.
- [ ] Run targeted model/fixture/page tests green.
### Task 3: Minimal visual fixes and technical missing values
**Files:**
- Modify `central/templates/login.html`
- Modify `central/templates/agents.html`
- Modify `central/templates/agent_detail.html`
- Modify `central/static/control-panel.css`
- Modify `central/tests/test_control_panel_pages.py`
- [ ] Add a login wrapper/card sizing and center layout, explicit `for`/`id` labels, spacing, responsive width, and existing accent colors.
- [ ] Remove the installation UUID secondary text from the Agents table; retain it in detail technical IDs/API.
- [ ] Replace empty/unknown technical displays with `Não informado` and `Desconhecida`; do not invent Firebird version/ODS or version drift.
- [ ] Investigate green empty control: confirm rendered anchor is a real reprocess link with visible text; if visually empty due styles, correct styles; if action absent, remove element. Add assertion for non-empty accessible name/text.
- [ ] Run focused page tests green.
### Task 4: Documentation and final verification
**Files:**
- Modify `docs/CONTROL_PANEL_V1.md` and/or `docs/CONTROL_PANEL_OPERATIONAL_UX_V1.md` only if required to define the residual precisely.
- [ ] Document that unclassified is the snapshot residual without a supplied operational state, not a fiscal rule.
- [ ] Run exactly `python -m pytest tests -q`, `python -m pytest central/tests -q`, and `python -m pytest agent/tests -q`.
- [ ] Rebuild only local Compose `fiscal-hub`; verify PostgreSQL healthy and Hub `/health` 200.
- [ ] Smoke `/login`, `/`, `/agents`, `/agents/{id}`; check coverage reconciliation, login, UUID removal, visible reprocess control, and missing technical values.
- [ ] Measure route latency and record maximum.
- [ ] Verify flags, no deploy/Firebird operations, diff check, commits, and clean worktree; report exact requested final fields.
@@ -0,0 +1,309 @@
# Control Panel Operational UX V1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Turn the existing server-rendered Control Panel into a bounded operational queue that identifies client/server problems, presents distinct technical/fiscal metrics, and offers safe audited context resolution.
**Architecture:** Keep FastAPI, Jinja2, project CSS, and minimal JavaScript. Extend `control_plane.py` with bounded query/view-model functions and add a focused `panel_presentation.py` for timestamps, duration, labels, priorities, and attention ordering. Use existing `FiscalContextEntry`, `Job`, `AuditEvent`, CSRF, and human authorization; do not add remote write capabilities or fiscal-engine changes.
**Tech Stack:** Python 3.12, FastAPI, Jinja2, SQLAlchemy, PostgreSQL/SQLite tests, pytest, vanilla CSS/JavaScript.
## Global Constraints
- Keep FastAPI, Jinja2, CSS próprio, and JavaScript mínimo; do not introduce React, Vue, Node build chain, or SPA.
- Do not alter fiscal logic, requiredness, Hub/Agent protocol, Firebird writes, AI writes, deployment, Proxmox, or real Agents.
- `safe_blocked_products` means distinct active tenant/product pairs; `open_blockers` means active blocker rows.
- Enforce `agents_total == online + degraded + offline + failed + unknown` in code and tests.
- UI timezone is configured by `UI_TIMEZONE`, default `America/Bahia`; storage/API timestamps remain UTC.
- Every mutating human action requires human auth, CSRF, deterministic validation, and an `AuditEvent`.
- Weak local passwords are allowed only with `ALLOW_WEAK_DEV_PASSWORD=1` outside production; production requires the strong policy.
- Never log or commit credentials, session tokens, passwords, SQL, shell commands, or raw connection strings.
- Preserve bounded pagination and avoid N+1 queries on 100-installation fixtures.
---
### Task 1: Presentation Helpers And Password Policy
**Files:**
- Create: `central/app/panel_presentation.py`
- Modify: `central/app/config.py:16-39`
- Modify: `central/app/human_auth.py:create_admin`
- Modify: `central/tests/test_control_plane.py`
- Create: `central/tests/test_panel_presentation.py`
- Modify: `docker-compose.yml` to explicitly enable weak development passwords only for the local service
**Interfaces:**
- Produces `format_ui_datetime(value) -> str`, `format_age(value, now=None) -> str`, `format_duration(started_at, finished_at, now=None) -> str`, `operational_priority(blocker) -> str`, and `health_label(value) -> str`.
- `format_ui_datetime` returns `-` for no value, a localized `dd/mm/YYYY HH:MM` string, and accepts UTC-naive values as UTC.
- `format_age` returns `-`, `agora`, `há 1 min`, `há N min`, `há 1 h`, or `há N h`; future values return `agora`.
- `format_duration` returns `-` when no start exists, `em andamento` when only started exists, and compact seconds/minutes/hours when both exist.
- `create_admin` accepts short passwords only when `settings.environment != "production"` and `ALLOW_WEAK_DEV_PASSWORD=1`; otherwise it retains the strong minimum.
- [ ] **Step 1: Write failing timestamp, duration, priority, and password-policy tests**
```python
def test_ui_times_use_configured_timezone_and_human_age(monkeypatch):
monkeypatch.setenv("UI_TIMEZONE", "America/Bahia")
value = datetime(2026, 9, 22, 19, 11, tzinfo=timezone.utc)
assert format_ui_datetime(value) == "22/09/2026 16:11"
assert format_age(value, datetime(2026, 9, 22, 19, 19, tzinfo=timezone.utc)) == "há 8 min"
def test_duration_and_priority_are_deterministic():
start = datetime(2026, 9, 22, 19, 0, tzinfo=timezone.utc)
finish = start + timedelta(seconds=194)
assert format_duration(start, finish) == "3 min 14 s"
assert operational_priority(SimpleNamespace(blocker_type="TECHNICAL_FAILURE", field="", reason_code="X")) == "CRITICAL"
def test_short_local_password_requires_explicit_setting(monkeypatch):
monkeypatch.delenv("ALLOW_WEAK_DEV_PASSWORD", raising=False)
with pytest.raises(ValueError):
create_admin(session, "admin", "admin")
monkeypatch.setenv("ALLOW_WEAK_DEV_PASSWORD", "1")
assert create_admin(session, "admin", "admin").username == "admin"
```
- [ ] **Step 2: Run the focused tests and verify they fail**
Run: `python -m pytest central/tests/test_panel_presentation.py central/tests/test_control_plane.py -q`
Expected: FAIL because the presentation module and environment-aware password gate do not yet exist.
- [ ] **Step 3: Implement the helpers and environment-aware password check**
Use `zoneinfo.ZoneInfo(settings.ui_timezone)` for display conversion. Add `ui_timezone` and `allow_weak_dev_password` configuration fields, but read the weak-password environment flag at call time so tests and explicit local process settings cannot be stale. Update the local Compose service with `ALLOW_WEAK_DEV_PASSWORD=1`; do not set it for production configuration.
- [ ] **Step 4: Run focused tests and inspect the diff**
Run: `python -m pytest central/tests/test_panel_presentation.py central/tests/test_control_plane.py -q` and `git diff --check`.
Expected: PASS with no whitespace errors and no credential values in the diff.
- [ ] **Step 5: Commit the slice**
```text
git add central/app/panel_presentation.py central/app/config.py central/app/human_auth.py central/tests/test_panel_presentation.py central/tests/test_control_plane.py docker-compose.yml
git commit -m "feat: add operational panel presentation helpers"
```
### Task 2: Bounded Operational Query View-Models
**Files:**
- Modify: `central/app/control_plane.py`
- Modify: `central/app/models.py` only if a missing relationship-free field is proven necessary; do not add a migration for presentation-only data
- Modify: `central/tests/test_control_plane.py`
- Create: `central/tests/test_operational_view_models.py`
**Interfaces:**
- Produces `dashboard_summary(session, now=None)` with `health_counts`, `agents_total`, `safe_blocked_products`, `open_blockers`, `open_blocker_breakdown`, `attention_items`, `failed_jobs`, and `outdated_agents`.
- Produces `list_installations(..., sort="display_name")` with `fiscal_completion_percent`, `technical_health`, `fiscal_status`, `version_drift`, formatted presentation fields, and `installation_id`/`agent_id` as secondary data.
- Produces `list_jobs(..., client=None, search=None)` joining `Job` to `Agent`, `Installation`, and `Tenant`, returning client/display/hostname and duration fields.
- Produces `list_blockers(..., client=None, installation=None, status="OPEN", age=None, search=None)` with summaries, priority, age, and stable pagination.
- Produces `blocker_detail(session, blocker_id)` with blocker, installation identity, related events, related jobs, and resolution audit rows.
- [ ] **Step 1: Add failing tests for dashboard reconciliation and metric semantics**
Seed a small SQLite fixture with five health states, two blockers for one product, one blocker for another product, and distinct clients. Assert that `agents_total` equals the five state sum, `safe_blocked_products == 2`, and `open_blockers == 3`.
- [ ] **Step 2: Add failing tests for attention ordering and limit**
Seed more than ten installations with mixed health, required blockers, and outdated versions. Assert that the first items are `FAILED`, then `DEGRADED`, `OFFLINE`, `UNKNOWN`, and that the result length is at most ten. Assert every item contains client, display name, hostname, problem, age, and link target.
- [ ] **Step 3: Add failing tests for agents, jobs, blockers, and detail view-models**
Assert agent rows expose percentage and separate technical/fiscal labels; job rows expose client/display/hostname and `3 min 14 s`; blocker filters reduce the result server-side; detail rows expose evidence, related job/event IDs, and audit resolution history.
- [ ] **Step 4: Implement bounded SQL/query helpers**
Use one installation join for agent/dashboard data, aggregate open blockers with `count(*)` and `count(distinct (tenant_id, product_code))` semantics compatible with SQLite/PostgreSQL, and bulk-load related job/event/audit rows for details. Keep page size capped at 100. Map blocker fields to `CFOP`, `ST`, `ICMS`, `CSOSN`, `NCM`, or `OTHER`. Never turn `safe_blocked` into technical `DEGRADED`.
- [ ] **Step 5: Run view-model tests and verify query count does not grow per row**
Run: `python -m pytest central/tests/test_operational_view_models.py central/tests/test_control_plane.py -q`
Expected: PASS; dashboard and list functions execute bounded joins without per-installation detail queries.
- [ ] **Step 6: Commit the slice**
```text
git add central/app/control_plane.py central/tests/test_control_plane.py central/tests/test_operational_view_models.py
git commit -m "feat: add operational control panel view models"
```
### Task 3: Safe Blocker Resolution Workflow
**Files:**
- Create: `central/app/blocker_resolution.py`
- Modify: `central/app/context_keys.py` only to expose the existing enum choices cleanly to the form
- Modify: `central/app/control_api.py`
- Modify: `central/app/web.py`
- Modify: `central/tests/test_control_plane.py`
- Create: `central/tests/test_blocker_resolution_panel.py`
**Interfaces:**
- `validate_resolution_input(payload) -> normalized, error` validates `key`, `value`, `scope`, `scope_key`, and `reason` through `validate_entry`, allowing only `PRODUCT`, `GROUP`, and `COMPANY` in this workflow.
- `resolve_blocker_with_context(session, blocker_id, admin, payload, request_ip=None) -> {"blocker_id": int, "job_id": int, "audit_id": int}` performs one transaction.
- Human routes are `GET /blockers/{blocker_id}` and `POST /blockers/{blocker_id}/resolve-context`; JSON API gains the same validated behavior without accepting arbitrary context text.
- [ ] **Step 1: Write failing validation, CSRF, authorization, and audit tests**
Test valid `entry_scope=INTER` and `product_role=RESALE`; reject invalid enum values, unknown keys, unsupported `NCM` workflow scope, empty reason, missing product scope key, missing CSRF, unauthenticated access, and non-admin access. Assert successful resolution creates `FiscalContextEntry`, a `REPROCESS_PRODUCTS` job, and an audit context containing before/after and job ID.
- [ ] **Step 2: Run the focused tests and verify the new route/service fails**
Run: `python -m pytest central/tests/test_blocker_resolution_panel.py -q`
Expected: FAIL because the service, detail route, and form route do not yet exist.
- [ ] **Step 3: Implement normalized validation and the transaction service**
Derive `company_key` from the installation/company context, derive product scope key from the blocker product code, read the current matching `FiscalContextEntry` as `before`, upsert the normalized entry, create a `REPROCESS_PRODUCTS` job with exactly one product code, and record a sanitized audit event with `admin_user_id`, tenant, installation, product, blocker, context key/scope/value, reason, before, after, and `reprocess_job_id`. Commit once; rollback on any validation or persistence error.
- [ ] **Step 4: Implement detail and form routes**
Render all required blocker evidence before the form, show select options for known enums, preserve CSRF, return a readable validation error on invalid input, redirect to the blocker detail after success, and remove the old `data-confirm`/hidden-context form.
- [ ] **Step 5: Run focused tests and verify security boundaries**
Run: `python -m pytest central/tests/test_blocker_resolution_panel.py central/tests/test_control_plane.py -q`.
Expected: PASS; no route accepts `AI_WRITE_ALLOWED`, `FIREBIRD_WRITES`, SQL, shell, or arbitrary context keys.
- [ ] **Step 6: Commit the slice**
```text
git add central/app/blocker_resolution.py central/app/context_keys.py central/app/control_api.py central/app/web.py central/tests/test_blocker_resolution_panel.py central/tests/test_control_plane.py
git commit -m "feat: add audited blocker context resolution"
```
### Task 4: Server-Rendered Operational Pages
**Files:**
- Modify: `central/app/web.py`
- Modify: `central/templates/base.html`
- Modify: `central/templates/dashboard.html`
- Modify: `central/templates/agents.html`
- Modify: `central/templates/agent_detail.html`
- Modify: `central/templates/blockers.html`
- Create: `central/templates/blocker_detail.html`
- Modify: `central/templates/jobs.html`
- Modify: `central/static/control-panel.css`
- Modify: `central/static/control-panel.js`
- Create: `central/tests/test_control_panel_pages.py`
**Interfaces:**
- Existing URLs remain stable: `/`, `/agents`, `/agents/{agent_id}`, `/blockers`, `/jobs`, `/login`.
- New URL: `/blockers/{blocker_id}`; POST resolution is `/blockers/{blocker_id}/resolve-context`.
- Query strings are server-side filters and remain in pagination links.
- [ ] **Step 1: Add failing page-render tests**
Log in with the test admin and assert dashboard HTML contains all five health labels, separate `Produtos bloqueados` and `Blockers abertos`, `ATENÇÃO NECESSÁRIA`, `Ver todos`, failed-job and outdated sections, and no raw ISO timestamp. Assert agents contain client/display/hostname, fiscal percentage, and separate health/status badges. Assert jobs contain client/hostname/duration. Assert blockers contain visible filter labels, summary metrics, priority, detail links, and no `window.confirm`.
- [ ] **Step 2: Run page tests and verify they fail**
Run: `python -m pytest central/tests/test_control_panel_pages.py -q`
Expected: FAIL against the current templates and routes.
- [ ] **Step 3: Implement route context and filters**
Pass the view-model results and current filter values to templates. Add agent worst-coverage sort, blocker filter parameters, job status/search parameters, event filters on agent detail, and bounded pagination links. Keep all data server-rendered.
- [ ] **Step 4: Replace templates with operational Portuguese UI**
Use accessible `<label>` elements, table `<caption>`/`<thead>` semantics, visible badges with text, empty states, detail links, technical IDs in secondary/details areas, and `title` attributes for full timestamps. Render health and fiscal badges separately.
- [ ] **Step 5: Update CSS and minimal JavaScript**
Add responsive grid breakpoints for 1366/768/tablet widths, horizontal table scrolling, visible focus styles, sufficient contrast, status styling that does not rely on color alone, and only harmless progressive enhancement. Remove the generic confirmation click handler for blocker resolution.
- [ ] **Step 6: Run page tests and commit**
Run: `python -m pytest central/tests/test_control_panel_pages.py -q`.
Expected: PASS.
```text
git add central/app/web.py central/templates central/static central/tests/test_control_panel_pages.py
git commit -m "feat: refine operational control panel pages"
```
### Task 5: Fixture Coverage, Documentation, And Regression Tests
**Files:**
- Modify: `central/scripts/generate_control_panel_fixture.py`
- Modify: `central/tests/test_control_panel_fixture.py`
- Modify: `docs/CONTROL_PANEL_V1.md`
- Create: `docs/CONTROL_PANEL_OPERATIONAL_UX_V1.md`
- Modify: `central/tests/test_control_plane.py` and focused test modules as needed
**Interfaces:**
- The existing fixture command remains explicit and non-production-only.
- Fixture includes all five health states, current/outdated/unknown versions, failed jobs, safe-blocked products, and multiple blockers for at least one product.
- [ ] **Step 1: Extend fixture tests before changing fixture data**
Assert the fixture has multiple blockers sharing a tenant/product, failed jobs, all health/version states, and enough client/server identity fields for every operational table.
- [ ] **Step 2: Update fixture deterministically**
Keep the existing 100-agent shape and return contract. Add a second blocker for selected products, varied blocker types/fields and reasons, explicit finished/started timestamps for jobs, and events carrying job/product/blocker references without secrets.
- [ ] **Step 3: Add documentation tests/contract assertions**
Check the documentation names health-vs-fiscal semantics, safe-blocked-vs-open-blocker definitions, attention ordering/limit, timezone setting, resolution audit flow, and production password policy.
- [ ] **Step 4: Update both documentation files**
Document Portuguese operational labels, URL/filter examples, the resolution workflow, supported context enums/scopes, no-write boundaries, fixture command, and `UI_TIMEZONE`/`ALLOW_WEAK_DEV_PASSWORD` configuration.
- [ ] **Step 5: Run all central tests and commit**
Run: `python -m pytest central/tests -q`.
Expected: PASS with only existing non-failing warnings.
```text
git add central/scripts/generate_control_panel_fixture.py central/tests docs/CONTROL_PANEL_V1.md docs/CONTROL_PANEL_OPERATIONAL_UX_V1.md
git commit -m "test: cover operational panel fixture and contracts"
```
### Task 6: Runtime, Performance, And Full Verification
**Files:**
- Modify only files required by failing verification; do not add unrelated refactors.
- Create no runtime artifacts containing credentials or session values.
- [ ] **Step 1: Run the complete offline regression**
Run exactly:
```text
python -m pytest tests -q
python -m pytest central/tests -q
python -m pytest agent/tests -q
```
Expected: all suites pass; record counts and non-failing warnings.
- [ ] **Step 2: Rebuild and start only the local Hub stack**
Run: `docker compose up -d --build fiscal-hub`.
Verify PostgreSQL is healthy and `GET http://localhost:8000/health` returns `200`. Do not deploy or publish additional ports.
- [ ] **Step 3: Run authenticated route smoke checks**
Using the existing local admin without printing its password or cookies, request `/`, `/agents`, `/blockers`, `/jobs`, one agent detail, one blocker detail, and submit one valid resolution form with CSRF in a disposable/local fixture context. Verify redirects, `200` responses, no raw ISO timestamps in primary HTML, and no secret values in response/log output.
- [ ] **Step 4: Run bounded performance smoke**
Against the 100-agent fixture, time dashboard, agents, blockers, jobs, and agent detail requests with a monotonic clock. Report the maximum route time in milliseconds and fail the check if any route exceeds 1000 ms locally or if a route produces an unbounded query loop.
- [ ] **Step 5: Verify safety flags and repository state**
Check `AI_WRITE_ALLOWED=false`, `FIREBIRD_AI_WRITES=0`, no remote Firebird write path, no deployment command, `git diff --check`, `git status --short`, and recent commit list. Ensure no password, token, cookie, or secret appears in tracked files or generated reports.
- [ ] **Step 6: Produce the consolidated mission report**
Report `STATUS`, all requested UX readiness flags, measured `MAX_ROUTE_TIME_MS`, test counts, PostgreSQL/Hub health, safety flags, commits, Git status, and a verdict that distinguishes completed behavior from any residual warning.
@@ -0,0 +1,198 @@
# GTI Fiscal Control Panel V1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a tested FastAPI/Jinja2 control panel that identifies every Agent installation, exposes deterministic health and coverage, centralizes redacted events, and supports safe authenticated operations.
**Architecture:** Keep the existing Hub as the only application boundary. Add focused SQLAlchemy models and query/service modules for installations, snapshots, events, human sessions, and audits. Keep Agent Bearer authentication separate from human cookie sessions, and render the operator UI with Jinja2, CSS, and small JavaScript.
**Tech Stack:** FastAPI, SQLAlchemy 2, Alembic, PostgreSQL, Jinja2, standard-library PBKDF2-HMAC password hashing, HttpOnly cookie sessions, server-rendered HTML, CSS, vanilla JavaScript.
## Global Constraints
- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` remain unchanged.
- No fiscal decision logic changes, production deployment, real-client installation, raw log upload, SQL execution, shell execution, or remote Windows restart.
- Agent authentication remains Bearer/token based; human authentication uses a separate expiring cookie session.
- `installation_id` is a persistent UUID and IP is never a primary identity.
- `SAFE_BLOCKED` does not independently make an Agent degraded or failed.
- All human mutations require authentication, CSRF validation, explicit confirmation, and an audit event.
- Dashboard queries use snapshots and bounded aggregates; list endpoints are paginated with bounded page sizes.
- Existing `tests`, `central/tests`, and `agent/tests` must remain passing.
---
## File Map
### Foundation
- Create `central/alembic/versions/0017_control_plane_foundation.py`: installation, snapshot, event, admin/session/audit tables and indexes.
- Modify `central/app/models.py`: focused models for the new tables.
- Modify `central/app/config.py`: heartbeat/offline, retention, session, cookie, and version settings.
- Create `central/app/installation_identity.py`: UUID generation, masked CNPJ/path, metadata allowlist, health/version functions.
- Create `central/app/events.py`: event type validation, redaction, size bounds, ingestion and retention.
- Create `central/app/control_plane.py`: snapshot aggregation, summary queries, filters, and pagination services.
- Create `central/app/human_auth.py`: password hashing, admin lookup, session and CSRF helpers.
- Create `central/app/audit.py`: immutable human action audit helper.
- Modify `central/app/api.py`: enriched Agent registration/heartbeat and Agent event/snapshot ingestion routes; preserve existing routes.
- Modify `central/app/main.py`: mount foundation/control-plane routes and templates/static files.
- Modify `central/app/auth.py`: registration lookup by installation identity without weakening Bearer auth.
- Modify `agent/src/gti_fiscal_agent/state.py`: persist installation UUID and local registration metadata.
- Modify `agent/src/gti_fiscal_agent/client.py`: registration, enriched heartbeat, event sending methods.
- Modify `agent/scripts/run_agent_service.py`: load stable installation identity and send structured lifecycle/heartbeat metadata.
- Create `central/scripts/create_admin.py`: interactive secure admin creation CLI.
- Create `central/scripts/maintain_events.py`: retention maintenance command.
### APIs And Web
- Create `central/app/control_api.py`: human-authenticated JSON dashboard, agents, blockers, jobs, and events endpoints.
- Create `central/app/web.py`: login/logout, dashboard/list/detail pages, safe action form handlers, CSRF and confirmation checks.
- Create `central/templates/base.html`, `login.html`, `dashboard.html`, `agents.html`, `agent_detail.html`, `blockers.html`, `jobs.html`, `events.html`.
- Create `central/static/control-panel.css`, `central/static/control-panel.js`.
- Create `central/tests/test_control_plane.py`, `test_human_auth.py`, `test_control_api.py`, `test_web_panel.py`, `test_event_security.py`.
- Create `central/scripts/generate_control_panel_fixture.py`: explicit non-production 100-Agent fixture generator.
- Create `central/tests/test_control_panel_fixture.py`: fixture distribution and bounded query checks.
- Create `docs/CONTROL_PANEL_V1.md`: architecture, auth, identity, health, events, retention, actions, endpoints, screens, operations.
- Modify `docs/AI_PROJECT_STATE.md`: record panel readiness and security invariants.
## Tasks
### Task 1: Add Persistent Control-Plane Schema
**Files:** migration and model files above.
**Interfaces:** Models expose `Installation`, `AgentSnapshot`, `AgentEvent`, `AdminUser`, `WebSession`, and `AuditEvent`; later services use `installation_id`, indexed event timestamps, snapshot counts, and session hashes.
- [ ] Write model tests asserting UUID uniqueness, one installation per Agent, snapshot upsert shape, event indexes, admin/session uniqueness, and audit append-only fields.
- [ ] Add the models with nullable metadata for backward-compatible existing Agents. Store `installation_id` as a 36-character UUID string with a unique index; store secrets only as hashes.
- [ ] Add migration `0017_control_plane_foundation` with foreign keys to `tenants`, `agents`, `jobs`, and `blocked_products` where applicable. Add indexes on tenant/installation, health/last_seen, snapshot update time, event installation/time/severity/type, job agent/status/time, and blocker tenant/type/resolution.
- [ ] Run `python -m pytest central/tests/test_control_plane.py -q` and apply migration only to the configured development PostgreSQL database when available.
- [ ] Commit `feat: add control plane foundation schema`.
### Task 2: Implement Installation Identity And Agent Metadata
**Files:** identity service, existing Agent auth/API, Agent state/client/service, tests.
**Interfaces:** `ensure_installation_id(state_path: str) -> str`, `sanitize_registration(payload: dict) -> dict`, `mask_cnpj(value: str | None) -> str | None`, `sanitize_database_path(value: str | None) -> str | None`, and `agent_registration(payload, bearer) -> dict`.
- [ ] Write failing tests proving the same local state path returns the same UUID, a new explicit installation creates a new UUID, hostname/IP changes do not change identity, and registration never stores password/token/API-key/connection-string values.
- [ ] Add an atomic local `installation_id` record to AgentState and include it in registration/heartbeat payloads. Keep the existing `agent_id` field separate.
- [ ] Extend registration/heartbeat metadata with display name, customer code, notes, OS/version, machine fingerprint, masked company CNPJ, UF/CRT, Firebird version/ODS, sanitized database identifier, capabilities, fiscal engine, datasets, started time, and last job data.
- [ ] Upsert the Hub installation record by `(agent_id, installation_id)` and reject conflicting installation IDs rather than silently reassigning an existing installation.
- [ ] Run `python -m pytest central/tests/test_control_plane.py agent/tests/test_state.py agent/tests/test_run_agent_service.py -q`.
- [ ] Commit `feat: persist agent installation identity and metadata`.
### Task 3: Implement Health, Version Drift, And Snapshots
**Files:** `central/app/installation_identity.py`, `central/app/control_plane.py`, config, heartbeat/job completion routes, tests.
**Interfaces:** `compute_agent_health(snapshot, now) -> str`, `compute_version_drift(agent_version, server_version) -> str`, `upsert_agent_snapshot(session, installation, payload) -> AgentSnapshot`, `dashboard_summary(session, filters) -> dict`, and `paginate(query, page, page_size) -> dict`.
- [ ] Write tests for `UNKNOWN`, recent healthy `ONLINE`, recent technical failure `DEGRADED`, persistent technical failure `FAILED`, stale `OFFLINE`, and safe-blocked-only `ONLINE`.
- [ ] Implement configurable `AGENT_HEARTBEAT_INTERVAL_SECONDS`, `AGENT_OFFLINE_AFTER_SECONDS` with a three-interval default, `CURRENT_SERVER_VERSION`, and `AGENT_EVENT_RETENTION_DAYS=90`.
- [ ] Persist product counts, coverage, failed products, blocker aggregates, last/last-successful job, heartbeat, health, and version drift in one upserted snapshot.
- [ ] Add dashboard aggregate queries that only read snapshot rows plus bounded grouped counts. Normalize datetimes as UTC for SQLite and PostgreSQL tests.
- [ ] Run health/snapshot tests and existing job tests; verify `SAFE_BLOCKED` does not degrade health.
- [ ] Commit `feat: add deterministic agent health and snapshots`.
### Task 4: Add Structured Events, Redaction, Retention, And Audit
**Files:** `central/app/events.py`, `central/app/audit.py`, models/migration if needed, API routes, maintenance CLI, tests.
**Interfaces:** `sanitize_event(message, context) -> tuple[str, dict]`, `record_event(session, *, installation_id, severity, event_type, message, context, job_id=None, product_code=None, blocker_id=None) -> AgentEvent`, `purge_expired_events(session, now, retention_days) -> int`, and `record_audit(session, actor, action, target, context) -> AuditEvent`.
- [ ] Write tests covering case-insensitive redaction of passwords, tokens, API keys, connection strings and secret env names in both messages and nested JSON; assert event message/context maximums.
- [ ] Implement allowlisted event types and severities from the mission. Store structured context only, truncate safely, and never persist raw log files.
- [ ] Record Agent lifecycle, heartbeat, preflight, job, blocker, dataset/schema, connection, retry, version, and write-audit events at the relevant existing route boundaries.
- [ ] Implement retention deletion for non-audit events older than `AGENT_EVENT_RETENTION_DAYS`; preserve audit events, blocker history, write audit, and critical events.
- [ ] Add `python -m central.scripts.maintain_events --dry-run` and a write mode that only deletes eligible events.
- [ ] Run event security and retention tests, then commit `feat: add redacted operational events and retention`.
### Task 5: Implement Human Admin Authentication And Session Security
**Files:** `central/app/human_auth.py`, `central/app/web.py`, config/main, CLI, tests.
**Interfaces:** `hash_password(password) -> str`, `verify_password(password, encoded_hash) -> bool`, `create_admin(session, username, password) -> AdminUser`, `create_web_session(session, admin, user_agent, ip) -> tuple[str, str]`, `require_human(request) -> AdminUser`, and `require_csrf(request, submitted_token) -> None`.
- [ ] Write tests for PBKDF2 password storage, wrong-password rejection, inactive admin rejection, session expiry, cookie flags, session rotation, logout invalidation, and CSRF mismatch/replay rejection.
- [ ] Add PBKDF2-HMAC-SHA256 hashes with random salts and constant-time verification using only the standard library. Store only session token hashes and CSRF token hashes server-side.
- [ ] Add `gti_admin_session` HttpOnly cookie with `Secure` according to production config, `SameSite=Lax`, bounded max age, and no Agent token reuse.
- [ ] Add `python -m central.scripts.create_admin` using `getpass`, refusing empty/mismatched passwords and duplicate usernames.
- [ ] Run auth tests and commit `feat: add separate human admin sessions`.
### Task 6: Add Control-Plane APIs And Safe Actions
**Files:** `central/app/control_api.py`, API integration, audit/events, tests.
**Interfaces:**
```text
GET /v1/dashboard/summary
GET /v1/agents
GET /v1/agents/{agent_id}
GET /v1/agents/{agent_id}/health
GET /v1/agents/{agent_id}/coverage
GET /v1/agents/{agent_id}/blockers
GET /v1/agents/{agent_id}/jobs
GET /v1/agents/{agent_id}/events
GET /v1/blockers/global
GET /v1/jobs/global
POST /v1/agents/{agent_id}/actions/reprocess
POST /v1/blockers/{blocker_id}/resolve-human
```
- [ ] Write tests for authentication, client/health/version/last-seen/blocker/severity/date filters, bounded pagination, empty pages, and cross-tenant access denial.
- [ ] Implement query services with `page`, `page_size`, `total`, `items`, max page size 100, explicit sort allowlists, and no relationship N+1 queries.
- [ ] Implement safe actions only: create idempotent reprocess jobs, resolve only with explicit resolution context, and record actor/action/target/context audit events.
- [ ] Reject requests that attempt Firebird writes, AI flags, credentials, database paths, CRT changes, SQL, shell, restart, or arbitrary action names.
- [ ] Run API tests and commit `feat: expose authenticated control plane APIs`.
### Task 7: Build Server-Rendered Operational UI
**Files:** templates, CSS/JS, `central/app/web.py`, main, tests.
**Interfaces:** browser routes `/login`, `/logout`, `/`, `/agents`, `/agents/{agent_id}`, `/blockers`, `/jobs`, and `/events`; all mutations use POST forms with hidden CSRF and a visible confirmation field.
- [ ] Write template tests asserting dashboard cards, client/display-name/hostname identity, health indicator, blocker/job/event pagination, technical IDs only in detail views, and no secret values in rendered HTML.
- [ ] Implement dashboard cards for Agent health, product coverage, technical failures, recent failed jobs, stale heartbeats, version drift, and top blocker types.
- [ ] Implement responsive Agents/Servers table with search/sort/filter/page controls and links to detail tabs: overview, fiscal, blockers, jobs, events, capabilities, technical info.
- [ ] Implement global blockers/jobs/event pages and per-installation timelines with bounded pagination and filters.
- [ ] Add plain CSS responsive layout, recognizable health badges, compact tables, confirmation dialogs, and minimal JavaScript for filters/confirmations only.
- [ ] Run `python -m pytest central/tests/test_web_panel.py -q` and commit `feat: add server-rendered control panel`.
### Task 8: Add Fixture Generator And Performance Checks
**Files:** fixture script, tests, optional fixture helpers.
**Interfaces:** `generate_fixture(session, agent_count=100, seed=...) -> dict`; must be explicit and never run during app startup or migration.
- [ ] Write tests asserting exactly 100 installations with distributed `ONLINE`, `DEGRADED`, `OFFLINE`, and `FAILED` states, blockers, jobs, and events.
- [ ] Implement deterministic fixture generation behind an explicit `--database-url` and `--confirm-non-production` guard; refuse production APP_ENV and refuse empty confirmation.
- [ ] Add query-count/performance checks that dashboard uses snapshot rows and event/job pages issue bounded queries for 100 Agents and large event counts.
- [ ] Run fixture tests without seeding the configured production database and commit `test: add control panel fixture and query checks`.
### Task 9: Document, Run Full Verification, And Consolidate
**Files:** `docs/CONTROL_PANEL_V1.md`, `docs/AI_PROJECT_STATE.md`, final tests.
- [ ] Document architecture, identity lifecycle, metadata allowlist, health thresholds, event taxonomy/redaction, retention command, auth/session/CSRF, safe actions, endpoints, screens, and fixture usage.
- [ ] Update project state with control-plane readiness and explicit `AI_WRITE_ALLOWED=false`, `FIREBIRD_AI_WRITES=0`, `DEPLOY_PERFORMED=false`.
- [ ] Run `python -m pytest tests -q`.
- [ ] Run `python -m pytest central/tests -q`.
- [ ] Run `python -m pytest agent/tests -q`.
- [ ] Run `git diff --check`, inspect `git status`, and verify no secrets or generated fixture database are tracked.
- [ ] Commit `docs: document control panel v1 operations`.
## Verification Matrix
| Requirement | Plan task | Evidence |
|---|---:|---|
| Persistent installation identity and metadata | 1-2 | identity tests, registration tests, migration |
| Deterministic health/offline/version drift | 3 | health/snapshot tests |
| Coverage and blocker summaries | 3, 6-7 | snapshot/API/template tests |
| Structured redacted events | 4 | redaction, size, ingestion tests |
| Retention and audit preservation | 4 | maintenance tests |
| Human auth, cookie session, CSRF | 5 | auth/session tests |
| Paginated filtered APIs and safe actions | 6 | API and audit tests |
| Dashboard and operational screens | 7 | template and route tests |
| 100-Agent simulation/performance | 8 | fixture/query checks |
| Documentation and regression | 9 | docs and three suites |
@@ -0,0 +1,249 @@
# Agent Installation Identity Recovery and First Heartbeat Implementation Plan
> **For agentic workers:** Use `superpowers:executing-plans` to implement this plan task-by-task. Steps use checkbox syntax for tracking.
**Goal:** Recover and persist Agent 2's existing Hub `installation_id`, send exactly the first safe heartbeat, and stop without claiming jobs or writing Firebird.
**Architecture:** The Hub exposes the already-linked installation identity over an authenticated, read-only endpoint. New enrollments return that same ID so Setup can persist it; legacy installations recover it at runtime and commit it into the existing SQLite identity table before heartbeat. `AGENT_JOBS_ENABLED=0` skips the job API entirely during activation.
**Tech Stack:** FastAPI, SQLAlchemy, pytest, Python `fdb`, SQLite local state, PyInstaller onedir service host, Windows SCM / PowerShell.
## Global Constraints
- Keep Agent ID `2`; do not create an Agent, reenroll, or create a second Installation.
- Never generate a fallback `installation_id` for an already-registered Agent.
- Do not call `/v1/jobs/next`, classification endpoints, or any job execution path during activation.
- Preserve `FIREBIRD_WRITES=0`, `FISCAL_MODE=DRY_RUN`, and empty `AUTHORIZED_WRITE_DATABASE_PATH`.
- All Firebird connections used by preflight remain read-only; issue no Firebird DML/DDL.
- Do not rerun Setup.exe or replace `agent.env` / `credentials.bin` during service-host update.
- Keep published `GTIFiscalAgent-Setup-1.1.1.exe` immutable; update only the installed service runtime.
- Never print or log `FB_PASSWORD`, the Agent bearer token, or credential-file contents.
- Do not commit unless the operator explicitly requests a commit.
---
### Task 1: Hub installation identity response
**Files:**
- Modify: `central/app/api.py`
- Test: `central/tests/test_jobs.py`
**Interfaces:**
- Add authenticated `GET /v1/agents/me/identity`.
- Return exactly `{agent_id, installation_id, hostname, version, status}`; `installation_id` and `hostname` are `None` if the Agent has no Installation.
- Do not change the existing `GET /v1/agents/me` response.
- [ ] **Step 1: Write the failing identity endpoint tests**
Add tests that create an Agent with one Installation (`installation_id` fixed UUID, hostname `DESKTOP-B0HL02I`, version `1.1.1`) and assert:
```python
response = client.get("/v1/agents/me/identity", headers=headers)
assert response.status_code == 200
assert response.json() == {
"agent_id": agent.id,
"installation_id": installation.installation_id,
"hostname": "DESKTOP-B0HL02I",
"version": "1.1.1",
"status": "OFFLINE",
}
```
Also assert unauthenticated requests return 401 and an Agent without an Installation returns HTTP 200 with nullable identity fields, without creating an Installation.
- [ ] **Step 2: Run the focused tests and verify failure**
Run: `python -m pytest central/tests/test_jobs.py -q`
Expected: the identity route returns 404 before implementation.
- [ ] **Step 3: Implement the authenticated endpoint**
Use the existing `principal` dependency, query `Installation` by `agent.id`, and derive status with the same online-threshold rule as `/v1/agents/me`. The route must not commit or mutate database state.
- [ ] **Step 4: Run the focused tests**
Run: `python -m pytest central/tests/test_jobs.py -q`
Expected: all tests pass; existing `/v1/agents/me` tests remain unchanged and green.
### Task 2: Return and persist identity during new enrollment
**Files:**
- Modify: `central/app/agent_distribution.py`
- Test: `central/tests/test_install_code_enrollment.py`
- Modify: `installer/install_common.py`
- Test: `tests/test_installer_artifacts.py`
**Interfaces:**
- Successful `/v1/agents/enroll` response adds `installation_id=enr.enrollment_uuid`.
- `write_agent_env(install_dir_path, enroll_result)` writes `INSTALLATION_ID` as a non-secret field. It never writes `agent_token` or `AGENT_TOKEN`.
- [ ] **Step 1: Add failing enrollment and Setup-env assertions**
In the existing install-code happy-path test, assert `body["installation_id"] == uuid`. In the installer artifact test, pass a fixed ID such as `00000000-0000-4000-8000-000000000222` and assert the env includes it while the token is absent.
- [ ] **Step 2: Run focused tests and verify failure**
Run: `python -m pytest central/tests/test_install_code_enrollment.py::test_enroll_with_install_code_alone_creates_agent tests/test_installer_artifacts.py -q`
Expected: failures show the response/env currently omit `installation_id`.
- [ ] **Step 3: Add the returned ID and env serialization**
Add `"installation_id": enr.enrollment_uuid` to the successful enrollment JSON. Add `INSTALLATION_ID` to `write_agent_env`; do not change credential storage or Firebird guard values.
- [ ] **Step 4: Run focused tests**
Run the same command as Step 2.
Expected: enrollment and installer artifact tests pass; `agent.env` remains token-free with `FIREBIRD_WRITES=0` and `FISCAL_MODE=DRY_RUN`.
### Task 3: Persist or recover identity before heartbeat
**Files:**
- Modify: `agent/src/gti_fiscal_agent/state.py`
- Modify: `agent/src/gti_fiscal_agent/client.py`
- Modify: `agent/scripts/run_agent_service.py`
- Test: `agent/tests/test_state.py`
- Test: `agent/tests/test_run_agent_service.py`
**Interfaces:**
- `AgentState.installation_id(candidate: str | None = None) -> str` returns the persisted UUID; if no UUID is stored, it persists a valid candidate; if neither exists, it raises instead of generating a random UUID. A different candidate from an already-persisted UUID raises.
- `HubClient.identity() -> dict` calls authenticated `GET /v1/agents/me/identity`.
- `resolve_installation_id(config, env, state, client) -> str` selects persisted state, then `INSTALLATION_ID`, then authenticated identity; verifies returned `agent_id` equals `config.agent_id`; persists before returning.
- [ ] **Step 1: Write failing state identity tests**
Cover: empty store plus valid candidate persists and returns it; second call returns the same ID; conflicting candidate raises; missing candidate does not generate; malformed UUID raises.
- [ ] **Step 2: Run state tests and verify failure**
Run: `python -m pytest agent/tests/test_state.py -q`
Expected: the missing identity case currently generates a UUID and the persistence API is absent.
- [ ] **Step 3: Implement deterministic SQLite identity persistence**
Validate candidate values with `uuid.UUID`. Within a SQLite transaction, read the existing row, compare any candidate, insert only when absent, and return the single stored canonical UUID. Remove random UUID creation from the registered-agent path.
- [ ] **Step 4: Write failing runtime bootstrap tests**
Use a fake Hub identity response and fake state. Assert missing local identity calls `identity()` once, verifies matching Agent ID, persists the returned UUID, and uses it in heartbeat. Assert a mismatched Agent ID, null ID, or identity request error fails before heartbeat. Assert an already-persisted ID avoids the identity request.
- [ ] **Step 5: Run runtime tests and verify failure**
Run: `python -m pytest agent/tests/test_run_agent_service.py -q`
Expected: current loop calls the random-UUID-producing state method and has no authenticated identity bootstrap.
- [ ] **Step 6: Implement identity bootstrap before the heartbeat loop**
Add `HubClient.identity()`, implement `resolve_installation_id`, and invoke it after constructing the Hub client/state but before entering the loop. An identity failure must exit before heartbeat and before job polling.
- [ ] **Step 7: Run focused state/runtime tests**
Run: `python -m pytest agent/tests/test_state.py agent/tests/test_run_agent_service.py -q`
Expected: persisted/recovered identity tests pass and mismatches fail closed.
### Task 4: Disable job polling without disabling heartbeat
**Files:**
- Modify: `agent/scripts/run_agent_service.py`
- Test: `agent/tests/test_run_agent_service.py`
- Modify on this host only: `C:\Program Files\GTI\FiscalAgent\agent.env`
**Interfaces:**
- `Config.agent_jobs_enabled` parses `AGENT_JOBS_ENABLED`, default `True`.
- When false, the loop sleeps for `poll_interval_seconds` and continues after heartbeat without calling `HubClient.next_job()`.
- [ ] **Step 1: Add the failing no-poll test**
Use a fake Hub whose `next_job()` raises if called. Make its first heartbeat set the external stop flag. Run one loop cycle with `AGENT_JOBS_ENABLED=0` and assert one heartbeat, zero `next_job` calls, and zero job/classification calls.
- [ ] **Step 2: Run the focused test and verify failure**
Run: `python -m pytest agent/tests/test_run_agent_service.py -q`
Expected: the current loop calls `next_job()` immediately after heartbeat.
- [ ] **Step 3: Implement the gate with a backward-compatible default**
Parse `AGENT_JOBS_ENABLED` as enabled unless its stripped, uppercased value is `0`, `FALSE`, or `NO`. After successful heartbeat, if disabled, sleep once and continue before constructing or calling the job endpoint.
- [ ] **Step 4: Run focused runtime tests**
Run: `python -m pytest agent/tests/test_run_agent_service.py -q`
Expected: disabled polling makes no `next_job` request; default configuration retains the existing job behavior.
### Task 5: Full regressions and guard review
**Files:**
- Verify: `central/tests/`, `agent/tests/`, `tests/`
- [ ] **Step 1: Run Hub and Agent regressions**
Run: `python -m pytest central/tests -q` and `python -m pytest agent/tests -q`.
Expected: both suites pass.
- [ ] **Step 2: Run repository offline regressions**
Run: `python -m pytest tests -q`.
Expected: all tests pass with only the repository's pre-existing skip.
- [ ] **Step 3: Verify write and secret invariants**
Assert tests and source still guarantee `FIREBIRD_WRITES=0`, `FISCAL_MODE=DRY_RUN`, empty `AUTHORIZED_WRITE_DATABASE_PATH`, no token in `agent.env`, and identity endpoint contains only the approved fields. Do not connect to Firebird for this check.
### Task 6: Deploy the API and update only the installed service runtime
**Files/artifacts:**
- Deploy Hub source: `central/app/api.py`, `central/app/agent_distribution.py`
- Build artifact: `dist/installer/GTIFiscalAgentService/` only
- Existing install: `C:\Program Files\GTI\FiscalAgent\GTIFiscalAgentService\`
- [ ] **Step 1: Deploy the Hub code without a migration**
Use the approved production deployment path and preserve the existing Hub `.env`, PostgreSQL volume, release volume, SSO, and DNS. Build/restart only `fiscal-hub`; verify `/health`, `/version`, and authenticated `/v1/agents/me/identity` return Agent 2's already-existing identity. Do not query the database directly to retrieve the ID.
- [ ] **Step 2: Build only the service-host onedir**
Record SHA-256 of `dist/installer/GTIFiscalAgent-Setup-1.1.1.exe` before building. Verify no unversioned `GTIFiscalAgent-Setup.exe` exists, then run `scripts/build_windows_installer.ps1 -SkipSetup -SkipUninstall`. Confirm the versioned Setup hash is unchanged and no Setup.exe is run.
- [ ] **Step 3: Back up and replace only the service runtime**
With `GTIFiscalAgent` stopped, copy the current installed `GTIFiscalAgentService` directory to a timestamped local backup. Replace only that directory with the newly built service onedir. Leave `agent.env`, `credentials.bin`, `Uninstall.exe`, registry/ARP entries, and state files untouched.
- [ ] **Step 4: Prepare the existing installation for heartbeat-only operation**
Elevated-edit `agent.env` to set `AGENT_JOBS_ENABLED=0`, preserving `FIREBIRD_WRITES=0`, `FISCAL_MODE=DRY_RUN`, the existing Agent ID, Firebird configuration, and SYSTEM/Administrators ACL. Do not put the bearer token in this file.
- [ ] **Step 5: Verify recovery before SCM start**
Run the installed runtime's read-only identity bootstrap/preflight with the existing DPAPI credential. Confirm recovered `agent_id=2`, `installation_id` equals the endpoint value, the SQLite state contains that same ID, and jobs are disabled. If any check fails, leave the service stopped.
### Task 7: First heartbeat only, then stop
**Files:**
- Existing Windows service: `GTIFiscalAgent`
- Existing Hub identity endpoint: `GET /v1/agents/me/identity`
- [ ] **Step 1: Capture a baseline**
Before service start, record the existing local `ipi_write_log` row count and confirm service state is STOPPED. Do not call `/v1/jobs/next` or submit classification requests.
- [ ] **Step 2: Start the existing service**
Run `sc.exe start GTIFiscalAgent`; wait for SCM state RUNNING and verify the identity endpoint shows `agent_id=2`, the same stored `installation_id`, correct hostname, version `1.1.1`, and status ONLINE.
- [ ] **Step 3: Stop immediately after first heartbeat**
Run `sc.exe stop GTIFiscalAgent`; verify STOPPED. Keep `AGENT_JOBS_ENABLED=0` in `agent.env`.
- [ ] **Step 4: Verify zero jobs and writes**
Compare local `ipi_write_log` count with baseline; assert unchanged and equal to zero. Confirm no classification request or job was sent, `FIREBIRD_WRITES=0`, and `WRITE_ATTEMPTS=0` / `WRITE_SUCCESSES=0`. Verify authenticated identity still maps to Agent 2 and the original installation ID; do not create or enroll anything.
- [ ] **Step 5: Verify user-visible status if panel access is available**
Confirm Agent 2 displays ONLINE with the host name and version 1.1.1. If the panel requires unavailable human authentication, report the authenticated identity endpoint evidence and leave the panel UI item as a blocker; do not bypass authentication.
## Rollback and stop gates
- If Hub identity response is absent, mismatched, or does not name Agent 2, stop before writing the local identity or starting SCM.
- If Hub deployment or service-host replacement fails, restore the backed-up service directory and leave the service stopped.
- If `AGENT_JOBS_ENABLED=0` is not verified, do not start the service.
- If preflight, identity, hostname, version, or write guards fail, stop and report; do not retry enrollment, generate another UUID, run a job, or write Firebird.
@@ -0,0 +1,114 @@
# GTI FISCAL WINDOWS INSTALLER EXE V1 — Implementation Plan
- Date: 2026-09-23
- Spec: `docs/superpowers/specs/2026-09-23-windows-installer-exe-v1-design.md`
- Branch strategy: work on `master` in focused commits (repo convention; no remote).
## Order of work (TDD front-loaded)
Phase A — Backend install_code (code-first, tests first)
1. `central/app/enrollment.py`: add Crockford Base32 install-code module
(`generate_install_code`, `normalize_install_code`, `install_code_digest`,
`valid_install_code`). 16 chars, alphabet `0123456789ABCDEFGHJKMNPQRSTVWXYZ`,
grouped `XXXX-XXXX-XXXX-XXXX`.
2. Migration `0023_install_codes.py` on `agent_enrollments`:
`install_code_hash`, `install_code_expires_at`, `install_code_attempts`,
`install_code_created_at`. Update `AgentEnrollment` model.
3. `central/app/agent_distribution.py` — `POST /v1/agents/enroll`:
- Accept `install_code` (primary) OR legacy (`installation_id` +
`enrollment_token`).
- If both bodies present → 400 `INVALID_REQUEST` (ENROLLMENT_REJECTED).
- Install-code path resolves Installation by hash alone; validates status,
expiry, one-time, attempts, release PUBLISHED; mints agent+installation,
marks code used (hash cleared after success → replay-proof), audits
`ENROLLMENT_STARTED/SUCCEEDED/REJECTED` (never the code).
- Rate limit: keep per-IP `enroll` bucket AND per-code/installation bucket
(add `record_attempt("enroll_code:<enrollment_uuid>", ip)`).
- `install_code_attempts` increments on each failed attempt.
4. `central/app/web.py` + templates:
- `agent_pending_detail.html` / `agent_detail.html`: Código de instalação
`XXXX-XXXX-XXXX-XXXX` + [Copiar], Expira em, [Baixar instalador (.EXE)],
[Gerar novo código], [Revogar]. No raw tokens.
- POST `/agents/pending/{uuid}/code` → generate/rotate code (audit
`INSTALLATION_CODE_CREATED`); POST `/agents/pending/{uuid}/revoke` already
exists → also clear code + audit `INSTALLATION_CODE_REVOKED`.
- `agent_releases.html` main button → EXE.
5. Downloads serve universal EXE: `build_and_serve` → when release artifact is
an EXE, serve the stored EXE bytes (no bundle zip / no enrollment.json for the
EXE flow). Filename for EXE = stored release filename. Audit
`INSTALLER_DOWNLOADED`.
6. `central/scripts/publish_agent_release.py`: generalize `--zip` to `--artifact`
(accept `.zip` or `.exe`); filename derivation handles `GTIFiscalAgent-Setup-<v>.exe`;
enforce immutability (refuse replacing a PUBLISHED artifact). `published_at`
fix already in place (commit 21c9261).
7. Tests in `central/tests/test_agent_distribution.py` + new
`test_install_codes.py` (RED→GREEN). Full suite green.
Phase B — Agent runtime credential isolation
8. `agent/src/gti_fiscal_agent/credential_store.py`: DPAPI LocalMachine
encrypt/decrypt (`win32crypt.CryptProtectData`/`CryptUnprotectData` with
`CRYPTPROTECT_LOCAL_MACHINE`), ACL enforcing (SYSTEM/Administrators),
plaintext-token fallback only in dev (env `GTI_ALLOW_PLAINTEXT_CREDENTIAL=1`).
9. `run_agent_service.py` Config: read AGENT_TOKEN from env OR decrypt from
`credentials.bin` (`GTI_CREDENTIAL_FILE`, default Program Files path);
keep `agent.env` non-secret; enforce FIREBIRD_WRITES=0 (already fail-closed).
10. Tests: credential_store round-trip + no-write enforcement.
Phase C — Installer (Windows build host = this dev machine)
11. `installer/` package (self-contained sources bundled by PyInstaller):
- `installer/setup_main.py` (uac-admin entitlement via PyInstaller
`--uac-admin`): tkinter UI with single code field; fixed prod Hub URL
(dev override only via `GTI_HUB_URL` env/CLI); POST enroll; write
`agent.env` (non-secret) + `credentials.bin` (DPAPI LocalMachine + ACL);
extract bundled Service host + Uninstall.exe into Program Files;
register service + recovery `sc.exe failure ...`; report preflight;
write ARP key; best-effort `WINDOWS_INSTALL_COMPLETED`.
- `installer/service_main.py`: minimal Windows Service host entry reusing
`gti_agent_windows_service` logic (token from credential_store).
- `installer/uninstall_main.py`: stop/remove service, remove Program Files
tree + ARP, remove credentials.
12. `scripts/build_windows_installer.ps1`: clean dist/installer, read
`release/VERSION` (1.1.1), PyInstaller builds (Setup onefile --uac-admin,
service host onedir, uninstall onefile), embed service+uninstaller as data
in Setup, write `.sha256`. Fails on missing tool/dep.
Reproducibility = automated/repeatable + immutable release (not byte-identical
hashes).
13. Local build + smoke: produce EXE; validate layout, sha256, no credential in
`agent.env`, FIREBIRD_WRITES=0.
Phase D — Local E2E (this Windows dev machine)
14. Run Setup against a LOCAL Hub (control-plane image, scratch postgres or
SQLite test env) for the UI/install flow: invalid code → clear error; valid
code → enroll → verify utils/env/credential/service registration; reinstall
idempotence; uninstall.
(Real production E2E happens after deploy, Phase F.)
Phase E — Production deploy + release
15. Backup: pg_dump of `gti_fiscal` + release storage volume copy (via container)
to backup location; validate restore (EXE + metadata) from the backup copy.
16. Build + deploy Hub image (migration 0023 via `alembic upgrade head`);
postgres intact; smoke: health, panel, downloads.
17. Publish release 1.1.1 EXE (immutable), `AGEING NOT overwritten`: 1.1.0 zip
untouched.
Phase F — Production E2E (REAL Windows)
18. Create E2E installation in production panel; generate code; download real
EXE from `https://gtifiscal.gtigrupo.com.br`; run on this real Windows host,
enter code, enroll. Verify Installation→ENROLLED, Agent created,
`credentials.bin` DPAPI-protected (ACL), `agent.env` no token, service
registered (Automatic/Delayed, recovery, NOT started since Firebird blank),
WINDOWS_INSTALL_COMPLETED audit.
19. Rejection E2E: invalid code, expired code, replayed/used code, code+token
together → 400. Each audited.
20. Uninstall via Uninstall.exe; revoke/deactivate E2E agent in panel; audit
preserved.
Phase G — Report
21. Final report in the mission's exact field list + COMMITS + GIT_STATUS.
## Verification gates
- Entire `central/tests` suite green: `python -m pytest central/tests -q` and
`python -m pytest tests -q` (repo-level offline tests) after each phase.
- Migration head check: `0023` newest; fresh-DB upgrade test green.
- Artifacts: `dist/installer/GTIFiscalAgent-Setup-1.1.1.exe` + `*.sha256`.
- No `AGENT_TOKEN` literal anywhere in `agent.env`/installed tree.
@@ -0,0 +1,77 @@
# GTI Fiscal Operational UI Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. Work inline; the operator authorized proceeding without additional design confirmation.
**Goal:** Make Resumo, Agents, Blockers and Jobs a fast, accessible, PT-BR operational workflow where each important aggregate reaches a filtered detail in one click, with revoked/inactive history excluded from live indicators.
**Architecture:** A single SQLAlchemy active-installation predicate is shared by dashboard aggregates and paged list queries. Small presenter helpers translate labels; Jinja templates render native links and bounded summaries using Gerenciador tokens. Routes preserve GET querystrings and offer explicit active/all history scopes.
**Tech Stack:** Existing FastAPI, SQLAlchemy, Jinja, CSS tokens, minimal vanilla JS, pytest, Playwright, `scripts/deploy_hub.ps1`.
## Global Constraints
- `OPERATIONALLY_ACTIVE`: `Tenant.active IS TRUE AND Agent.enabled IS TRUE`, excluding linked `AgentEnrollment` status `REVOKED`, `FAILED`, `EXPIRED`; no inference from client/installation names.
- `Installation.processing_enabled=false` means paused jobs, not inactive server; keep it in health counts. Agents without linked enrollment remain active if Tenant and Agent are active.
- Preserve history via `scope=all`; default `scope=active` across operational lists. Existing resolved blocker/history URLs remain reachable.
- Preserve GTI Gerenciador client combobox, SSO, themes, HTTPS, Agent/runtime, fiscal engine and `FIREBIRD_WRITES=0`; no deletion of historical data.
- All filtering and pagination server-side (50/page); bounded SQL aggregates before LIMIT 10; no N+1 or client-side loading of products.
- Do not commit unless explicitly asked. Deploy exclusively through `scripts/deploy_hub.ps1`; no ad-hoc SSH/deploy script.
---
### Task 1: Shared active scope and paginated inventory
**Files:** `central/app/control_plane.py`, `central/tests/test_operational_view_models.py`, `central/app/web.py`.
**Interfaces:** `operational_installation_predicate()` yields a SQLAlchemy condition on Tenant/Agent/Installation, using `~exists(select(AgentEnrollment.id).where(AgentEnrollment.agent_id == Agent.id, AgentEnrollment.status.in_(["REVOKED", "FAILED", "EXPIRED"])))`. `list_installations(..., scope="active", version=None)` accepts `scope=active|all`; `version="outdated"` restricts to outdated versions. Web route validates scope/health/sort/version before querying and passes `query_base` with these filters.
- [ ] RED: seed active, Tenant.active=False, Agent.enabled=False, ENROLLED and REVOKED linked cases and assert default `list_installations` contains only active ids; `scope="all"` contains every id and preserves `health` filtering. Assert a `processing_enabled=False` active installation remains visible. Run `python -m pytest central/tests/test_operational_view_models.py -q` and confirm missing scope behavior fails.
- [ ] GREEN: add `AgentEnrollment` import/predicate, apply same predicate before both paged list and count; support `version="outdated"` via existing drift expression. Preserve SQL `LIMIT/OFFSET` and existing joins. Re-run focused test.
- [ ] RED/GREEN: web TestClient `/agents?scope=active&health=UNKNOWN` and `/agents?scope=all` reflect selected filter, pagination links preserve scope; implement route validation and query_base, rerun `python -m pytest central/tests/test_control_panel_pages.py -q`.
### Task 2: Operational dashboard aggregates and grouped attention
**Files:** `central/app/control_plane.py`, `central/tests/test_operational_view_models.py`.
**Interfaces:** `dashboard_summary` returns same keys as today plus grouped `attention_items` (<=10) with `group_count`, `products_affected`, `link`, `priority` and client/server identity. Blocker totals and failed-job counts only include open/current records attributed to active installations. Limit candidate groups after SQL `GROUP BY (installation_id, field, blocker_type, reason_code)`; lower-case fields normalized to upper-case for breakdown. Dashboard does not load all blockers/products.
- [ ] RED: seed revoked Agent enrollment with UNKNOWN snapshot, inactive tenant, disabled Agent, 26 NCM blockers on one active installation plus historical resolved blockers; assert active health/coverage/blocker/job aggregates exclude inactive rows, NCM becomes one attention group with 26 products, and `len(attention_items) <= 10`. Run focused operational test and observe incorrect counts/duplicate rows.
- [ ] GREEN: reuse Task 1 predicate for health/snapshots/outdated/failed jobs. For blocker aggregation join Tenant→Installation→Agent, filter active/open; count distinct `(tenant_id, product_code)` and compute grouped fields before `.limit(10)`, then order technical alerts before fiscal groups and outdated. Keep a bounded fixed number of SQL queries independent of product volume. Re-run focused test.
- [ ] Measure number of executed SQL statements with SQLAlchemy `before_cursor_execute` for 1 vs 26 blockers; assert bounded query count and no statement per blocker. Compare SQL GROUP BY query on SQLite and PostgreSQL dialect compilation; inspect indexes from `blocked_products(installation_id,resolved_at,field)` and `jobs(agent_id,status)` before adding migration.
### Task 3: Scoped Blockers and Jobs data models
**Files:** `central/app/control_plane.py`, `central/app/web.py`, `central/tests/test_operational_view_models.py`, `central/tests/test_control_panel_pages.py`.
**Interfaces:** `list_blockers`/`blocker_filter_summary(..., scope="active")` and `list_jobs(..., scope="active")` reuse Task 1 active condition; `jobs_status_summary(..., scope="active")` returns counts PENDING/RUNNING/COMPLETED/FAILED using one GROUP BY. Web GET routes accept scope, keep valid filters/query_base on pagination, and expose a selected `job_id` result with untruncated error. `status=RESOLVED&scope=all` retains historic blocker access.
- [ ] RED: tests for active defaults excluding revoked/inactive records, `scope="all"` preserving history, `status="RESOLVED"`, `/blockers?field=NCM&scope=active`, `/jobs?status=FAILED&scope=active&job_id=123`, and no product dropdown; run focused tests and see active-scope failures.
- [ ] GREEN: apply active predicate after existing joins (for blocker query, require linked active installation only when `scope=active`); keep `page_size=50`, filters and count separate. Add small status GROUP BY and route params. Render one selected job detail from the returned page data or a single scoped `job_id` query; full error only there. Re-run focused tests.
### Task 4: Human-readable presentation and dashboard/filters markup
**Files:** `central/app/panel_presentation.py`, `central/templates/dashboard.html`, `central/templates/agents.html`, `central/templates/blockers.html`, `central/templates/jobs.html`, `central/tests/test_control_panel_pages.py`, `central/tests/test_operational_view_models.py`.
**Interfaces:** `job_status_label`, `job_kind_label`, `blocker_type_label`, `blocker_reason_label`, `priority_label`, `fiscal_status_label` map known enums, falling back to readable text; retain raw value on details. All dashboard health cards are `<a class="metric metric-link" href="/agents?scope=active&health=...">`; blocker field positive links preserve scope/status; zero stays text. Filter chips and clear links display active selections.
- [ ] RED: TestClient HTML assertions for card→health filter, breakdown NCM→field, failed jobs→FAILED, outdated→version, selected filters/clear actions, human PT-BR labels; confirm old HTML fails.
- [ ] GREEN: add presenter mappings to `panel_presentation.py` and include presentation fields in view-model dictionaries; render cards as links, attention groups with count/CTA, compact failed jobs/outdated; edit Agents table to combine identity columns, Blockers table product link + human labels, Jobs compact columns + detail/error. Keep historical routes intact, use `query_base`/`urlencode` to preserve filters and avoid XSS/unescaped params. Re-run HTML assertions.
- [ ] Retain the Nova instalação combobox unchanged: TestClient asserts `client_id` hidden, searchable active Gerenciador-only clients, keyboard/mouse JS and server-side validation still present.
### Task 5: Interaction/CSS and responsive presentation
**Files:** `central/static/control-panel.css`, `central/static/control-panel.js`, templates from Task 4.
- [ ] RED: browser preview at 1366/1920/375 exposes horizontal scroll on Agents today and non-clickable cards; capture snapshot/measure `scrollWidth`, focus order and toolbar visibility.
- [ ] GREEN: use only `var(--cor-*)` tokens for `.metric-link:hover/:focus-visible`, active-filter chips, grouped attention, semantic badges, compact tables and responsive 375px row layout. Set Agents table sizing to available content area and hide secondary fields below tablet; no 960px minimum for Agents on desktop. Keep Blockers/Jobs list bounded and filters stacked on mobile. Vanilla JS event delegation on `tr[data-href]` for mouse clicks outside existing links; keep primary `<a>` for Tab/Enter and avoid nested controls/intercepting modifier clicks.
- [ ] Browser check at 1366x768, 1920x1080, 375px for focus-visible, Tab/Enter (and Space on buttons), no desktop horizontal scroll in Agents, readable names, light/dark styles. Adjust CSS and repeat visual checks until criteria pass.
### Task 6: Full verification and official deploy
**Files:** `scripts/deploy_hub.ps1` (execute only; do not edit), test suites.
- [ ] Run `python -m pytest central/tests -q`, `python -m pytest tests -q`, and `python -m pytest agent/tests -q`; capture totals and failures. Verify no runtime/engine/SSO/auth files changed; `git diff --check` and inspect status/diff.
- [ ] Verify dashboard query count and, if SQL plan shows a missing index, first add a small Alembic migration plus migration test, re-run affected and full tests, and deploy with the same official script. No migrations solely for aesthetics.
- [ ] Use authenticated browser smoke on Resumo, Agents, Blockers, Jobs and Nova instalação; click Resumo→Online/Total/NCM/failed job/outdated Agent and navigate back. Recheck server-side selected filters and disabled/revoked exclusion at 1920x1080, 1366x768, 375px in light and dark. If production SSO login is unavailable, do equivalent smoke with TestClient/local preview and report production visual check as blocked.
- [ ] Run only `powershell -ExecutionPolicy Bypass -File .\scripts\deploy_hub.ps1` (or `pwsh -File .\scripts\deploy_hub.ps1` if script requires pwsh) as documented in `docs/DEPLOY_HUB.md`. Check `logs/last_deploy_result.json`, release health and public `https://gtifiscal.gtigrupo.com.br/health`. No ad-hoc SSH, no write to Firebird, no client action.
- [ ] Report the user-requested field list with observed evidence; `COMMITS=nenhum` if no explicit request to commit, and truthful blockers where browser authentication/production smoke cannot be completed.
@@ -0,0 +1,208 @@
# cClassTrib Applicability Implementation Plan
> **For agentic workers:** Execute this plan inline task-by-task with test checkpoints. Do not delegate or publish an Agent without following the release gate in Task 5.
**Goal:** Gate cClassTrib validation, blockers, and completeness by live company CRT and its official effective date, instead of treating global RTC enablement as universal requiredness.
**Architecture:** Keep `PARAMETROS.USAR_REFORMA_TRIBUTARIA` as the ERP-level RTC switch, then calculate a separate company-level cClassTrib applicability state from the already-read `C000004.CRT` and the current date. The shared pure evaluator owns the date/regime rule; the Agent carries the state into per-product findings and fingerprints; existing Hub presentation surfaces show a concise non-applicable label without adding another coverage panel.
**Tech Stack:** Python, pytest, SQLAlchemy-backed Hub presentation, Jinja templates, existing read-only Firebird Agent connection.
## Global Constraints
- Do not change architecture or add a new SGV flag, schema, API, or second coverage panel.
- Do not change NCM, PIS, COFINS, or other fiscal rules.
- Do not write to Firebird; reads use the existing `READ COMMITTED READ ONLY` Agent connection.
- Preserve Agent 2's current `rtc_write_mode=DRY_RUN`; `REAL_WRITES=0` and `CCLASSTRIB_WRITES=0`.
- `PARAMETROS.USAR_REFORMA_TRIBUTARIA` remains the RTC feature switch, not proof that every product requires cClassTrib.
- cClassTrib applicability uses live company CRT: CRT 3 from 2026-08-03; CRT 1/2/4 from 2027-01-01; unresolved/unsupported CRT is not applicable until resolved.
- Agent 2 baseline: total 521, complete 0, safe-blocked 521, failed 0, unclassified 0, coverage 0%; RTC 521 blockers; PIS 1, COFINS 1, NCM 26 distinct products/29 open rows.
- Current code uses Agent-side assessment; do not claim the live Agent 2 result changed until the updated Agent code is running and a new job completes.
---
### Task 1: Add failing pure applicability tests
**Files:**
- Modify: `central/tests/test_unified_fiscal_coverage.py`
- Test: `central/app/unified_fiscal_coverage.py`
**Interfaces:**
- Existing input is `evaluate_product(product, company, rtc, today)`; use `company["crt"]`, `rtc["enabled"]`, and the supplied date.
- Applicability states remain in `result["applicability"]["RTC"]`; basis is exposed in `result["basis"]["RTC"]`.
- [ ] **Step 1: Add the business-rule tests**
Add tests equivalent to:
```python
def test_cclasstrib_is_not_applicable_for_simples_before_2027_even_when_rtc_is_on():
result = evaluate_product({**EMPTY, "CCLASSTRIB": None}, SIMPLES, ON, date(2026, 9, 25))
assert result["applicability"]["RTC"] == "NOT_APPLICABLE"
assert "CCLASSTRIB" not in result["mandatory_fields"]
assert not [b for b in result["blockers"] if b["category"] == "RTC"]
def test_cclasstrib_effective_date_is_inclusive_by_crt():
for crt in ("1", "2", "4"):
before = evaluate_product({**EMPTY, "CCLASSTRIB": None}, {"crt": crt}, ON, date(2026, 12, 31))
on_date = evaluate_product({**EMPTY, "CCLASSTRIB": None}, {"crt": crt}, ON, date(2027, 1, 1))
assert before["applicability"]["RTC"] == "NOT_APPLICABLE"
assert "CCLASSTRIB_MISSING" in kinds(on_date)
before = evaluate_product({**EMPTY, "CCLASSTRIB": None}, NORMAL, ON, date(2026, 8, 2))
on_date = evaluate_product({**EMPTY, "CCLASSTRIB": None}, NORMAL, ON, date(2026, 8, 3))
assert before["applicability"]["RTC"] == "NOT_APPLICABLE"
assert "CCLASSTRIB_MISSING" in kinds(on_date)
def test_non_applicable_cclasstrib_does_not_mask_pis_cofins_blockers():
product = {**EMPTY, "CCLASSTRIB": None, "PISCOFINS": "S", "INCIDENCIA_PISCOFINS": "MONOFÁSICA"}
result = evaluate_product(product, SIMPLES, ON, date(2026, 9, 25))
assert kinds(result) == {"PIS_CST_MISSING", "COFINS_CST_MISSING"}
```
Also cover RTC off, missing/unsupported CRT, and applicable CRT 3 with empty cClassTrib.
- [ ] **Step 2: Run the tests and confirm the current code fails**
Run: `python -m pytest central/tests/test_unified_fiscal_coverage.py -q`
Expected: new applicability assertions fail because RTC enabled currently makes the empty field required for CRT 1/2/4 and before the effective dates.
---
### Task 2: Implement one shared applicability gate and carry it through Agent findings
**Files:**
- Modify: `central/app/unified_fiscal_coverage.py`
- Modify: `agent/src/gti_fiscal_agent/unified_fiscal.py`
- Test: `central/tests/test_unified_fiscal_coverage.py`
- Test: `agent/tests/test_unified_fiscal.py`
**Interfaces:**
- Add a pure helper with the stable signature `cclasstrib_applicability(crt, rtc_enabled, today) -> dict` returning `state`, `basis`, and `effective_from` as an ISO-8601 string or `None` so the summary is JSON serializable.
- Pass the result through the existing `rtc` context to `evaluate_rtc`; do not add a Firebird setting or database write.
- Add the applicability inputs to each product input fingerprint so changes in company CRT/effective state invalidate stale decisions.
- Include the company-level applicability result and source (`C000004.CRT` plus effective-date rule) in the Agent's `fiscal_context` summary.
- [ ] **Step 1: Implement the helper and make `evaluate_rtc` return no cClassTrib blocker when the gate is false**
Use these effective dates and exact CRT sets:
```python
CCLASSTRIB_EFFECTIVE_FROM = {
"3": date(2026, 8, 3),
"1": date(2027, 1, 1),
"2": date(2027, 1, 1),
"4": date(2027, 1, 1),
}
```
RTC disabled, CRT absent/unsupported, and dates before the CRT effective date must return `NOT_APPLICABLE`. For an applicable company, retain the existing catalog validation and missing-value blocker.
- [ ] **Step 2: Update the Agent evaluation context and input fingerprint**
Calculate the company gate once per assessment using the live `company["crt"]` and `today`; pass it to each `evaluate_product`. Extend fingerprint input with the resulting applicability state (not a cached Hub value). Continue to issue only SELECT statements to Firebird.
- [ ] **Step 3: Extend the Agent fake-cursor regression tests**
Cover RTC param on + CRT 1 before 2027 (no RTC finding), RTC param on + CRT 3 on/after 2026-08-03 (empty value blocked), separate PIS/COFINS findings preserved, the observed applicability source/state in `fiscal_context`, and a SELECT-only statement list.
- [ ] **Step 4: Run pure and Agent evaluator tests**
Run: `python -m pytest central/tests/test_unified_fiscal_coverage.py agent/tests/test_unified_fiscal.py -q`
Expected: PASS; all cursor statements in Agent discovery tests are SELECT-only.
---
### Task 3: Prove no candidates or writes while cClassTrib is not applicable
**Files:**
- Modify: `agent/tests/test_rtc_auto_apply.py` only if an existing case does not cover the no-finding path.
- Test: `agent/src/gti_fiscal_agent/rtc_auto_apply.py` and the assessment output from Task 2.
**Interfaces:**
- Candidate creation continues to consume the existing RTC findings; a non-applicable product must have no RTC missing finding and therefore no R3 candidate.
- Auto-apply authorization and allowed field remain unchanged.
- [ ] **Step 1: Add a focused candidate/write regression if needed**
Use RTC enabled, CRT 1, empty `CCLASSTRIB`, and the existing fake read-only Firebird connection. Assert zero `CCLASSTRIB_MISSING*` findings, zero preview candidates, zero write attempts, zero write successes, and no SQL other than SELECT on the assessment path.
- [ ] **Step 2: Run RTC auto-apply tests**
Run: `python -m pytest agent/tests/test_rtc_auto_apply.py agent/tests/test_unified_fiscal.py -q`
Expected: PASS; existing applicable CRT 3 safe-write unit tests remain valid and the non-applicable case has no candidate/write.
---
### Task 4: Present non-applicability on existing Hub views
**Files:**
- Modify: `central/app/panel_presentation.py`
- Modify: `central/templates/rtc_control.html`
- Modify: `central/templates/agent_detail.html`
- Test: `central/tests/test_control_panel_pages.py`
- Test: `central/tests/test_operational_view_models.py`
**Interfaces:**
- Preserve the existing `ops.rtc.enabled` value for the global RTC status.
- Add a presentation state for cClassTrib applicability; when false, do not render the current “cClassTrib pendente em N produtos” summary or RTC blocker empty-state warning for that field.
- Render one concise label in the existing RTC status areas: `cClassTrib: Não utilizado neste cliente`.
- [ ] **Step 1: Add a presentation test using Agent 2-like data**
Supply `rtc_enabled=true`, CRT 1, 521 resolution rows, 521 prior cClassTrib blocker products, and the derived non-applicable state. Assert the displayed label and assert that “cClassTrib pendente em 521 produtos” is absent on the Agent detail and RTC pages. Keep the single existing fiscal coverage metrics unchanged.
- [ ] **Step 2: Implement the smallest existing-view changes**
Have `rtc_status_view` use the Agent-reported applicability state. Show the concise label in the existing summary areas and suppress cClassTrib-pending metrics when not applicable. Do not add a new panel or alter the current RTC write mode.
- [ ] **Step 3: Run presentation tests**
Run: `python -m pytest central/tests/test_control_panel_pages.py central/tests/test_operational_view_models.py -q`
Expected: PASS; applicable CRT 3 views continue showing the existing candidate/coverage information.
---
### Task 5: Run the full suite, then gate Agent release and live reprocess
**Files:**
- Version/release files only after the user confirms the release action: `release/VERSION`, `central/app/version.py`, `release/RELEASE_MANIFEST.json`, and the installer metadata generated by `scripts/build_windows_installer.ps1`.
- No Firebird tables or records are modified.
**Interfaces:**
- The updated Agent must be running for Agent 2's live `REPROCESS_ALL` to exercise the new gate; the current production Agent is version 1.2.4.
- Official Hub deployment entry point is `scripts/deploy_hub.ps1`; a new Agent release uses `-PublishAgent` and the installer build script.
- [x] **Step 1: Run all offline tests before any version bump**
Run: `python -m pytest tests -q`
Also run the relevant Agent and Hub tests together: `python -m pytest central/tests/test_unified_fiscal_coverage.py central/tests/test_control_panel_pages.py central/tests/test_operational_view_models.py central/tests/test_zero_touch_hub.py agent/tests/test_unified_fiscal.py agent/tests/test_rtc_auto_apply.py agent/tests/test_zero_touch_agent.py -q`
Expected: PASS with no Firebird writes.
- [x] **Step 2: Pause at the Agent release gate**
Reported that the rule change is in Agent-executed logic, listed the Agent files/tests, and stated the current deployed version `1.2.4`. The operator explicitly authorized bumping to `1.2.5`, building/publishing, and upgrading Agent 2 in place.
- [ ] **Step 3: After release approval, build and publish through official scripts**
Run the repository's supported installer build script, then:
```powershell
powershell -ExecutionPolicy Bypass -File .\scripts\deploy_hub.ps1 -PublishAgent -Version 1.2.5
```
Do not use ad-hoc SSH/Docker deployment. Confirm the deployed Hub health/version and that Agent 2 is running the new Agent version before queuing its full reprocess.
- [ ] **Step 4: Reprocess Agent 2 and compare read-only ERP results**
Queue `REPROCESS_ALL` only after the updated Agent is confirmed running. Keep `rtc_write_mode=DRY_RUN`. Compare the new snapshot/job against the recorded baseline for total, complete, safe-blocked, failed, unclassified, coverage, cClassTrib blockers, and distinct NCM/PIS/COFINS blockers. Confirm `PARAMETROS.USAR_REFORMA_TRIBUTARIA` was observed live, `C000004.CRT=1` led to cClassTrib non-applicability as of 2026-09-25, and Firebird `WRITE_ATTEMPTS=0`, `WRITE_SUCCESSES=0`, `CCLASSTRIB_WRITES=0`.
Expected for this CRT 1 client before 2027-01-01: RTC cClassTrib blockers fall from 521 to 0; other fiscal field findings remain evaluated; RTC mode is unchanged; no candidates or writes are produced.
@@ -0,0 +1,57 @@
# Firebird Installation Inventory
## Scope and Safety
- Inventory performed read-only on 2026-09-16.
- `D:\PROJETOS\GTISISTEMAS\` outside `Projeto` was treated as a read-only external system.
- No source code, ERP files, services, processes, configuration, database objects, or permissions were modified.
- `ERP_FILES_MODIFIED = 0`
- No target database was accessed during inventory. `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` was not opened, connected to, queried, or inspected through a database tool.
- Another database file was visible under the external ERP tree, but it was also not opened.
- Credentials, passwords, connection values, and other secret values are omitted as `<redacted>`.
## Firebird Installation
- Installation root: `C:\Program Files\Firebird\Firebird_2_5\`.
- Registry instance `DefaultInstance` points to that installation root.
- Server family/version: Firebird 2.5, build `2.5.9.27139` (`WI-V2.5.9.27139`).
- Present server/client binaries include `fbserver.exe`, `fbguard.exe`, `fbclient.dll`, `isql.exe`, `gbak.exe`, `gstat.exe`, `gfix.exe`, `fbsvcmgr.exe`, `fbtracemgr.exe`, and `instsvc.exe`; these report build `2.5.9.27139` where version metadata is available.
- Firebird ICU DLLs report version `3.0.0.0`.
- `security2.fdb` exists in the installation root; it was not opened.
## Services and Process
- `FirebirdGuardianDefaultInstance`: running, automatic start, `LocalSystem`; executable is `fbguard.exe -s DefaultInstance`.
- `FirebirdServerDefaultInstance`: running, manual start, `LocalSystem`; executable is `fbserver.exe -s DefaultInstance`.
- `fbserver.exe` is running and responsive; process ID was observed but is not recorded as a stable inventory fact.
- Firebird server is listening on `0.0.0.0:3050`.
- No service was started, stopped, restarted, or reconfigured.
## Configuration
- `firebird.conf` exists and contains an active `RemoteAuxPort = 3051` setting.
- `aliases.conf` exists; no active alias facts were recorded.
- `databases.conf` was not present.
- Configuration values that could identify credentials, database locations, or endpoints are intentionally `<redacted>` or omitted.
## ERP Artifacts
- ERP executables observed include `GtiVendas_NFCe.exe` version `3.5.5.475`, `SGV_5.0.6.124.exe` version `5.0.6.124`, `configpdv.exe` version `1.00`, `Envio NF-e.exe` version `1.0.1.1`, and `GtiServer_v2.0.1.18.exe` version `2.0.1.18`.
- ERP Firebird client libraries observed include `fbclient.dll` version `2.5.9.27139` and `GDS32.DLL` version `2.5.8.27089`.
- Additional ERP DLLs and support libraries were inventoried; missing or blank embedded version metadata was recorded as unavailable rather than inferred.
- ERP configuration/document file types found include `.ini`, `.xml`, `.cfg`, `.conf`, and `.txt`, including `com.ini`, `Config_GtiServer.ini`, `ConfigBoleto.ini`, and `FTPConfig.ini`.
- Configuration and document contents were not reproduced in this report. Secret-bearing and database-related values are `<redacted>`.
## Verification and Git
- Exact command: `python -m pytest tests -q`
- Reported result: `58 passed in 0.15s`.
- Git identity check: `GIT_IDENTITY_NOT_CONFIGURED`.
- No commit was created and Git configuration was not changed.
## Unavailable Checks and Concerns
- The process API did not expose a usable executable path or command line for the running `fbserver.exe`; service configuration was used for the executable path.
- `isql-fb` was not available as a distinct command; the installed Firebird 2.5 `isql.exe` was present.
- No database metadata, schema, connection test, or database-level configuration was collected because database access was explicitly prohibited for this task.
- The running Firebird 2.5 service and the presence of `GDS32.DLL` version `2.5.8.27089` should be considered when planning later read-only discovery work.
@@ -0,0 +1,28 @@
# Task 3 Report
Status: COMPLETE
Runtime evidence/status:
- Pre-import dependency/client gate: `UNKNOWN` unless provenance, client
version, client architecture, and Python/client architecture match are
proven before importing `fdb`.
- Runtime compatibility fixture: `VALID` for `fdb 2.0.4`, client `2.5.9.27139`,
matching x64 architecture, and the exact read-only TPB.
- `validate_firebird_runtime(fdb_module=None)`: `UNKNOWN` with
`compatibility_unproven`; read-only API proof is mandatory.
- Success output exposes only compatibility categories/statuses and excludes
client paths, database identities, and internal provenance evidence.
- Blocked output exposes only status, blocker, and safety counters; provenance
path fields are not serialized.
Safety:
- No database, external ERP file, DLL, service, registry, or system
configuration was accessed or modified.
- `GIT_IDENTITY_NOT_CONFIGURED`.
Tests:
- `python -m pytest tests/test_dependency_compatibility.py -q`: 6 passed.
- `python -m pytest tests -q`: 255 passed, 1 skipped.
@@ -0,0 +1,85 @@
# Mission 00: Firebird Connectivity and Discovery
## Goal
Use the Firebird installation associated with the locally installed ERP as
read-only evidence to connect only to the authorized copy
`D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`, perform complete schema
discovery, and document confirmed findings without implementing fiscal writes,
image writes, services, or schedulers.
## Scope and boundaries
- Files outside `D:\PROJETOS\GTISISTEMAS\Projeto` are read-only.
- Only `TESTESOFICIAIS.FDB` may be opened or queried.
- Other database files may be listed but must not be connected to, copied, or
processed.
- No DML, DDL, procedure execution, generator changes, metadata changes,
permission changes, or service lifecycle changes are allowed.
- Credentials and secrets found in ERP configuration are evidence of location
only and must never enter source, reports, logs, or commits.
## Connection strategy
Use the installed Firebird server and its compatible Python driver/client as
the primary path. First inspect services, processes, executable and DLL
versions, tools, and ERP configuration. Select the client only after its
version and architecture are evidenced. Use an embedded/local client only as a
documented fallback if the server path is unavailable and compatibility is
proven. Do not install an arbitrary Firebird version.
The connection layer will enforce the absolute authorized database path and
require database-level or transaction/driver-level read-only enforcement. The
first database operation will be a minimal identity query. The sanitized
connection description will include driver, client, architecture, host/port
form, and target path, but no secrets.
## Discovery flow
1. Detect Firebird capabilities and execute only safe catalog SELECT queries.
2. Collect engine version, dialect, ODS, charset, page size, database identity,
relation counts, fields, keys, indexes, dependencies, generators, triggers,
procedures, parameters, and compatible monitoring metadata.
3. Identify product candidates using names, fields, keys, indexes, counts,
relationships, types, and minimal masked scalar samples.
4. Map product, stock, fiscal, company/branch, image/file, audit, and
incremental concepts. Mark every mapping `CONFIRMED`, `INFERRED`, or
`UNKNOWN` based on direct evidence.
5. Inspect trigger and routine source only as metadata text, without invoking
routines or collecting source/blob contents that are sensitive or
unnecessary.
6. Build fingerprint V1 from engine context, product structure, fiscal fields,
relationships, image structure, and critical triggers.
## Safety and sanitization
The existing read-only SQL boundary remains mandatory. Catalog capability
failures are recorded as unavailable or query errors rather than treated as
negative findings. Samples are small, scalar, masked, and purpose-limited.
Raw BLOBs, credentials, customer data, and unnecessary commercial data are
excluded. A path guard prevents accidental access to the ERP's original
database even if configuration reveals it.
## Project changes and verification
Changes are limited to the authorized project directory: path guard/driver
integration, discovery tests and fixtures, fingerprint support, and the three
current-state documents plus sanitized mission evidence. The `.FDB` remains
ignored by Git. Verification includes focused safety/parser tests, full
`python -m pytest tests -q`, `git diff --check`, sensitive-data review, and
Git status/identity inspection. A single mission commit is created only when
Git identity is already configured; otherwise report
`GIT_IDENTITY_NOT_CONFIGURED`.
## Outcomes
`MISSION_00_COMPLETE` requires real connection, confirmed engine details,
evidence-backed product and fiscal mapping, image/incremental/trigger review,
fingerprint V1, green tests, zero application writes, and no external file
modification. Any unresolved connectivity or compatibility blocker produces
`MISSION_00_BLOCKED` with the evidence and remaining gaps documented.
## Explicit non-goals
Do not implement fiscal rules, APIs, image storage, product UPDATEs, Windows
services, production schedulers, migrations, triggers, or database changes.
@@ -0,0 +1,95 @@
# Mission 00D: Structured Firebird Read-Only Boundary
## Goal
Remove arbitrary SQL from the public safety boundary, preserve legitimate
Firebird metadata expressions internally, then connect read-only only to
`D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` and complete evidence-backed
discovery when credentials and compatibility permit.
## Root cause
The previous `ReadOnlyConnection.execute(sql)` accepted consumer-provided SQL
and used a blacklist/allowlist parser as its execution authority. Quoted
identifiers could hide callable names from the textual scan, allowing potential
procedure/UDF invocation. The same incomplete callable allowlist rejected
legitimate Firebird expressions such as `CAST` and aggregate functions. This
is a boundary-design failure, not a missing regex case.
## Architecture
- Remove `execute(sql)` and equivalent raw execution methods from the public
adapter consumed by collectors and mapping code.
- Keep driver execution in a private implementation that accepts only internal
query identifiers and internally owned static SQL templates.
- Expose structured catalog operations for database info, relations, columns,
primary/foreign keys, indexes, triggers, procedures, generators,
dependencies, and monitoring metadata.
- Expose relation sampling only after relation and column identities have been
returned by the catalog. Validate those identities structurally, quote them
as Firebird identifiers, and cap the integer row limit.
- Keep transaction-level Firebird read-only enforcement and fail closed if it
cannot be proven. Do not expose commit, rollback, DDL, DML, procedure
execution, or generator modification.
- `collect_metadata` must require the approved structured reader at runtime,
not merely by type annotation.
## Query ownership
All metadata SQL belongs to the implementation and is selected by fixed query
keys. Internal queries may use `CAST`, `COUNT`, `SUM`, `MIN`, `MAX`,
`COALESCE`, joins, grouping, and other required Firebird syntax without going
through the old arbitrary-SQL parser. Procedures and UDFs are queried only as
metadata and never invoked.
The old parser may remain as an offline diagnostic/test utility, but it has no
authority to decide whether runtime SQL is executable. If retained, its
documentation must clearly state that quoted identifiers and Firebird grammar
are not security guarantees.
## Sampling
Sampling accepts only a relation identity previously returned by catalog
metadata, a column set previously returned for that relation, and a validated
small integer limit. It rejects expressions, clauses, procedures, UDFs,
fragments, arbitrary `WHERE`/`ORDER BY`, unknown relations/columns, malformed
quoted identifiers, and alternate database paths. It returns sanitized scalar
evidence only; BLOB content, credentials, customer data, and unnecessary
commercial values are excluded.
## Verification gates
Before any database access, tests must prove that arbitrary SQL cannot reach
the driver, quoted identifiers cannot reintroduce callable execution,
procedures/UDFs cannot be invoked, legitimate internal Firebird expressions are
supported, and arbitrary adapters are rejected. Only after all focused tests
and the full suite pass may the installed Firebird 2.5.9 client be used.
The connection then targets only the authorized `.FDB`, performs a minimal
identity query first, and uses a read-only transaction. If credentials,
compatibility, or identity verification fail, the result remains
`MISSION_00_BLOCKED` and no discovery query continues.
## Discovery and documentation
After a successful connection, collect capability-aware catalog metadata,
identify the product relation using structural and masked sample evidence,
correlate the ERP UI fields to database fields with `CONFIRMED`, `INFERRED`,
or `UNKNOWN` classifications, inspect fiscal/image/incremental structures,
and analyze trigger/procedure metadata without execution. Generate a real
schema fingerprint only from sanitized evidence. Update the current-state
documents and final report to describe the actual adapter, parser role,
runtime protections, transaction behavior, connection facts, and remaining
gaps.
## Safety invariants
- `AUTHORIZED_DATABASE` is the sole file opened or queried.
- `OTHER_DATABASES_ACCESSED = 0`.
- `APPLICATION_DDL_DML_WRITES = 0`.
- `ERP_FILES_MODIFIED = 0`.
- Files outside `Projeto` are read-only.
- No credentials, raw BLOBs, full rows, or original ERP database paths enter
project documentation.
- No API fiscal, image API, service, scheduler, product UPDATE, or migration
is implemented.
@@ -0,0 +1,88 @@
# Mission 00E: Threat Model, FDB Provenance, and Real Discovery
## Goal
Formalize the trusted-process threat model, validate the actual `fdb` module
provenance and Firebird client compatibility, then run read-only discovery only
against `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`.
## Threat model
`TRUSTED_PROCESS = TRUE`: only versioned application code runs in the process;
there are no untrusted plugins, user-provided scripts, external `eval`/`exec`,
or externally selected dynamic imports. Deliberate introspection or monkey
patching by code that already has arbitrary Python execution is equivalent to
application compromise and is outside this component's boundary.
The boundary protects against implementation mistakes, accidental unauthorized
operations, arbitrary SQL exposed to normal callers, parameter-to-SQL injection,
routine execution, wrong database selection, unintended writes, and dependency
shadowing/import mistakes.
## API boundary
The public API exposes only structured catalog operations and catalog-issued
bounded sampling. It exposes no `execute(sql)`, `query(sql)`, `raw_execute`,
operation-registration mutator, procedure/UDF invocation, arbitrary clauses, or
alternate database path. The registry remains module-private, immutable, and
centrally constructed. Python internals are not treated as a cryptographic
sandbox.
The old SQL parser is offline-only diagnostic/test code. It has no authority to
approve runtime SQL. Internal Firebird query templates may use legitimate
`CAST`, aggregates, joins, and grouping because they are selected by fixed
operation keys owned by the project.
## Provenance gate
Before any database connection, inspect the actual `import fdb` module:
- `fdb.__file__` and `fdb.__package__`;
- module version and `importlib.metadata` distribution/version/location;
- interpreter, working directory, `sys.path`, and `PYTHONPATH`;
- local `fdb.py`, `fdb` directory, and unexpected `.pth` shadowing candidates.
The imported module must correspond to the expected installed distribution and
version, without ambiguous duplicate installations. The installed Firebird
client/server evidence must match the selected driver compatibility. Any
ambiguity blocks before import/use of the driver for database access.
## Connection and discovery
After the provenance and safety gates pass, connect only to the authorized FDB,
using the installed compatible client and a Firebird read-only transaction. Run
the minimal identity operation first and compare the engine-reported identity
to the authorized target. Then collect catalog metadata through structured
operations, use only catalog-issued sample identities, and inspect product,
stock, fiscal, image, incremental, trigger, and procedure metadata without
executing routines or reading BLOB contents.
Correlate the ERP UI fields `NCM`, `CFOP`, `CST ICMS`, ICMS rate, `CEST`,
`CSOSN`, PIS/COFINS, IPI, origin, IAT, incidence, and IPPT to table, column,
type, domain/FK, dependencies, triggers, evidence, and confidence. Classify
each as `CONFIRMED`, `INFERRED`, or `UNKNOWN`. Generate a real fingerprint only
from sanitized complete evidence; otherwise use `SCHEMA_UNKNOWN`.
## Evidence and safety
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened.
- External ERP files, services, registry, DLLs, and configuration are read-only.
- `APPLICATION_DDL_DML_WRITES = 0`.
- `OTHER_DATABASES_ACCESSED = 0`.
- `ERP_FILES_MODIFIED = 0`.
- No credentials, original ERP database path, raw BLOB/image, full rows, or
unnecessary customer/commercial data enters source or reports.
- No fiscal API, image API, Windows service, scheduler, product UPDATE, or
migration is implemented.
## Verification and outcome
Tests cover the public API, registry immutability, SQL non-authority,
quoted-identifier and routine cases, legitimate internal expressions, path
allowlisting, provenance/shadowing, client compatibility, structured reader,
sampling, fingerprint, and discovery. The final report includes provenance,
client/engine facts, connection result, schema mappings, image/incremental/
trigger evidence, tests, Git state, and real gaps. Missing credentials,
ambiguous provenance, compatibility failure, identity mismatch, or incomplete
discovery results in `MISSION_00_BLOCKED` without a database connection or
fallback to another database.
@@ -0,0 +1,19 @@
# Fiscal Data Foundation V1
## Goal
Provide independently versioned, official fiscal datasets for the Hub without determining taxes for products.
## Dataset lifecycle
Each domain runs independently: official download, raw SHA-256, versioned parser, structural validation, `PENDING` import, semantic/volume validation, then `ACTIVE` promotion. A parser failure records `FAILED` and `failure_reason`; the previous `ACTIVE` version remains queryable. No secondary source can promote a canonical dataset.
Every version records `domain`, `source_url`, `downloaded_at`, `raw_sha256`, `parser_version`, `effective_from`, `effective_to`, `record_count`, `status`, `failure_reason`, and `raw_payload_reference` when retention is allowed.
## Domains
Existing NCM/Siscomex and CEST/CONFAZ remain intact. New isolated importers cover BA ST/MVA, CFOP, ICMS CST/CSOSN/origin, PIS/COFINS, TIPI/IPI, RTC, FCP, and monophase ICMS/ANP. Each has a normalized operational table only when queried by Hub consumers.
## Verification
Parser tests prove fail-closed behavior and independent lifecycle behavior. The final smoke reads only `ACTIVE` versions and produces the dataset matrix. This mission does not classify products or propose tax fields.
@@ -0,0 +1,151 @@
# GTI Fiscal Hub V1 - Mission 02 Design
## Scope
Deliver one end-to-end vertical slice: a FastAPI fiscal hub backed by
PostgreSQL, a versioned official NCM dataset, and a Windows-ready Python agent
that reads the authorized Firebird database and submits 20 real products to the
hub. The agent never writes to Firebird in this mission.
The Hub and agent are separate applications. The Hub owns fiscal knowledge;
the agent owns ERP communication. The Hub never accesses a customer Firebird
database and the agent never accesses central PostgreSQL.
## Components
`central/app` is a modular FastAPI monolith with these conceptual modules:
- API and bearer-token authentication.
- Dataset registry and NCM importer.
- Product classification: normalized description, GTIN, NCM, and CEST.
- Fiscal determination: operation and company context with NCM/CEST, CFOP,
CST/CSOSN, PIS/COFINS, IPI, and future IBS/CBS/cClassTrib inputs.
- Deterministic classification cache.
- Audit and provenance.
- Provider interfaces, including `RtcCalculatorProvider`.
Product classification and fiscal determination remain separate modules inside
the same process. This makes an existing valid NCM distinct from a fully proven
tax treatment.
`agent/src` contains a Firebird reader, schema fingerprint gate, product
fingerprinting, SQLite state/outbox, HTTP client, retry/backoff handling, and
comparison/report generation.
`docker-compose.yml` starts `postgres` and `fiscal-hub` only. The Firebird
database remains installed locally on Windows and outside Docker.
## Hub Persistence
PostgreSQL stores `datasets`, `ncm_entries`, `tenants`, `agents`,
`classification_requests`, `classification_results`, and
`classification_cache`. Dataset metadata includes source URL, provider,
download time, SHA-256, source validity/version data when present, record
count, lifecycle status, and import status.
Agent tokens are supplied by environment. PostgreSQL stores only token hashes,
each associated with one tenant and agent. Tenant identity is resolved from the
token and must agree with request metadata.
The agent uses a separate SQLite database. It stores source and result
fingerprints, fiscal dataset versions, timestamps, reason codes, retry data,
and an outbox with this lifecycle:
`DISCOVERED -> PENDING_API -> SENT -> RESULT_RECEIVED -> APPLIED`
`APPLIED` is reserved for a future write-capable mission and is not reached in
Mission 02.
## NCM Dataset Lifecycle
The only canonical source in this mission is the Receita Federal / Portal
Unico Siscomex JSON endpoint:
`https://portalunico.siscomex.gov.br/classif/api/publico/nomenclatura/download/json`
The importer downloads raw content, calculates SHA-256, parses JSON, and
inspects its real schema before mapping fields. It must not assume field names
before this inspection.
The lifecycle is:
`DOWNLOAD -> HASH -> PENDING -> IMPORT -> VALIDATION -> READY -> ATOMIC PROMOTION -> ACTIVE`
The previous active version becomes `SUPERSEDED` only inside the same short
promotion transaction that activates the ready version. NCM entries are scoped
to their dataset version, so clients observe either the complete prior dataset
or the complete new dataset, never a mixture. Failed imports remain
non-active, and prior versions remain queryable for rollback and audit.
Validation requires a successful HTTP response, parseable JSON, recognized
schema, plausible nonzero record volume, persisted hash and provenance, and a
fully successful import. A scheduled updater runs this same flow and promotes
only a validated version.
## Classification and Decisions
`POST /v1/validate/ncm` normalizes a NCM code and validates it against the
active official dataset. `/v1/classify` and `/v1/classify/batch` accept a
minimal product plus extensible company and operation context.
If an existing NCM is valid, the classification result is `NCM_VALIDATED`.
This is not permission to populate or change CST, CFOP, CSOSN, rates, CEST, or
any other fiscal field. Fields without official deterministic support return
`null` with explicit reason codes.
Products without a valid NCM use deterministic normalized-text candidate
ranking. Outcomes are `HIGH_CONFIDENCE_CANDIDATE`, `AMBIGUOUS`, or `NOT_FOUND`;
description similarity never creates a definitive NCM classification.
Each response carries `engine_version`, `dataset_versions` (including `ncm`),
and `decision_version`. These versions participate in central cache keys and
agent incremental decisions. A product with unchanged source fields is
reprocessed when relevant dataset or engine versions change.
`can_auto_apply` is always false in Mission 02 because no Firebird writes are
implemented. `NCM_VALIDATED` is a classification status only.
## Cache, Idempotency, and Audit
The Hub derives a request fingerprint from tenant, normalized product identity,
relevant context, engine version, and active dataset versions. A matching cache
entry returns an equivalent decision and does not create uncontrolled duplicate
audit records. A changed tenant, context, engine, or dataset version cannot
reuse a result.
The agent derives its product fingerprint from description, GTIN, NCM and
relevant current fiscal fields. It avoids an API call only if this fingerprint,
the prior decision versions, and result state all remain current. Interrupted
or failed sends stay in the outbox for retry. If the API is offline, the agent
does not affect the ERP and continues processing other queued items.
## Provider Status
NCM is implemented. CEST, CFOP, CST/CSOSN, cClassTrib, IBS/CBS, and the RTC
calculator are investigated and recorded with one of `IMPLEMENTED`, `PARTIAL`,
`DISCOVERED`, or `NOT_AVAILABLE`. The RTC adapter implements the provider
interface but returns `NOT_CONFIGURED` unless a concrete official integration
is verified to be simple and safe within this scope. It never fabricates tax
results.
## End-to-End Proof
The agent reads the 20 sample products from `C000025` using only SELECTs after
its schema fingerprint equals the recorded `REFERENCE_SCHEMA_V1` fingerprint.
It sends them to the authenticated batch endpoint, persists results locally,
and writes `docs/MISSION_02_PRODUCT_RESULTS.json` plus
`docs/MISSION_02_REPORT.md`.
The report includes coverage and timing metrics, API request and cache counts,
paid API counters fixed at zero, Firebird safety counters, dataset provenance,
integration statuses, and real gaps. No BLOB, image, Firebird schema change,
or Firebird DML/DDL is allowed.
## Verification
Focused tests cover NCM import and lifecycle, normalization/ranking,
authentication and tenant isolation, endpoints, cache/idempotency, agent
payloads, read-only Firebird access, fingerprint gating, incremental outbox,
and offline API behavior. The complete project test suite runs afterward.
Docker health checks cover PostgreSQL and the Hub; Hub health reports failed
when PostgreSQL is unavailable.
@@ -0,0 +1,74 @@
# PRE-AI Local Completion V3 Design
## Goal
Finish and verify every local, deterministic part of the semantic completion
product without external LLM calls, Firebird semantic writes, deployment, or
new fiscal rules. When credentials become available, only the real benchmark
execution should remain.
## Scope and Order
1. Offline AI plumbing: deterministic fake provider, batch correlation,
retries, budget enforcement, cache invalidation, and end-to-end dry-run.
2. Knowledge base: canonical identity and conflict audit, conservative
deduplication, retrieval V3 evaluation, final candidate preparation, and
payload compaction.
3. Operations: guided company/group questionnaires, rollback preview/guard,
idempotent local validation, and a concise health artifact.
The phases are sequential. Phase 1 is rerun after Phase 2 with the final work
items and candidate sets. No fake answer is persisted as semantic truth.
## Safety Invariants
- `DETERMINISTIC_GATE_CLOSED=true` remains unchanged.
- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` are asserted by tests and
reports.
- External HTTP/API calls are forbidden in offline commands and tests.
- Firebird access remains read-only for discovery; no uncertain semantic value
is written to Firebird.
- Tenant fiscal context is never promoted into global product identity.
- Conflicts are retained and classified; automatic consolidation requires
exact structural evidence and equivalent normalized descriptions.
- `MIXED_OR_DYNAMIC` never produces a single invented `CFOP_ENTRADA`.
- Rollback changes only rows whose current value equals the recorded expected
post-write value.
## Architecture
The fake provider implements the existing semantic selection contract and is
driven by named scenarios. A batch harness maps responses by `work_item_id`,
validates candidate confinement, retries only transport failures, and records
provider calls, token usage, and budget decisions. Cache keys include input,
candidate set, model, prompt version, and all relevant dataset versions.
KB consolidation remains additive: `canonical_product_id` is a stable global
identity for equivalent observations, while tenant-specific values remain in
tenant context. Conflict audit and candidate preparation are persisted as
versioned JSON artifacts or Hub records; no fiscal decision is inferred.
Operational tools are CLI-first and preview-first. They reuse the existing
question and context persistence mechanisms, produce reviewable JSON/CSV, and
are idempotent. The full validation command composes existing read-only
scripts plus the new preparation/report steps.
## Acceptance Criteria
- Fake scenarios cover correct, wrong, abstain, invalid JSON, missing,
duplicate, out-of-candidate, timeout, 429, and 500 responses.
- Batch sizes 5, 10, and 20 are tested with out-of-order responses and
partial/retry failures, always correlated by ID.
- Budget overrun blocks before provider invocation; unknown prices remain
explicitly unknown.
- Cache hit performs zero provider calls; each key component independently
causes a miss; dataset drift yields `NEEDS_REVALIDATION` or `CONFLICT`.
- KB conflict audit contains all 163 baseline conflicts or an explicit
measured current count, with no silent merge.
- Retrieval V3 never lowers GOLD recall@20 below 0.954 on the holdout and only
becomes active when the comparison artifact proves improvement or equal
recall with better recall@1/MRR.
- Final E2E simulation completes GOLD -> gate -> SILVER/NOISY -> residual plan
with zero Firebird writes.
- Full local validation is idempotent and emits `docs/LOCAL_PRODUCT_HEALTH_V1.json`.
- Full test suite passes.
@@ -0,0 +1,37 @@
# PRE-ZAI Blocker Closure Design
## Goal
Close every locally verifiable blocker before the real Z.ai GOLD benchmark,
without changing the 90 valid NCM conflicts, inferring human facts, calling
external AI, deploying, or enabling Firebird AI writes.
## Design
The historical conflict audit joins every available seed-payload observation by
GTIN and fingerprint, preserves descriptions/tenant/product/NCM evidence, and
classifies each historical row as `RESOLVED_EQUIVALENT_PRODUCT`, `FALSE_MATCH`,
`VALID_NCM_CONFLICT`, `STALE_OBSERVATION`, or `INSUFFICIENT_EVIDENCE`. Missing
evidence remains unresolved and never selects an NCM by majority.
The operator session exposes only COMPANY questions first. It is guided,
cancelable, review-first, and produces a context import artifact only after
explicit confirmation. IPI options include real local impact counts where the
source artifacts expose them; rollback remains a plan until the operator
chooses it. `MIXED_OR_DYNAMIC` produces no universal CFOP.
Final health derives from existing artifacts and the regenerated residual plan.
The report uses `COMPANY_CONTEXT_COMPLETE=false` until answers are actually
persisted, and labels GROUP/PRODUCT decisions as human-pending rather than
pipeline failures. Retrieval V2 remains active unless a complete holdout proves
V3 improvement.
## Safety and Acceptance
- Current 90 conflicts remain `NCM_CONFLICT` unless objective evidence changes a
specific row.
- Historical rows without evidence are `INSUFFICIENT_EVIDENCE`.
- All reports state `FIREBIRD_AI_WRITES=0` and `AI_WRITE_ALLOWED=false`.
- No answer is auto-persisted by the guided session.
- The final artifact contains no placeholder TODOs and lists only external AI
quota/credential plus deliberate human decisions as remaining blockers.
@@ -0,0 +1,71 @@
# Autonomous Fiscal Completion V1 Design
## Goal
Reduce deterministic fiscal gaps across the five explicitly allowlisted local
Firebird databases without calling an AI provider, answering human decisions,
or treating a populated ERP field as mandatory by default.
## Safety Boundary
All discovery and evidence extraction is read-only. Deterministic writes are
permitted only through the existing guarded executor, only for exact allowlisted
paths and proven write targets. A write requires provenance, `CHANGE_REQUIRED`,
`write_eligible=true`, path/company/schema/type guards, compare-before-write,
exact rowcount, post-write equality, and a no-unplanned-columns audit. If any
guard fails, the result remains pending. AI flags remain
`AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0`.
## Workstreams
### 1. Requiredness and completeness
Create a versioned requiredness matrix for the 23 existing fiscal contracts.
Rules are keyed by company regime/UF and product/operation facts, and produce
one of `REQUIRED`, `CONDITIONAL`, `OPTIONAL_REFERENCE`, `NOT_APPLICABLE`, or
`ERP_NOT_USED`. Implement `FiscalCompletenessEvaluator` to return a status per
field and aggregate product metrics without requiring empty `NOT_APPLICABLE`
fields to be filled.
### 2. Inbound evidence
Perform schema discovery across all five databases without assuming table
names. Identify headers, items, suppliers, keys, XML/BLOB/text evidence, and
relationships. Produce a sanitized evidence map and read-only analysis counts.
Supplier observations remain observational until conservative deterministic
rules prove they are reusable.
### 3. Deterministic resolution
Build provenance-aware evidence aggregation for origin, product role, ST/CEST,
and dynamic CFOP entry scope. Use supplier/document agreement metrics rather
than majority alone. Keep conflicts, missing evidence, company/group/product
questions, and `ST_WITHHELD` uncertainty pending. Reuse only compatible global
KB facts; tenant-specific policy fields are never copied cross-tenant.
Run dry-run and completeness metrics for all five databases. Execute only
eligible deterministic writes, then reprocess and require zero second-pass
writes. Preserve all current NCM conflicts.
### 4. Providers and AI controls
Document external product-data provider candidates without purchasing or
querying them. Add an abstract provider contract and permanent-cache schema for
GTIN/provider/version/hash/normalized data. Add persistent central AI usage
ledger and atomic pre-call limits for run, daily, monthly, requests, and new
products. Prove idempotent cache behavior with offline tests; no provider or AI
network call is made.
## Artifacts
The implementation must produce the requiredness matrix, inbound evidence map,
provider evaluation, consolidated JSON/Markdown autonomy reports, and updated
`AI_PROJECT_STATE.md`. Artifacts must report measured values and explicit
unknowns rather than inferred fiscal answers.
## Verification
Each workstream adds focused tests. The final gate runs the complete `tests` and
`central/tests` suites, `git diff --check`, read-only invariant checks, dry-run
and second-pass metrics, and confirms no external AI calls, no non-allowlisted
database access, and no semantic Firebird writes.
@@ -0,0 +1,111 @@
# Control Panel Operational UX Refinement V1
## Scope
Refine the existing FastAPI/Jinja2 control panel for daily GTI operations. The
panel must answer which client/server needs attention, why, and which safe
operator action is available. This work does not change fiscal rules, the
Hub/Agent protocol, Firebird access, AI write controls, or deployment.
## Design Choices
The implementation keeps the existing server-rendered architecture:
- `control_plane.py` remains the bounded query/view-model layer.
- A small presentation helper formats UTC timestamps using a configurable UI
timezone, defaulting to `America/Bahia`.
- Existing models are reused. `FiscalContextEntry` stores validated operator
context, `Job` schedules reprocessing, and `AuditEvent` records the complete
action context. No new write capability is introduced.
- Jinja2 templates receive operational view models, not raw ORM rows. Technical
IDs remain secondary detail information.
- CSS remains project-local and JavaScript is limited to progressive enhancement;
blocker resolution is a server-rendered form, not a `window.confirm` action.
## Dashboard Semantics
Health cards include `ONLINE`, `DEGRADED`, `OFFLINE`, `FAILED`, and `UNKNOWN`.
The query layer exposes a reconciled total and tests enforce:
`agents_total == online + degraded + offline + failed + unknown`.
Fiscal metrics are separate:
- `safe_blocked_products`: distinct active products represented by open blockers.
- `open_blockers`: active blocker rows.
The dashboard labels these independently and never presents one as the other.
Attention items are a deterministic maximum of ten, ordered by `FAILED`,
`DEGRADED`, `OFFLINE`, `UNKNOWN`, required blockers, and important version
drift. Each item contains client, display name, hostname, main problem, age,
and a safe detail/filter link. Failed jobs and outdated Agents are compact
sections with empty states.
## Presentation
UTC remains the storage/API representation. The UI renders both a human short
form (`22/09/2026 16:11`) and relative age (`há 8 min`); the full technical
timestamp is available as a `title` attribute. The timezone comes from
`UI_TIMEZONE`, with `America/Bahia` as the development default.
Agents lead with client, display name, and hostname. Technical health and fiscal
status are separate badges. Fiscal coverage includes total, complete, safe
blocked, failed, and percentage with a worst-coverage sort option.
Jobs show client, installation/display name, hostname, job ID, kind, status,
human timestamps, duration, and error. Blockers show client/installation,
product, field, type, deterministic operational priority, age, and links to a
detail page.
## Blocker Operations
The blockers page has visible server-side filters for client, installation,
field, blocker type, status, age, and product/search. Its summary reports open
blockers, distinct affected products, distinct affected clients, and field
breakdown (`CFOP`, `ST`, `ICMS`, `CSOSN`, `NCM`, `OTHER`).
Priority is operational only and deterministic:
- `CRITICAL`: technical/unsafe condition that prevents processing.
- `HIGH`: required field missing for an active product or a relevant conflict.
- `NORMAL`: safe blocker without immediate operational urgency.
- `LOW`: optional/reference enrichment.
Clicking a blocker opens a detail page with evidence, first/last seen, retry
count, related events/jobs, and resolution history. The resolution form shows
the client, server, product, description, field, type, reason, evidence, and
current value. It accepts a validated context key, value, scope (`PRODUCT`,
`GROUP`, or `COMPANY`), and reason/note. Known enum keys are validated by the
existing `context_keys` rules; arbitrary context values cannot bypass backend
validation.
On save, the transaction creates/updates the validated `FiscalContextEntry`,
creates a bounded reprocess job for the product, and records an audit event
containing operator, timestamp, client, installation, product, blocker,
context key/scope/value, reason, before/after values, and reprocess job ID.
The operation remains protected by human auth, CSRF, and authorization. It
cannot enable AI writes, Firebird writes, SQL, or shell execution.
## Detail Pages and Events
Agent detail shows client/display name/hostname first, technical/fiscal/version
badges, human last-seen, version, company, UF, CRT, and secondary technical IDs.
Fiscal summary is bounded and links to filtered blockers. Event rows have human
timestamps, severity, event type, message, and related job/product links, with
server-side severity/type/date/job/product filters and pagination.
## Password Policy
Weak local development passwords are allowed only when explicitly enabled by
`ALLOW_WEAK_DEV_PASSWORD=1` and only when `APP_ENV` is not `production`.
Production always requires the strong password policy. The existing local admin
credential is not changed.
## Verification
Tests cover health reconciliation, metrics distinction, timestamps, attention
ordering/limit, fiscal percentage, human job identity/duration, blocker filters
and detail, resolution validation/audit/CSRF/authorization, events, and
environment-aware password policy. Full `tests`, `central/tests`, and
`agent/tests` regressions plus local route/runtime and bounded performance
smokes are required before completion.
@@ -0,0 +1,108 @@
# GTI Fiscal Control Panel V1 Design
## Scope
Build the operational control plane and web panel in the existing FastAPI Hub.
The implementation has two ordered blocks: a backend foundation and a
server-rendered interface. Fiscal decision logic remains unchanged, AI writes
remain disabled, and no remote Firebird write control is exposed.
## Architecture
The Hub remains the single application boundary. SQLAlchemy models and Alembic
migrations store installation identity, operational snapshots, structured
events, human administrators, sessions, and audit entries. FastAPI exposes two
strictly separate authentication paths:
- Agent routes continue using the existing Bearer token principal.
- Human routes use an opaque, expiring HttpOnly cookie backed by a server-side
session record and a CSRF token for mutations.
The interface uses Jinja2 templates, project-owned CSS, and small progressive
enhancement JavaScript. No SPA, React, Vue, Node build chain, or duplicated
frontend API is introduced.
## Identity And Registration
`installation_id` is a UUID generated once by the Agent install/configuration
flow and persisted in local Agent state. Registration sends it explicitly and
the Hub treats it as the stable installation key. `agent_id` remains the Hub
Agent record identity. Hostname, IP, display name, customer code, and company
context are metadata and never replace the installation identity.
Registration and heartbeat accept only allowlisted operational metadata. CNPJ
is masked before storage/display, database path is sanitized, and credentials,
tokens, passwords, and keys are rejected or discarded.
## Operational Model
The Hub stores a current per-installation snapshot rather than recalculating
product or event tables on every page request. The snapshot includes product
counts, coverage, blocker aggregates, latest job outcomes, heartbeat, health,
version drift, company context, Firebird metadata, capabilities, and dataset
versions.
Health is deterministic:
- `UNKNOWN`: registered but no usable heartbeat/snapshot yet.
- `OFFLINE`: last heartbeat is older than the configured threshold.
- `FAILED`: persistent technical failure prevents processing.
- `DEGRADED`: technical/job failures are relevant but processing remains
possible.
- `ONLINE`: recent heartbeat with no critical technical failure.
Fiscal `SAFE_BLOCKED` counts never independently cause `DEGRADED` or `FAILED`.
Offline threshold is configurable and defaults to three heartbeat intervals when
the configured interval is available.
## Events, Redaction, And Retention
Agents submit structured operational events, not raw log files. The Hub
redacts case-insensitive secret keys and credentials embedded in strings,
limits event message/context size, and persists severity, type, timestamps,
installation, optional job/product/blocker references, and structured context.
Default retention is 90 days via `AGENT_EVENT_RETENTION_DAYS`. A maintenance
operation removes only non-audit events older than the retention period. Write
audit, blocker history, and critical audit records are preserved.
## Human Actions
Human mutations require an authenticated session, valid CSRF token, explicit
confirmation in the UI, and an audit event. Allowed operations are reprocess
all, reprocess selected products/blockers, resolve a blocker with explicit
context, display-name/customer/notes edits, and CSV exports. The panel cannot
change Firebird write flags, AI flags, credentials, Firebird paths or CRT,
execute SQL, run a shell, or restart a Windows service.
## APIs And UI
The panel APIs provide summary, installation list/detail/health/coverage,
blockers, jobs, and events with real pagination and filters. Global blockers and
jobs reuse the same query services. The UI includes dashboard, agent list,
installation detail tabs, global blockers, global jobs, and event timelines.
Every primary row uses client, display name, and hostname; UUIDs remain in
technical details.
## Performance
Indexes cover tenant, installation, health, heartbeat, version, event time and
severity/type, job agent/status/time, and blocker agent/status/type/time.
Dashboard cards use snapshot aggregates and bounded grouped queries. Detail and
global lists use limit/offset pagination for V1, with bounded page sizes and no
N+1 relationship loading.
## Verification
Tests cover persistent installation identity, sanitized registration and
heartbeat metadata, health/offline transitions, snapshot aggregation, event
redaction and size limits, retention, admin hashing/login/session/CSRF, API
filters and pagination, safe action audit trails, templates, and the existing
main/central/agent suites. A non-production fixture generator creates 100
varied installations without auto-seeding production.
## Explicit Non-Goals
No fiscal rule changes, AI invocation, AI writes, Firebird writes, production
deployment, real-client installation, remote Agent update, raw log upload,
shell execution, or SPA frontend.
@@ -0,0 +1,107 @@
# Agent Installation Identity Recovery and First Heartbeat
## Goal and scope
Activate the already-enrolled Agent 2 on this Windows host without creating a
new Agent, enrolling again, or registering a second installation. Recover and
persist its existing Hub `installation_id`, send a first real heartbeat, and
stop before any job is claimed. Firebird stays strictly read-only and the
installation remains `FIREBIRD_WRITES=0` / `FISCAL_MODE=DRY_RUN`.
## Current root cause
`POST /v1/agents/enroll` creates the Hub `Installation` with
`installation_id=enrollment_uuid`, but its response omits that identity.
`installer/setup_main.py` therefore does not persist it. On a host with no
`agent_state.sqlite`, `AgentState.installation_id()` invents a UUID. The Hub
heartbeat route rejects a different UUID for an Agent that already has an
Installation, preventing the first heartbeat.
The service loop also calls `GET /v1/jobs/next` immediately after a successful
heartbeat, so an explicit runtime gate is required for this activation.
## Design
### Identity delivery and recovery
- Return `installation_id` in the successful enrollment response. The value is
the existing `enrollment_uuid` used to create the Hub Installation.
- Persist this non-secret value in `agent.env` for new installs.
- Add authenticated `GET /v1/agents/me/identity`, returning only
`agent_id`, `installation_id`, `hostname`, `version`, and `status`. Keep the
existing `/v1/agents/me` response unchanged for compatibility.
- At runtime, identity precedence is: existing SQLite `installation_identity`,
then configured `INSTALLATION_ID`, then the authenticated identity endpoint.
Recovery must verify the returned `agent_id` equals local `AGENT_ID`; absent,
malformed, or mismatched identity fails closed without generating a UUID or
sending a heartbeat.
- Persist a recovered/configured UUID atomically in the existing SQLite state
before sending heartbeat. Restarts reuse this stored ID. A conflict between
configured and already-persisted values fails closed.
### Job-poll gate
- Add `AGENT_JOBS_ENABLED`, defaulting to `1` to preserve current behavior for
existing deployments.
- When it is `0`, the loop may send heartbeat and sleep, but must not call
`next_job`, claim jobs, classify products, or execute jobs.
- Set `AGENT_JOBS_ENABLED=0` only for this existing installation while
validating the first heartbeat. Leave it disabled after the service is
stopped, until a separate operator action authorizes job processing.
### Deployment boundary
- The identity endpoint is Hub application code only; it needs no DB migration.
Deploy that Hub code without changing the immutable Agent release or the
production SSO/DNS configuration.
- The installed service binary must contain the identity recovery and job gate.
Build only the service-host onedir and update the existing
`GTIFiscalAgentService` payload while the service is stopped. Do not run Setup,
replace `agent.env`/`credentials.bin`, recreate an Agent, or enroll again.
- Keep the published 1.1.1 Setup artifact immutable. The installer source fix
gets regression coverage for a future release; this activation updates only
the installed service runtime needed by the current host.
### Write safety
No Firebird write path is added or enabled. The existing agent configuration
must continue to contain `FIREBIRD_WRITES=0`, `FISCAL_MODE=DRY_RUN`, and an empty
`AUTHORIZED_WRITE_DATABASE_PATH`. The first heartbeat reports metadata only;
this mission must not call `/jobs/next` or a classification endpoint.
## Failure behavior
- Identity endpoint unavailable, no existing installation, or Agent ID mismatch:
fail closed before heartbeat and before job polling.
- SQLite persistence failure: fail closed; do not send a guessed identity.
- Jobs disabled: continue the heartbeat loop without touching the job API.
- If Hub deployment or the service-host update cannot be completed, do not
start the installed service; report the deployment blocker instead.
- Service is stopped immediately after the first successful heartbeat is
observed and verified against Agent 2's existing installation.
## Regression coverage
1. Enrollment response returns the exact `enrollment_uuid` as
`installation_id`; Setup persists it without placing any token in `agent.env`.
2. Authenticated identity endpoint returns only the approved identity fields
and derives hostname/version/status from the existing Agent/Installation.
3. Runtime recovers and persists an ID when local state is missing, validates
Agent ID equality, reuses the ID on restart, and fails closed on mismatch or
missing identity.
4. With `AGENT_JOBS_ENABLED=0`, heartbeat runs but `next_job` is never called;
default-enabled behavior remains covered.
5. Existing write-safety tests continue to prove zero Firebird writes in
`DRY_RUN` / `FIREBIRD_WRITES=0`.
## Acceptance for this activation
- Existing Agent ID remains 2; no enrollment or Agent creation occurs.
- Recovered identity equals the Hub's existing `Installation.installation_id`
and is persisted locally before start.
- Jobs are disabled before service start; service reaches RUNNING, sends one
successful heartbeat for the existing installation, then is stopped.
- Panel shows Agent 2 online with the correct hostname and version 1.1.1; no
duplicate installation is created.
- `FIREBIRD_WRITES=0`, `WRITE_ATTEMPTS=0`, `WRITE_SUCCESSES=0`; no job polling,
classification, or Firebird writes occur.
@@ -0,0 +1,264 @@
# GTI FISCAL WINDOWS INSTALLER EXE V1 — Design
- Date: 2026-09-23
- Status: APPROVED_WITH_ADJUSTMENTS (operator 2026-09-23)
- Related: previous mission "Distribuição ZIP V1" shipped release 1.1.0.
## Objective
Replace the ZIP download UX with a real Windows `.EXE` installer
(`GTIFiscalAgent-Setup-<version>.exe`) plus an installation/enrollment code per
installation. One universal installer per version. Publish release 1.1.1 in the
production Hub, run a REAL Windows E2E, and stop before the first client.
Acceptance (operator visible): Fiscal → Agents → Nova instalação → Criar →
Baixar Agent → browser downloads `GTIFiscalAgent-Setup-1.1.1.exe` directly (an
executable, not a ZIP). Run the EXE on Windows, type the code, install/enroll.
ZIP as the main flow = FAIL.
## Approved adjustments (operator)
1. Setup.exe uses `requireAdministrator`/`--uac-admin` manifest. NO hybrid
asInvoker + self-elevation flow.
2. Setup.exe is self-contained: embeds the Service host + runtime + Uninstall.exe.
It never downloads a second ZIP during install.
3. `install_code` alone resolves the Installation on the Hub. User types only the
code. If both `install_code` and `enrollment_token` are present, the request
is rejected.
4. Production: the permanent credential NEVER goes into `agent.env`. It lives
only in `credentials.bin`, protected by DPAPI LocalMachine + ACL
SYSTEM/Administrators. `agent.env` holds only non-secret configuration plus
`FIREBIRD_WRITES=0`.
5. "Reproducible build" = an automated, repeatable build process and an
immutable release — NOT byte-identical hashes across rebuilds. After
PUBLISHED, 1.1.1 can never be replaced/overwritten.
Also fixed in this design:
- 16 Crockford Base32 chars = 80 bits of entropy for the install code.
- Enrollment rate limit keyed by IP AND by code/installation.
- Install code is normalized (hyphens/case) before hashing.
- Release 1.1.0 stays untouched.
- `FIRST_REAL_CLIENT_AGENT_INSTALLED=false` this mission.
## Installer technology
**Chosen: PyInstaller (all-Python).** No Inno Setup.
Reasoning (INSTALLER_TECHNOLOGY / INSTALLER_REASON):
- PyInstaller 6.21 is already installed on the Windows dev machine; Inno Setup is
not. A fully automated, fail-fast build is achievable today with `pip`-managed
tools.
- The enrollment wizard already needs HTTPS (httpx) and DPAPI (pywin32
win32crypt); those are Python layers, so the wizard must be Python regardless.
Adding Inno Setup would add a second tool without removing the Python wizard
requirement.
- pywin32 gives the battle-tested Windows Service wrapper, and PyInstaller's own
runtime sulution replaces a venv in Program Files, so no Python/pip is needed
on clients.
- The previous mission already validated the service wrapper + `sc.exe failure`
recovery path in the pilot checklist.
- Reproducible build is achieved with a dedicated PowerShell build script (clean
output dir, official VERSION, SHA256, metadata, fail on missing dep); exact
byte-identical output is NOT required.
### Artifacts (universal; built on the Windows dev machine)
| Artifact | PyInstaller mode | Contents / role |
|---|---|---|
| `GTIFiscalAgent-Setup-<v>.exe` | onefile, `--uac-admin` | GUI wizard (tkinter). Embeds Agent runtime sources + service host + `Uninstall.exe` as bundled data. Enroll → write config → DPAPI credential → extract Service host → register service → preflight. Main download artifact. |
| `GTIFiscalAgentService.exe` (+ `_internal/`) | onedir | The Windows Service host (bundled inside Setup.exe, extracted to `C:\Program Files\GTI\FiscalAgent\`). Reuses `gti_agent_windows_service.py` logic; token comes from `credentials.bin`. |
| `Uninstall.exe` | onefile | Stops/removes the service, removes Program Files tree + ARP entry. Invoked from Settings → Apps → GT1 Fiscal Agent → Uninstall. |
Setup.exe behavior:
1. Program runs elevated (requireAdministrator manifest). If not elevated,
Windows UAC prompts.
2. tkinter window: title "GTI Fiscal Agent", a single install-code field, Hub
URL fixed at `https://gtifiscal.gtigrupo.com.br` (no prompt; dev override via
env / CLI flag only).
3. "Instalar" → POST `/v1/agents/enroll` `{"install_code": "XXXX-XXXX-XXXX-XXXX"}`.
4. On success (agent_id, agent_token, hub_url, tenant, release info):
- Write `agent.env` (non-secret): GTI_HUB_URL, AGENT_ID, COMPANY_ID/UF
(optional), FIREBIRD_DATABASE_PATH (blank), FB_USER/FB_PASSWORD blank,
FISCAL_MODE=DRY_RUN, FIREBIRD_WRITES=0, AGENT_VERSION, POLL_INTERVAL_SECONDS,
AGENT_LOG_PATH, AGENT_STATE_DB, EXPECTED_SCHEMA_FINGERPRINT.
- Write `credentials.bin` via DPAPI LocalMachine encryption
(`win32crypt.CryptProtectData(..., CRYPTPROTECT_LOCAL_MACHINE)`); contains
AGENT_TOKEN (and optional secrets). Apply ACL: SYSTEM/Administrators only
(icacls). `agent.env` never holds the token.
- Extract bundled Service host onedir to `C:\Program Files\GTI\FiscalAgent\`.
- Install service `GTIFiscalAgent` (Automatic/Delayed), set recovery
`sc.exe failure "GTIFiscalAgent" reset= 86400 actions= restart/60000/restart/60000/restart/60000`.
5. Preflight: Firebird path not configured → report "serviço instalado; configure
o arquivo de banco para ativar" and do NOT start the service (no error loop).
If fully configured → optionally start.
6. Write ARP uninstall key (DisplayName "GTI Fiscal Agent",
UninstallString = Uninstall.exe).
7. `WINDOWS_INSTALL_COMPLETED` audit (best effort) once install succeeds.
The installer never stores the enrollment code after success, never embeds any
permanent credential, and never enables Firebird/AI writes (no UI to do so).
## Installation code (`install_code`)
- Format: 16 chars, Crockford Base32 alphabet
(`0123456789ABCDEFGHJKMNPQRSTVWXYZ` — excludes I, L, O, U), grouped
`XXXX-XXXX-XXXX-XXXX`. 16 chars × 5 bits = 80 bits entropy.
- Generation: `secrets.randbelow`/random pick of 16 chars; return formatted.
- Normalization before hashing: uppercase, strip hyphens/spaces.
- Storage: only SHA-256 of the normalized code. Never the plaintext.
- Columns on `agent_enrollments` (migration `0023`):
`install_code_hash` (str 64), `install_code_expires_at` (timestamptz),
`install_code_attempts` (int, default 0), `install_code_created_at` (timestamptz).
- TTL: `AGENT_ENROLLMENT_TTL_SECONDS` (default 86400) reused.
- One active code per installation at a time; "Gerar novo código" rotates it
(old code invalidated, previous hash kept for audit but unusable).
- Revocation (`Revogar`) invalidates the code and, when an agent exists,
deactivates it.
- Rate limit: `rate_limit.record_attempt(bucket="enroll_code:"+normalized_code_hash or agent_enrollment_id, origin=IP)` in addition to per-IP bucket.
## Enrollment endpoint (`POST /v1/agents/enroll`)
- Body: `{"install_code": "..."}` OR legacy `{"installation_id": ..., "enrollment_token": ...}`.
- If both `install_code` and `enrollment_token` are provided → 400, `ENROLLMENT_REJECTED` (reason INVALID_REQUEST).
- With `install_code` (primary):
- Resolve the Installation by normalized-code hash. No installation_id needed.
- Validate: exists, status=PENDING (not REVOKED), release PUBLISHED, code not
expired, one-time. `install_code_attempts` guard + rate limit.
- On success: digest the code (mark used), set installation status ENROLLED,
create the Agent with a fresh permanent token, return
`{agent_id, agent_token, installation_id, hub_url, ...}`.
- `ENROLLMENT_STARTED / SUCCEEDED / REJECTED` audit; never log the code.
- Legacy `enrollment_token` path kept for backward compatibility (published 1.1.0
bundles) but is no longer produced for new installations.
## Downloads
- `/agents/releases` main button + pending detail + temp link: serve the SAME
universal EXE (`GTIFiscalAgent-Setup-<v>.exe`) stored as the release artifact.
- Filename/querystring carry no code/token.
- `INSTALLER_DOWNLOADED` audit event on download.
## Release model
- New release row: `version=1.1.1`, `platform=windows`, `architecture=x64`,
`filename=GTIFiscalAgent-Setup-1.1.1.exe`, `status=PUBLISHED`,
`published_at` (per the prior fix), `sha256`, `size_bytes`.
- Storage path generalized (EXE artifact, not hardcoded `.zip`). The internal
ZIP remains a build/debug artifact only (kept under dist/, not the main flow).
- Release 1.1.0 row and artifact remain untouched.
- After PUBLISHED, the artifact is immutable: publish script refuses to replace
an existing PUBLISHED filename for the same version+platform+arch.
## Agent runtime changes (FIREBIRD_WRITES=0, DPAPI credential)
- New module `gti_fiscal_agent/credential_store.py`:
- `read_credential(path)` — decrypt DPAPI LocalMachine blob (fallback: if the
file is a plaintext token, accept in dev only).
- `write_credential(path, payload)` (installer side; reused by Setup.exe) —
encrypt + set ACL.
- `install_env_from_credentials(path, env)` — fill `AGENT_TOKEN` from
`credentials.bin`; `agent.env` stays non-secret.
- `Config`/`main`/service wrapper read the token via
`AGENT_TOKEN` if set, else look for `GTI_CREDENTIAL_FILE` (default
`C:\Program Files\GTI\FiscalAgent\credentials.bin`) and decrypt.
- `FIREBIRD_WRITES=0` enforced (already fail-closed in `Config.firebird_writes`
and `open_write_connection`); drift test asserts the installer template sets 0
and no UI/flag enables it.
## Panel UI changes
- `agent_pending_detail.html` + `agent_detail.html`: show
- Versão, Status,
- Código de instalação `XXXX-XXXX-XXXX-XXXX` + [Copiar]
- Expira em (formatted TTL deadline)
- [Baixar instalador (.EXE)] → EXE
- [Gerar novo código] → rotates code (audit INSTALLATION_CODE_CREATED)
- [Revogar] → revokes (audit INSTALLATION_CODE_REVOKED / INSTALLATION_REVOKED)
- `/agents/releases`: download button → EXE (main flow).
- Never show raw enrollment tokens; the install code is shown to the operator who
created the installation only (SSO-authenticated panel).
## Build script
- `scripts/build_windows_installer.ps1`
- Reads official `release/VERSION` (1.1.1).
- Cleans `dist/installer` output.
- Builds service host (onedir) and Setup.exe (onefile, `--uac-admin`) and
Uninstall.exe with PyInstaller from the bundled sources; Setup embeds the
service host dir + Uninstall.exe as data.
- Computes sha256, writes `GTIFiscalAgent-Setup-<v>.exe.sha256`.
- Fails on missing dependency/tool.
- Deterministic inputs; byte-identical output not required.
## Tests (TDD)
Central (`central/tests`, pytest):
- install code generation/normalization/hash round trip (format, alphabet,
uppercase, hyphen removal, no ambiguous chars).
- enroll with `install_code` resolves installation by code alone; success creates
enrolled agent + credential issuance; `install_code_attempts` increments;
expiry/revoked/used-one-time → REJECTED; code+token together → 400;
rate limit by IP and by code; duplicate/rotation invalidation.
- migration 0023 up/down.
- publish script refuses to overwrite a PUBLISHED artifact and stamps
`published_at`.
- downloads serve EXE with `INSTALLER_DOWNLOADED` audit.
- panel pages render code/button actions without leaking tokens.
Agent (`agent/tests` + root `tests`): credential_store encrypt/decrypt, DRY_RUN /
FIREBIRD_WRITES=0 enforcement, `agent.env` has no token, service wrapper reads
credential.
Build verification: PowerShell script dry-run on dev machine produces EXE +
sha256; PyInstaller starts the setup binary (elevation triggered) or we verify
artifact layout.
## E2E plan
1. Local: build 1.1.1 EXE; launch against the PRODUCTION Hub once the backend is
deployed (or against a local Hub in a scratch container for the UI pass).
2. Production Hub (real, at `https://gtifiscal.gtigrupo.com.br`):
- Create an E2E installation via the panel.
- Generate an install code.
- Download the real EXE (assert filename, MIME application/octet-stream /
executable, not ZIP).
- Run on the real Windows dev machine, enter code, complete enrollment
(real `POST /v1/agents/enroll`), verify:
- Installation → ENROLLED, Agent created.
- `credentials.bin` exists (DPAPI-encrypted, not plaintext token).
- `agent.env` has no token, FIREBIRD_WRITES=0, blank Firebird path.
- Service `GTIFiscalAgent` registered, Automatic/Delayed, recovery set,
NOT started (Firebird not configured) — or started if configured.
- Windows firewall/dir ACL ok (SYSTEM/Administrators).
- Uninstall via Uninstall.exe → service removed, Program Files cleaned.
- Revoke/deactivate the E2E agent in the panel; keep audit trail.
- No real Firebird database is ever connected.
3. Rejections E2E: invalid code, expired code, replayed/used code (each audited
ENROLLMENT_REJECTED), code+token together → 400.
## Deploy sequence
1. Backup production: PostgreSQL logical dump (`pg_dump`) of `gti_fiscal` +
copy of release storage volume (via container) to the backup location.
2. Build image with migration `0023`; `alembic upgrade head` (fresh-DB and
head checks).
3. Deploy Hub (overlay, docker compose up -d --build), postgres intact.
4. Publish release 1.1.1 EXE (immutable).
5. Production smoke: health endpoint, panel pages, download EXE.
6. Run production E2E (above).
7. Validate restore: restore EXE artifact + PG metadata from backup copy.
## Constraints (non-negotiables)
- FIREBIRD_WRITES=0; AI_WRITE_ALLOWED=false; no UI enabling either.
- No real client install this mission.
- No SSO/NPM/DNS/hardware/credentials changes.
- Release 1.1.0 not touched. 1.1.1 immutable once PUBLISHED.
- CODE_SIGNED=false. SmartScreen warning acceptable in the internal pilot.
- Credentials never logged/committed; `GTI_SSH_PASSWORD` env var only.
## Report (final)
Populate the mission's exact field list (STATUS, INSTALLER_TECHNOLOGY=PyInstaller,
INSTALLER_REASON, ..., PANEL_DOWNLOAD_FILENAME=GTIFiscalAgent-Setup-1.1.1.exe,
DPAPI=TRUE, ACL=system+administrators, FIREBIRD_WRITES=0, WINDOWS_E2E=..., etc.).
@@ -0,0 +1,41 @@
# GTI Fiscal — painel operacional com drill-down
**Objetivo.** Refinar as telas atuais Resumo, Agents, Blockers, Jobs e Nova instalação para operação de ~70 clientes, inclusive clientes com 20.000+ produtos. Preservar visual do Gerenciador, stack Jinja/FastAPI/SQLAlchemy e funcionalidades fiscais, Agent, SSO e sincronização existentes. Sem gráficos decorativos, framework frontend novo nem carga de produtos no browser.
## Semântica operacional e histórico
Uma expressão SQL compartilhada define `OPERATIONALLY_ACTIVE` para instalação vinculada a Agent: `Tenant.active IS TRUE`, `Agent.enabled IS TRUE` e ausência de enrollment vinculado em estado `REVOKED`, `EXPIRED` ou `FAILED` (estados finais/inativos). Agent legado sem enrollment não é revogado por inferência; clientes `LOCAL` não são classificados por nome. `Installation.processing_enabled=false` pausa processamento, **não** equivale a instalação revogada: ainda pode comunicar heartbeat e indicar indisponibilidade. Nunca excluir registros por nome `E2E` ou por versão antiga. O registro histórico permanece intacto.
Dashboard usa exclusivamente esta expressão para contagem de saúde, soma de snapshots/cobertura, blockers operacionais, jobs falhos operacionais, versão desatualizada e atenção. Blockers ativos requerem instalação vinculada operacionalmente ativa e `resolved_at IS NULL`; blockers legados sem instalação e blockers resolvidos permanecem nas listagens históricas. A página Agents usa `scope=active` por padrão e `scope=all` para inventário/histórico (com indicação de histórico e badges); Blockers/Jobs usam o mesmo `scope` para a lista e seus KPIs, preservando filtros adicionais. Um card do Resumo sempre leva à visão `scope=active` e ao filtro correspondente. O filtro `scope=all` inclui registros inativos/revogados, e `status=RESOLVED` em Blockers permite acessar problemas resolvidos. Totais do dashboard e contagens da listagem ativa são reconciliados pela mesma expressão SQL.
## Resumo: decisão em um clique
Seis cards `<a>` completos para Total, Online, Degraded, Offline, Failed, Unknown; destino `/agents?scope=active` com `health=<ENUM>` quando aplicável. Labels de apresentação PT-BR: Servidores, Online, Atenção, Offline, Falhou, Não informado. A identidade técnica do filtro permanece na URL e na query SQL. Cobertura: total/completos levam a Agents com ordenação fiscal (visão disponível), bloqueados levam a `/blockers?scope=active&status=OPEN`, falhas técnicas a `/agents?scope=active&health=FAILED`, e não classificados a Agents ordenados por pior cobertura. Sem prometer filtros de produto que o sistema não possui. O breakdown CFOP/ST/ICMS/CSOSN/NCM/OTHER liga valores positivos a `/blockers?scope=active&status=OPEN&field=<ENUM>`; zeros ficam visualmente neutros.
“Atenção necessária” usa até 10 **grupos**, não 10 linhas cruas: primeiro instalações com falha técnica, depois saúde Failed/Degraded/Offline, grupos de blockers fiscais por instalação/campo/tipo/motivo, versões desatualizadas e outros estados realmente acionáveis. Dentro do grupo fiscal: número de blockers abertos e produtos distintos; agrupar **em SQL antes do LIMIT**, com ordenação por gravidade e recência. Um grupo oferece link de lista filtrada de blockers ou detalhe de Agent/job. Instalações revogadas, agentes desabilitados, clientes inativos, itens resolvidos ou sem destino operacional não aparecem. Link “Ver todos” abre lista operacional correspondente. Evitar duplicar a mesma instalação como alerta de versão quando um alerta técnico de prioridade maior já a representa; histórico continua nas listas.
Jobs falhos: contagem e lista curta da operação ativa com Cliente, Agent/hostname, tipo, erro truncado, momento e link `Ver job` para `/jobs?scope=active&status=FAILED&job_id=<id>`. A página Jobs exibe erro integral quando `job_id` é informado. Link “Ver jobs” abre `status=FAILED`. Agents desatualizados: lista compacta em largura suficiente (sem sidebar estreita), cliente/hostname, versão atual/atualização e detalhe do Agent; poucos itens mais “Ver todos” com `version=outdated` aplicado.
## Agents / Servidores
Uma tabela densa, legível em 1366×768 sem scroll horizontal: cliente com instalação/hostname na mesma célula, saúde, fiscal (quantidades resumidas), cobertura, produtos, versão e último contato. Texto secundário só quando existir; badges mostram apenas o valor humano, sem repetir “Saúde técnica:” ou “Status fiscal:”. Busca, saúde (todos os cinco estados), ordenação e scope via GET; chip de filtro ativo e “Limpar filtros”; paginação mantém todos os parâmetros. Link primário visível no cliente/hostname é nativamente acessível por Tab/Enter; clique com mouse na restante da linha abre o mesmo detalhe via delegação pequena em JS, sem eliminar a semântica da tabela. Em telas pequenas, linhas podem assumir apresentação compacta com rótulos por célula, mantendo filtros e ações expostos.
## Blockers
KPIs coerentes com `scope`, `status` e demais filtros (abertos, produtos/clientes afetados, resultado filtrado), com links apenas quando a métrica tem destino fiel. Breakdown filtrado liga campo positivo ao mesmo conjunto de filtros, trocando só `field`; filtro selecionado fica visível. Filtros existentes permanecem server-side (cliente, instalação, campo, tipo, status, idade, busca e identificadores existentes), dispostos em uma ou duas linhas no desktop e empilhados no mobile, com “Limpar filtros” retornando à visão ativa padrão.
Tabela destaca código e descrição do produto como link para `/blockers/<id>`; exibe cliente/servidor, campo, tipo humano, prioridade humana, problema humano, último registro e CTA. Tradutor somente de apresentação mapeia enums conhecidos (`BLOCKED_MISSING_EVIDENCE` → `Evidência fiscal ausente`, `INVALID_NCM` → `NCM inválido`, prioridades `CRITICAL/HIGH/MEDIUM/LOW` → `Crítica/Alta/Média/Baixa`) e cai em rótulo legível para desconhecidos; o detalhe mantém o código técnico original. Paginação limitada (50), consulta com COUNT separado, sem dropdown de produtos nem N+1.
## Jobs
Atalhos compactos para `PENDING`, `RUNNING`, `COMPLETED`, `FAILED`, todos filtrando `scope=active` e mostrando a seleção. Status PT-BR: Pendente, Executando, Concluído, Falhou; tipos conhecidos (`SYNC_INCREMENTAL`, `INITIAL_SYNC`, `REPROCESS_ALL`, `REPROCESS_PRODUCTS`) ganham labels humanos. A lista agrupa cliente e servidor, depois tipo, status, criado/finalizado, duração e erro curto. Erro integral e enum técnico aparecem na vista filtrada `job_id` da própria página, sem criar uma tela artificial. Seleção de scope histórico e “Limpar filtros” mantêm os demais parâmetros úteis. Não acionar jobs, somente leitura do banco.
## Estilo, acessibilidade e escala
Usar somente `central/static/control-panel.css` sobre os tokens imutáveis `gti-theme.css`. Estados semânticos: sucesso verde, atenção amarelo, falha vermelha, histórico/desconhecido cinza, bloqueio fiscal laranja. Foco visível, navegação Tab/Enter e Space nos controles apropriados, links `<a>` em cards; nenhum `div` simulando botão. Mobile 375 px preserva busca e ações, desktop 1366×768 e 1920×1080 aproveitam largura sem blocos de vazio. Testar light/dark por screenshot, não só inspecionando tokens. Combobox de clientes ativos da Nova instalação permanece com busca nome/CNPJ, mouse/teclado, `client_id` hidden e validação server-side; só retoque estético se necessário.
Queries devem usar agregados e grupos SQL, bounded LIMIT, COUNT e paginação server-side; não carregar listas de produtos/snapshots inteiros nem emitir uma consulta por item. Medir número de queries do dashboard e inspecionar planos quando filtros/campos precisarem de índice. Se necessário, migration de índices pequena e reversível; nunca mudar engine fiscal ou dados históricos. Regressões confirmam exclusão de inativos/revogados, coerência card→querystring→resultado, agrupamento, labels, paginação e query count/ausência de N+1.
## Validação e deploy
Rodar suites existentes e regressões focadas. Browser smoke em Resumo, Agents, Blockers, Jobs, Nova instalação; navegar Resumo→Online/Total/NCM/Jobs falhos/Agent desatualizado e voltar; verificar light/dark nos viewports 1920×1080, 1366×768, 375 px e Tab/Enter/Space. Nova instalação não deve criar registros no smoke. Usar **exclusivamente** `scripts/deploy_hub.ps1` para deploy/backup/migration/rollback/health, sem SSH improvisado nem scripts paralelos. Verificar health público e métricas operacionais pós-deploy. Se autenticação visual do Gerenciador não estiver disponível, reportar explicitamente o bloqueio em vez de alegar screenshot de produção.
@@ -0,0 +1,67 @@
# cClassTrib applicability gate
## Purpose
Stop treating an empty `C000025.CCLASSTRIB` as a fiscal defect solely because
the global RTC feature is enabled. Keep the single existing fiscal-completeness
result and continue evaluating every other fiscal field normally.
## Evidence and selected applicability rule
The SGV database has `PARAMETROS.USAR_REFORMA_TRIBUTARIA=1` and RTC-capable
schema, but the inspected `PARAMETROS` and `C000004` schemas have no separate
cClassTrib enable flag. The SGV product screen exposes `C000025.CCLASSTRIB` as
an editable catalog-backed field; available evidence does not mark it required
for every product. The current code incorrectly promotes global RTC capability
to universal per-product requiredness.
Use the live company regime (`C000004.CRT`) together with the documented
effective-date gate:
- RTC capability/`USAR_REFORMA_TRIBUTARIA` must still be enabled for RTC
evaluation generally.
- CRT 3: cClassTrib is applicable from 2026-08-03.
- CRT 1, 2, or 4: cClassTrib is applicable from 2027-01-01.
- Missing, conflicting, or unsupported CRT: cClassTrib is not applicable for
completeness until the company context is resolved; never infer a regime.
- Before the applicable date, ignore cClassTrib completely: no blocker, no
candidate, no completeness penalty, and no write.
- Once applicable, preserve the existing validation/resolution behavior,
including a blocker for an empty value when no safe resolution is available.
This makes current Agent 2 (`CRT=1`, `OPTANTE_SIMPLES=S`) not applicable on
2026-09-25 even though `USAR_REFORMA_TRIBUTARIA=1`. Its current RTC write mode
(`DRY_RUN`) is not changed.
## Minimal implementation boundary
Add the gate to the existing shared fiscal evaluator/context using the already
resolved live company CRT and current date. Do not add a new SGV flag, table,
panel, or second coverage metric. Keep PIS/COFINS, NCM, resolver rules, RTC
write authorization, and Agent write allowlist unchanged. Because the evaluator
is executed by the Agent, this is an Agent code change. After test completion,
the operator explicitly authorized release 1.2.5 through the official Hub/Agent
deploy path and an in-place Agent 2 upgrade that preserves its identity.
The RTC page should present the existing single-panel status as
“cClassTrib: Não utilizado neste cliente” while the gate is false.
## Verification
Add the requested focused tests: not-applicable with empty cClassTrib (RTC off
and on), unrelated fiscal fields still evaluated, applicable with empty
cClassTrib still blocked, and zero write attempts. Then run a read-only
`REPROCESS_ALL` against Agent 2 and compare its latest result/snapshot and
blockers with the captured baseline. Preserve `RTC_WRITE_MODE=DRY_RUN` and
verify Firebird write attempts/successes remain zero. Report NCM/PIS/COFINS
blockers independently.
## Baseline captured before implementation
Agent 2 latest Hub snapshot/job: total 521, complete 0, safe-blocked 521,
failed 0, unclassified 0, coverage 0%; RTC blockers 521; PIS blocker products
1; COFINS blocker products 1; NCM blocker products 26 (29 open blocker rows).
The offline read-only Firebird discovery saw `USAR_REFORMA_TRIBUTARIA=1`,
`SISTEMA_TODO_FISCAL=0`, `C000004.CRT=1`, and
`C000004.OPTANTE_SIMPLES='S'`; its guarded query recorded zero write attempts
and zero successes.