chore: establish project baseline

This commit is contained in:
OpenCode
2026-09-21 15:51:47 -03:00
commit 9dcc263217
459 changed files with 447890 additions and 0 deletions
@@ -0,0 +1,183 @@
# Firebird Discovery Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Executar discovery reproduzível e estritamente read-only do banco Firebird disponível, documentando ambiente, schema, produto, tributação, imagens, incrementalidade, riscos, mapping e fingerprint.
**Architecture:** Um coletor local terá uma camada de validação que aceita somente consultas `SELECT`/metadados e um adaptador de conexão que não expõe operações de escrita. O relatório será gerado a partir de resultados do MCP Firebird e, quando possível, do coletor local; nenhuma tabela auxiliar, migration ou alteração será feita no banco.
**Tech Stack:** Python 3, driver Firebird disponível no ambiente somente se já instalado, `pytest`, SQL de `RDB$`/`INFORMATION_SCHEMA` compatível com Firebird.
## Global Constraints
- O banco deve ser tratado como read-only.
- São proibidos `INSERT`, `UPDATE`, `DELETE`, `ALTER`, `CREATE`, `DROP`, `TRUNCATE`, migrations, triggers, procedures, alteração de índices/constraints, criação de usuário e modificação de permissões.
- Não executar testes destrutivos contra o banco real.
- Não armazenar senhas ou secrets em arquivos, logs ou documentação.
- Não iniciar API fiscal, API de imagens, serviço Windows, atualização de produtos ou scheduler de produção.
- Achados devem ser classificados como `CONFIRMADO`, `INFERIDO` ou `NÃO CONFIRMADO`.
---
### Task 1: Preparar base e contrato read-only
**Files:**
- Create: `src/gti_discovery/__init__.py`
- Create: `src/gti_discovery/read_only.py`
- Create: `tests/test_read_only.py`
- Create: `pyproject.toml`
- Create: `.env.example`
- Create: `CLAUDE.md`
**Interfaces:**
- Produces `validate_read_only(sql: str) -> str`, que retorna SQL normalizado ou lança `ValueError`.
- Produces `ReadOnlyConnection`, com método `execute(sql: str, params: tuple = ())` e sem métodos de escrita.
- [ ] **Step 1: Write the failing tests**
```python
import pytest
from gti_discovery.read_only import validate_read_only
def test_accepts_select_and_with_queries():
assert validate_read_only(" SELECT * FROM RDB$DATABASE ") == "SELECT * FROM RDB$DATABASE"
assert validate_read_only("WITH x AS (SELECT 1 AS n FROM RDB$DATABASE) SELECT n FROM x")
@pytest.mark.parametrize("sql", ["INSERT INTO X VALUES (1)", "UPDATE X SET A=1", "DELETE FROM X", "CREATE TABLE X(A INT)", "ALTER TABLE X ADD A INT", "DROP TABLE X", "TRUNCATE TABLE X", "EXECUTE PROCEDURE P"])
def test_rejects_write_or_side_effect_sql(sql):
with pytest.raises(ValueError):
validate_read_only(sql)
```
- [ ] **Step 2: Run test to verify it fails**
Run: `python -m pytest tests/test_read_only.py -q`
Expected: FAIL because the package and validator do not exist.
- [ ] **Step 3: Implement the minimal guard**
Normalize whitespace, remove one trailing semicolon, reject multiple statements, comments and any first keyword other than `SELECT` or `WITH`. Reject side-effect keywords anywhere in the statement, including procedure execution. `ReadOnlyConnection.execute` must call the validator before delegating to the driver connection.
- [ ] **Step 4: Run test to verify it passes**
Run: `python -m pytest tests/test_read_only.py -q`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add src tests pyproject.toml .env.example CLAUDE.md
git commit -m "feat: add read-only discovery foundation"
```
If Git is still unavailable, record the exact blocker and do not initialize or commit without explicit authorization.
### Task 2: Coletar metadados e fingerprint
**Files:**
- Create: `src/gti_discovery/collector.py`
- Create: `src/gti_discovery/fingerprint.py`
- Create: `tests/test_fingerprint.py`
**Interfaces:**
- `collect_metadata(connection) -> dict` executes only predefined read-only queries.
- `build_fingerprint(metadata: dict) -> str` returns a stable SHA-256 digest over sorted relevant metadata.
- [ ] **Step 1: Write tests for stable fingerprints and changes**
```python
from gti_discovery.fingerprint import build_fingerprint
def test_fingerprint_is_order_independent():
a = {"tables": [{"name": "A", "columns": ["ID", "NAME"]}]}
b = {"tables": [{"columns": ["ID", "NAME"], "name": "A"}]}
assert build_fingerprint(a) == build_fingerprint(b)
def test_relevant_schema_change_changes_fingerprint():
a = {"tables": [{"name": "A", "columns": ["ID"]}]}
b = {"tables": [{"name": "A", "columns": ["ID", "NCM"]}]}
assert build_fingerprint(a) != build_fingerprint(b)
```
- [ ] **Step 2: Run tests and verify failure**
Run: `python -m pytest tests/test_fingerprint.py -q`
Expected: FAIL because fingerprint functions do not exist.
- [ ] **Step 3: Implement collection**
Use Firebird system metadata for database info, relations, fields, keys, indexes, foreign keys, triggers, procedures/packages/events where exposed. Include table counts and approximate row counts only when safely available. Redact values and credentials from serialized output. Query a minimal masked sample only for candidate product tables.
- [ ] **Step 4: Implement canonical fingerprinting**
Canonicalize JSON recursively with sorted object keys and deterministic list ordering for tables, columns, keys, indexes, relations and relevant engine/type metadata; hash UTF-8 canonical JSON with SHA-256. Expose classification helper returning `SCHEMA_SUPPORTED`, `SCHEMA_CHANGED` or `SCHEMA_UNKNOWN` based on an allowlisted expected fingerprint and missing/ambiguous product mapping.
- [ ] **Step 5: Run tests**
Run: `python -m pytest tests -q`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add src/gti_discovery/collector.py src/gti_discovery/fingerprint.py tests
git commit -m "feat: collect Firebird metadata and fingerprints"
```
### Task 3: Execute discovery against the available database
**Files:**
- Create: `artifacts/` output files only if explicitly safe and non-sensitive
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
**Interfaces:**
- Uses the Firebird MCP read-only tools and collector contracts from Tasks 1-2.
- [ ] **Step 1: Capture environment metadata**
Collect Firebird version, database identity without secrets, charset/collation, timezone when detectable, approximate size, relation count, engines, and relevant configuration.
- [ ] **Step 2: Enumerate schema relationships**
Inspect all candidate tables/columns for product, stock, barcode, fiscal, company/branch, image/file/attachment and audit concepts; document PKs, FKs, indexes, nullability, defaults and apparent relationships.
- [ ] **Step 3: Confirm product and fiscal semantics with minimal samples**
Select only masked, non-sensitive columns and a very small sample. Never select credentials, customer data, free-form secrets or full BLOB values. Mark names without direct evidence as `INFERIDO` or `NÃO CONFIRMADO`.
- [ ] **Step 4: Inspect operational objects**
List and describe relevant triggers, procedures, packages and events. Check replication/cache indicators only where readable; explicitly document unavailable evidence.
- [ ] **Step 5: Write the two discovery documents**
Record actual findings and uncertainties, including no-write evidence, images alternatives A/B/C, incremental processing options, risks and future decisions. Do not propose fiscal automation yet.
- [ ] **Step 6: Run offline safety tests and review outputs**
Run: `python -m pytest tests -q`
Expected: PASS; manually verify no password, connection string secret, raw BLOB or unnecessary personal data appears in docs/artifacts.
### Task 4: Update project state and final verification
**Files:**
- Create: `docs/AI_PROJECT_STATE.md`
- Modify: `CLAUDE.md` if needed
- [ ] **Step 1: Record current state**
State that this workspace contains the Firebird discovery target, that no ERP writes were performed, which artifacts exist, and what remains blocked or undecided. Keep it current rather than diary-like.
- [ ] **Step 2: Verify prohibited SQL is absent from executable discovery paths**
Run: `rg -n --glob '!docs/superpowers/plans/**' '\\b(INSERT|UPDATE|DELETE|ALTER|CREATE|DROP|TRUNCATE)\\b' src tests`
Expected: only intentional rejection tests/guard literals, with no executable write path.
- [ ] **Step 3: Run final tests and inspect repository status**
Run: `python -m pytest tests -q`; then `git status --short --branch`, `git diff --check`, and `git log -5 --oneline` when Git is available.
- [ ] **Step 4: Create one final commit**
Stage only mission files and create one clear final commit if the repository exists. If no Git repository exists, report `commit: não criado, diretório sem .git` and do not claim a clean Git status.
@@ -0,0 +1,294 @@
# Firebird Mission 00 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Connect read-only to the installed Firebird environment, inspect only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`, perform evidence-backed discovery, and document the result without application writes.
**Architecture:** A read-only connection module will validate the absolute target path and wrap a compatible Firebird connection configured read-only. A capability-aware collector will query catalog metadata and return sanitized evidence, while separate mapping and fingerprint functions classify findings conservatively. External ERP files are inventory inputs only and are never modified.
**Tech Stack:** Python 3.10+, `fdb` already available in the environment, installed Firebird server/client and native tools when present, pytest, existing `gti_discovery` modules.
## Global Constraints
- Files outside `D:\PROJETOS\GTISISTEMAS\Projeto` are `READ_ONLY_EXTERNAL_SYSTEM`.
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened or queried.
- `OTHER_DATABASES_ACCESSED = 0`.
- `APPLICATION_DDL_DML_WRITES = 0`.
- Do not install an arbitrary Firebird version or change service state.
- Do not execute procedures, generators, DML, DDL, migrations, or metadata changes.
- Never store passwords, connection secrets, raw BLOBs, full routine source, customer data, or unnecessary commercial data.
- Use installed Firebird compatibility evidence before selecting a client or driver.
- Preserve unrelated staged and unstaged work already present in the repository.
- Do not commit unless Git identity is already configured; otherwise report `GIT_IDENTITY_NOT_CONFIGURED`.
---
### Task 1: Inventory the installed Firebird environment
**Files:**
- Read-only external paths under `D:\PROJETOS\GTISISTEMAS\`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-5-report.md`
**Interfaces:**
- Produces a sanitized inventory of services, processes, executable/DLL versions, tools, and relevant ERP configuration paths.
- Does not open any `.FDB` during inventory; only the explicitly authorized target may be opened in a later task.
- [ ] **Step 1: Capture read-only Windows service and process evidence**
Run PowerShell commands that only query state:
```powershell
Get-CimInstance Win32_Service |
Where-Object { $_.Name -match 'Firebird|InterBase' -or $_.DisplayName -match 'Firebird|InterBase' } |
Select-Object Name,DisplayName,State,StartMode,PathName
Get-CimInstance Win32_Process |
Where-Object { $_.Name -match 'firebird|fbserver|fb_inet_server|ibserver' } |
Select-Object Name,ExecutablePath,ProcessId
```
Record names, state, executable path, and architecture indicators only. Do not start, stop, restart, or reconfigure anything.
- [ ] **Step 2: Locate client DLLs, server configuration, and tools without opening databases**
Search only names and file properties under `D:\PROJETOS\GTISISTEMAS\` and standard program directories for `fbclient.dll`, `firebird.conf`, `databases.conf`, `isql.exe`, `gstat.exe`, `gbak.exe`, `fbserver.exe`, and `fb_inet_server.exe`. Use file version metadata, not file replacement or loading against another database.
- [ ] **Step 3: Read ERP configuration files as sanitized text**
Read only `.ini`, `.xml`, `.cfg`, `.conf`, and `.txt` candidates that mention Firebird or database connectivity. Extract host, port, path pattern, client library, charset, and dialect when present. Replace every password, token, user secret, and credential value with `<redacted>` before recording evidence. Do not record the ERP's original database as a target.
- [ ] **Step 4: Run offline regression tests**
Run `python -m pytest tests -q`. Expected: all existing tests pass and no external file is changed.
- [ ] **Step 5: Record sanitized inventory**
Write only the resulting facts and unavailable checks to `task-5-report.md`; include `ERP_FILES_MODIFIED = 0`, no target database access yet, and any version/architecture uncertainty.
- [ ] **Step 6: Commit only if identity is configured**
Check `git config user.name` and `git config user.email`. If both are non-empty, stage only the new report and commit:
```text
git add docs/superpowers/sdd/2026-09-16-firebird-discovery/task-5-report.md
git commit -m "chore: record Firebird installation inventory"
```
If either is empty, do not configure Git and record `GIT_IDENTITY_NOT_CONFIGURED`.
### Task 2: Add an authorized-target connection boundary
**Files:**
- Modify: `src/gti_discovery/read_only.py`
- Create or modify: `src/gti_discovery/firebird_connection.py`
- Test: `tests/test_firebird_connection.py`
**Interfaces:**
- Produces `AUTHORIZED_DATABASE_PATH: Path` and `validate_authorized_database_path(path) -> Path`.
- Produces `connect_authorized_read_only(path, driver, client_library=None)`, which rejects every path except the resolved authorized target and returns a connection adapter with `read_only is True`.
- [ ] **Step 1: Write failing path-guard tests**
```python
def test_only_authorized_target_is_accepted(tmp_path):
target = Path(r"D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB")
assert validate_authorized_database_path(target) == target.resolve()
def test_other_database_is_rejected(tmp_path):
with pytest.raises(ValueError, match="authorized"):
validate_authorized_database_path(tmp_path / "other.FDB")
```
Also test case/relative path normalization, missing target rejection, and rejection of a configured ERP-original path.
- [ ] **Step 2: Run the focused tests and verify failure**
Run `python -m pytest tests/test_firebird_connection.py -q`. Expected: FAIL because the path guard module does not yet exist.
- [ ] **Step 3: Implement the smallest path guard and adapter**
Resolve both requested and authorized paths, compare them using Windows case-insensitive semantics, require the target to be a regular file, and reject any other `.FDB`. Wrap the installed driver connection behind an adapter exposing only `execute`, `close`, and an explicit `read_only = True` marker. Configure a read-only transaction through the driver API when supported; if it cannot be proven, fail closed rather than claim read-only.
- [ ] **Step 4: Add a minimal identity-query contract**
Expose a function that executes only `SELECT MON$DATABASE_NAME FROM MON$DATABASE` through `ReadOnlyConnection`, with no write probe. Assert the adapter is constructed only after the target guard and read-only configuration succeed.
- [ ] **Step 5: Run focused and full tests**
Run `python -m pytest tests/test_firebird_connection.py tests/test_read_only.py -q`, then `python -m pytest tests -q`. Expected: PASS.
- [ ] **Step 6: Commit the boundary when Git identity permits**
```text
git add src/gti_discovery/read_only.py src/gti_discovery/firebird_connection.py tests/test_firebird_connection.py
git commit -m "feat: guard authorized Firebird read-only target"
```
### Task 3: Establish real read-only connectivity
**Files:**
- Create: `scripts/run_firebird_discovery.py`
- Create: `.gitignore` update only if required for sanitized local outputs
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-6-report.md`
**Interfaces:**
- Consumes the installed service/client inventory and `connect_authorized_read_only`.
- Produces sanitized connection facts: driver version, client path/version, architecture, engine version, dialect, ODS, charset, page size, and engine-reported database name.
- [ ] **Step 1: Add a no-write script entry point test**
Test that the script's target argument is fixed to `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`, that an alternate path exits before driver invocation, and that the first SQL is exactly the minimal identity query.
- [ ] **Step 2: Implement installed-client selection**
Use only the version and architecture found in Task 1. Prefer the Firebird server connection form and the matching `fbclient.dll`; do not download or install another client. If no compatible client can be proven, stop with `MISSION_00_BLOCKED` and document the exact reason.
- [ ] **Step 3: Connect and execute the first query**
Connect only to the authorized absolute path, begin a read-only transaction, run the minimal identity query, and retrieve engine information using read-only metadata APIs. Never query or log credentials.
- [ ] **Step 4: Verify no-write boundaries**
Inspect the executed SQL log in memory and assert every statement is validated `SELECT`/`WITH`; assert no call to `commit` for application writes, `rollback` as a probe, procedure execution, or generator modification occurs.
- [ ] **Step 5: Run the script once and record sanitized output**
Write `task-6-report.md` with connection outcome, sanitized strategy, errors, and explicit safety values. Do not write raw query dumps or full database rows.
- [ ] **Step 6: Run offline tests again**
Run `python -m pytest tests -q`. Expected: PASS. If connection fails, preserve the failure as evidence and continue only with offline capability tests.
### Task 4: Expand capability-aware catalog collection
**Files:**
- Modify: `src/gti_discovery/collector.py`
- Modify: `src/gti_discovery/fingerprint.py`
- Test: `tests/test_collector.py`
- Test: `tests/test_fingerprint.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-7-report.md`
**Interfaces:**
- Produces `collect_metadata(connection) -> dict` with database facts, relations, fields, keys, indexes, foreign keys, generators, dependencies, triggers, procedures, parameters, packages, and optional monitoring capability results.
- Produces sanitized rows and explicit `errors`, `unavailable`, and `incomplete` fields.
- [ ] **Step 1: Add failing fixture tests for optional catalogs**
Use in-memory fake read-only readers to verify a missing `RDB$PACKAGES`, `RDB$DEPENDENCIES`, `RDB$GENERATORS`, or monitoring relation becomes an unavailable/optional result, while permission/query failures remain `query_error`.
- [ ] **Step 2: Add all catalog SELECT statements**
Add only catalog queries for the required system relations. Do not project trigger/procedure/package source, defaults, BLOB contents, secrets, or application routine execution. Validate every query through `ReadOnlyConnection`.
- [ ] **Step 3: Implement conservative row sanitization**
Redact binary values, sensitive key names and values, oversized strings, tuple-like raw DB-API rows where necessary, and routine source fields. Preserve only metadata needed for schema mapping and fingerprinting.
- [ ] **Step 4: Implement capability detection and collection**
Execute each optional query independently, record Firebird error type only, and continue with available metadata. Never infer absence from an unavailable catalog.
- [ ] **Step 5: Update fingerprint context**
Include sanitized engine context, relevant catalog structure, product mapping evidence, fiscal fields, image structure, and critical trigger metadata. Preserve ordering for positional fields and index segments; sort unordered entities deterministically.
- [ ] **Step 6: Run tests and record metadata outcome**
Run `python -m pytest tests/test_collector.py tests/test_fingerprint.py tests/test_read_only.py -q`, then run the real collector only against the authorized target. Save sanitized outcome in `task-7-report.md`.
### Task 5: Identify products and map fiscal/image/incremental evidence
**Files:**
- Create: `src/gti_discovery/mapping.py`
- Test: `tests/test_mapping.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-discovery/task-8-report.md`
**Interfaces:**
- Produces `identify_product_candidates(metadata, samples) -> list[dict]`.
- Produces `build_schema_mapping(metadata, samples) -> dict` with each mapping classified as `CONFIRMED`, `INFERRED`, or `UNKNOWN`.
- Produces no write operation and never returns raw BLOB or credential contents.
- [ ] **Step 1: Write failing mapping tests**
Create fixtures with explicit PK/FK/index metadata and masked sample rows. Assert that product identity is `CONFIRMED` only when structural and data evidence agree; name-only candidates are `INFERRED`; absent evidence is `UNKNOWN`.
- [ ] **Step 2: Implement candidate scoring from catalog evidence**
Use relation/field names, data types, PK/FK/index information, approximate counts when available, and minimal permitted scalar samples. Do not copy full rows or treat conventional names as proof.
- [ ] **Step 3: Implement product and stock mapping**
Map only fields directly supported by evidence: PK, internal code, descriptions, barcode, reference, unit, group, brand/manufacturer, status, timestamps, stock relation, and relevant constraints.
- [ ] **Step 4: Implement fiscal inventory mapping**
Inventory NCM, CEST, CST, CSOSN, CFOP, origin, ICMS, PIS, COFINS, IPI, FCP, ST, rates, reductions, nature/operation, company, branch, regime, and UF relationships. Describe representation only; do not implement tax rules.
- [ ] **Step 5: Implement image and incremental mapping**
Identify image/file/path/URL/BLOB candidates with subtype and relationship metadata, excluding binary contents. Identify timestamps, audit tables, generators, version fields, and triggers that can support new/changed product detection. If evidence is absent, return `UNKNOWN` and document the gap.
- [ ] **Step 6: Analyze trigger and routine impact as metadata**
Record relevant trigger timing/events and textual dependencies/source only after sanitization. Identify possible secondary tables and generators from metadata without executing any routine. State that future direct UPDATE safety remains unresolved unless evidence proves semantics.
- [ ] **Step 7: Run mapping tests and produce sanitized evidence**
Run `python -m pytest tests/test_mapping.py tests -q`; generate the mapping report with only masked samples and classifications.
### Task 6: Update current-state documentation and final verification
**Files:**
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
- Create: `docs/MISSION_00_REPORT.md`
- Create or modify: `.superpowers/sdd/2026-09-16-firebird-discovery/task-9-report.md`
**Interfaces:**
- Consumes sanitized reports from Tasks 1 through 5 and the generated fingerprint.
- Produces the mandatory final report with a single verdict and explicit database/filesystem safety assertions.
- [ ] **Step 1: Replace stale connectivity claims with current evidence**
Update the three current-state documents to distinguish confirmed connection facts from unavailable catalogs and inferred mappings. Do not state a schema finding without direct evidence.
- [ ] **Step 2: Write the mandatory final report sections**
Include verdict, filesystem safety, authorized database, other database count, application write count, installed Firebird details, connectivity, database facts, products, fiscal, images, incremental processing, triggers/procedures, fingerprint, tests, Git, gaps, and the next-step prohibition list.
- [ ] **Step 3: Scan for secrets and unauthorized targets**
Search project text for password-like keys, connection strings, raw BLOB markers, ERP-original paths, and other `.FDB` access claims. Remove only newly introduced sensitive content; do not alter unrelated user changes.
- [ ] **Step 4: Run final verification commands**
Run:
```text
python -m pytest tests -q
git diff --check
git status --short --branch
git config user.name
git config user.email
```
Verify `TESTESOFICIAIS.FDB` remains ignored, no external path was modified, no DDL/DML exists in executable discovery paths, and no other `.FDB` was opened.
- [ ] **Step 5: Commit only the mission changes when identity is valid**
If Git identity is configured, stage only the mission files and create one commit:
```text
git add src tests docs .superpowers/sdd/2026-09-16-firebird-discovery
git commit -m "feat: complete read-only Firebird discovery"
```
If identity is not configured, leave the working tree uncommitted and report `GIT_IDENTITY_NOT_CONFIGURED`; never change global Git configuration.
## Plan Self-Review
- The approved design's connection, path guard, sanitization, capability detection, mapping, fingerprint, testing, documentation, and Git requirements are covered by Tasks 1-6.
- Every database operation is constrained to the authorized target and read-only SQL; no task permits service lifecycle changes or external file writes.
- Later interfaces use the exact names and return shapes defined in earlier tasks.
- No task depends on an unproved Firebird version; compatibility is established from installed evidence before connection.
- A failed connection remains a documented `MISSION_00_BLOCKED` outcome rather than a reason to use another database or install an arbitrary client.
@@ -0,0 +1,231 @@
# Mission 00D Structured Firebird Boundary Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace arbitrary SQL execution with a structured read-only Firebird API, pass the pre-connection security gate, then connect only to the authorized database and complete discovery.
**Architecture:** A private executor selects static catalog queries by operation key and is the only layer allowed to call the Firebird cursor. `StructuredFirebirdReader` exposes fixed catalog operations and a catalog-issued identity for safe sampling. The old SQL parser remains offline-only or is removed from runtime authority; the live path never accepts consumer SQL.
**Tech Stack:** Python 3.10+, existing `fdb` 2.0.4 and installed Firebird 2.5.9 client, pytest, existing collector/fingerprint/mapping modules.
## Global Constraints
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened or queried.
- Files outside `Projeto` are `READ_ONLY_EXTERNAL_SYSTEM` and must not be modified.
- `OTHER_DATABASES_ACCESSED = 0`.
- `APPLICATION_DDL_DML_WRITES = 0`.
- The public adapter must not expose `execute(sql)`, `query(sql)`, `raw_execute(sql)`, `execute_statement(sql)`, or equivalents.
- Procedures and UDFs may be inventoried as metadata but never executed.
- Credentials, raw BLOBs, full rows, customer data, original ERP database paths, and secrets must not enter reports or source.
- No Firebird installation, DLL, service, registry, permission, or configuration changes.
- No real database access before the security gate in Task 4 passes.
- Git identity must not be invented or configured; report `GIT_IDENTITY_NOT_CONFIGURED` when absent.
---
### Task 1: Replace the raw execution boundary with a private query executor
**Files:**
- Create: `src/gti_discovery/firebird_queries.py`
- Modify: `src/gti_discovery/read_only.py`
- Modify: `src/gti_discovery/__init__.py`
- Test: `tests/test_read_only_boundary.py`
**Interfaces:**
- Produces private `_execute_catalog(adapter, operation_key, params=()) -> Any`.
- Produces immutable internal query mapping keyed by approved operation names, never by caller-provided SQL.
- Public package exports no raw SQL execution function or `ReadOnlyConnection.execute` method.
- [ ] **Step 1: Write failing API-boundary tests**
```python
def test_public_connection_has_no_sql_execute_method():
connection = make_structured_connection()
assert not hasattr(connection, "execute")
assert not hasattr(connection, "query")
assert not hasattr(connection, "raw_execute")
def test_unknown_operation_key_never_reaches_driver():
adapter = RecordingAdapter()
with pytest.raises(ValueError):
_execute_catalog(adapter, "SELECT * FROM X")
assert adapter.calls == []
```
- [ ] **Step 2: Run the focused tests and verify failure**
Run `python -m pytest tests/test_read_only_boundary.py -q`. Expected: FAIL because the current public wrapper exposes `execute` and the private operation executor does not exist.
- [ ] **Step 3: Define static internal query operations**
Move catalog SQL into `firebird_queries.py` under fixed keys such as
`database`, `relations`, `fields`, `keys`, `indexes`, `foreign_keys`,
`triggers`, `procedures`, `procedure_parameters`, `packages`, `generators`,
`dependencies`, and `monitoring`. The executor accepts only one of these
keys, binds values supplied by the operation implementation, and calls the
private driver cursor. It must not accept SQL text, fragments, clauses, or
callable expressions.
- [ ] **Step 4: Remove runtime authority from `ReadOnlyConnection.execute`**
Replace the public wrapper with a private connection implementation that is
constructed only by the structured reader. Keep read-only transaction setup,
adapter capability checks, close behavior, and fail-closed cleanup. Remove
`validate_read_only` and raw execution from package exports if no runtime
consumer depends on them; retain the parser only as an explicitly offline
diagnostic utility if existing tests need it.
- [ ] **Step 5: Run boundary and existing safety tests**
Run `python -m pytest tests/test_read_only_boundary.py tests/test_read_only.py -q`. Expected: PASS after updating tests to assert the new architecture rather than raw SQL execution.
- [ ] **Step 6: Record the root cause**
Add a report stating that quoted identifiers bypassed lexical callable detection and that the incomplete allowlist rejected `CAST`/aggregates. State that the parser is no longer a runtime security boundary.
### Task 2: Implement the structured catalog reader
**Files:**
- Modify: `src/gti_discovery/collector.py`
- Modify: `src/gti_discovery/firebird_connection.py`
- Test: `tests/test_collector.py`
- Test: `tests/test_firebird_connection.py`
**Interfaces:**
- Produces `StructuredFirebirdReader.get_database_info()`.
- Produces `list_relations()`, `get_relation_columns(relation)`, `get_primary_keys(relation)`, `get_foreign_keys(relation)`, `get_indices(relation)`, `get_triggers()`, `get_procedures()`, `get_generators()`, `get_dependencies()`, and optional monitoring methods.
- `collect_metadata(reader)` accepts only the approved reader runtime type and calls structured methods, never SQL.
- [ ] **Step 1: Write failing structured-reader tests**
Use a fake private executor keyed by operation name and assert each public method requests the expected key. Assert an arbitrary adapter, fake object with matching method names, and raw SQL string are rejected by `collect_metadata`.
- [ ] **Step 2: Implement fixed catalog methods**
Each method calls the private operation executor with a fixed key. Metadata queries may contain `CAST`, `COUNT`, `SUM`, `MIN`, `MAX`, `COALESCE`, joins, and grouping. No query is assembled from caller text. Optional catalog failures become `unavailable`; permission failures remain `query_error`.
- [ ] **Step 3: Preserve sanitized metadata**
Keep field names, relation names, types, nullability, positions, keys, indexes, trigger timing/type, procedure metadata, generator names, and dependency names. Redact binary/source/default/sensitive values and never select routine source or BLOB contents.
- [ ] **Step 4: Update collection and fingerprint fixtures**
Adapt collector tests to the structured reader and assert catalog-shaped rows retain safe fields. Assert fingerprint ordering, positional fields, optional capabilities, and `SCHEMA_UNKNOWN` for incomplete metadata remain correct.
- [ ] **Step 5: Run the focused collector suite**
Run `python -m pytest tests/test_collector.py tests/test_fingerprint.py tests/test_read_only_boundary.py -q`. Expected: PASS with no database access.
### Task 3: Add catalog-issued safe sampling
**Files:**
- Modify: `src/gti_discovery/collector.py`
- Create: `src/gti_discovery/identifiers.py`
- Test: `tests/test_sampling.py`
**Interfaces:**
- Produces `quote_firebird_identifier(name) -> str` for validated identifiers.
- Produces `StructuredFirebirdReader.sample_relation_rows(relation, columns, limit) -> list[dict]`.
- Sampling requires relation and columns previously issued by `list_relations`/`get_relation_columns`; it accepts no SQL, WHERE, ORDER BY, expression, procedure, or UDF.
- [ ] **Step 1: Write failing sampling tests**
Cover valid catalog-issued relation/column identities, quoted names, injection strings, unknown relation/column, procedure-like relation, expression input, negative/oversized/non-integer limit, and raw BLOB redaction.
- [ ] **Step 2: Implement identifier validation and quoting**
Require exact membership in the reader’s catalog snapshot, reject control characters, dots, parentheses, semicolons, comments, and expressions, and quote embedded double quotes using Firebird identifier rules. Preserve the original catalog identity for comparison; never normalize arbitrary text into an accepted identifier.
- [ ] **Step 3: Implement bounded sampling**
Generate the internal sample operation from validated identifiers and a validated limit capped at a small constant. The private executor receives an operation key plus structured identifier parameters; it never receives a complete SQL string from the caller.
- [ ] **Step 4: Sanitize sample output**
Return only permitted scalar metadata evidence. Redact binary values, sensitive names, oversized values, and customer/commercial values not needed for mapping. Never execute selectable procedures or functions.
- [ ] **Step 5: Run sampling tests**
Run `python -m pytest tests/test_sampling.py tests/test_collector.py -q`. Expected: PASS with all injection and quoted-identifier cases rejected structurally.
### Task 4: Pass the pre-connection security gate
**Files:**
- Modify: `tests/test_read_only.py` or replace with offline parser tests
- Modify: `tests/test_firebird_discovery_script.py`
- Modify: `docs/MISSION_00_REPORT.md`
- Create: `.superpowers/sdd/2026-09-16-firebird-mission-00d/task-4-report.md`
**Interfaces:**
- Produces a reproducible gate result proving no public arbitrary SQL reaches the driver.
- Produces no database access.
- [ ] **Step 1: Add regression tests for parser non-authority**
Keep offline tests for quoted identifiers, single/double quotes, comments, semicolons, `EXECUTE PROCEDURE`, selectable procedures, UDFs, `UNION`, subqueries, malicious aliases, reserved words in strings, valid quoted names, and valid relation/column names. Assert these tests exercise only the offline helper and cannot invoke the private executor.
- [ ] **Step 2: Add internal Firebird expression fixtures**
Test static internal query text containing `CAST`, `COUNT`, `SUM`, `MIN`, `MAX`, `COALESCE`, and `GROUP BY` through the operation-key executor fixture. Assert no blacklist parser is called and the driver receives only known internal operations.
- [ ] **Step 3: Run the gate commands**
Run:
```text
python -m pytest tests/test_read_only_boundary.py tests/test_sampling.py tests/test_collector.py tests/test_firebird_connection.py tests/test_firebird_discovery_script.py -q
python -m pytest tests -q
```
Expected: all pass, with no database connection attempt. If any gate condition fails, stop with `MISSION_00_BLOCKED`.
- [ ] **Step 4: Update the implementation report**
Document the actual public methods, private executor, transaction read-only behavior, parser role, and gate result. Do not claim live connectivity yet.
### Task 5: Connect and complete real discovery only after the gate
**Files:**
- Modify: `scripts/run_firebird_discovery.py`
- Modify: `src/gti_discovery/mapping.py`
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
- Modify: `docs/MISSION_00_REPORT.md`
- Create: `.superpowers/sdd/2026-09-16-firebird-mission-00d/task-5-report.md`
**Interfaces:**
- Consumes the structured reader and safe sampling API from Tasks 1-4.
- Produces sanitized engine facts, catalog metadata, UI-to-column mappings, image/incremental/trigger evidence, and real fingerprint only if connection succeeds.
- [ ] **Step 1: Revalidate installed Firebird read-only**
Inspect services, processes, client DLL, tools, and configuration under external ERP paths without changing them. Use the already identified Firebird 2.5.9 client only; do not install or start anything.
- [ ] **Step 2: Connect only to the authorized target**
Require the pre-connection gate result, validate the absolute target, require credentials from the process environment without logging them, configure Firebird read-only isolation, issue the minimal identity query first, and compare the returned identity with the authorized target. On any failure, stop with `MISSION_00_BLOCKED`.
- [ ] **Step 3: Execute structured discovery**
Collect database facts and catalogs through fixed operations. Use only catalog-issued sampling to investigate candidate products, stock, fiscal concepts from the ERP UI (`NCM`, `CFOP`, `CST ICMS`, `ICMS rate`, `CEST`, `CSOSN`, PIS/COFINS, IPI, origin, IAT, incidence, IPPT), image/photo tab structures, timestamps/audit/generators, and trigger/procedure dependencies. Mark each mapping `CONFIRMED`, `INFERRED`, or `UNKNOWN`.
- [ ] **Step 4: Generate fingerprint V1**
Hash sanitized engine context, product structure, fiscal fields, relationships, image structure, trigger metadata, and relevant positional catalog data. Treat incomplete or ambiguous collection as `SCHEMA_UNKNOWN`.
- [ ] **Step 5: Update final documents**
Replace stale blocked claims only when live evidence exists. Otherwise preserve `MISSION_00_BLOCKED` and record the exact blocker. Include all safety invariants and no future write/API/service/scheduler proposal.
- [ ] **Step 6: Run final verification**
Run `python -m pytest tests -q`, `git diff --check`, `git status --short --branch`, and Git identity checks. Confirm no other `.FDB` was accessed, no external file changed, no secret exists in project text, and no DDL/DML path exists.
## Plan Self-Review
- The root-cause, structured boundary, static query ownership, safe sampling, procedure/UDF exclusion, parser demotion, defense in depth, gate, live connection, discovery, fingerprint, documentation, and Git requirements from the specification map to Tasks 1-5.
- No task permits database access before Task 4 passes.
- Later interfaces use exact names introduced by earlier tasks.
- The old parser is never used to authorize internal Firebird query syntax.
- No placeholder, unbounded sampling, arbitrary SQL input, or external file modification is permitted.
@@ -0,0 +1,155 @@
# Mission 00E Provenance and Real Discovery Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans (recommended). Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Formalize the trusted-process boundary, prove the imported `fdb` provenance and Firebird compatibility, then execute complete read-only discovery only against the authorized FDB.
**Architecture:** The existing structured reader and private operation registry remain the runtime boundary; Python same-process introspection is explicitly outside the threat model. A provenance module will inspect module origin, distribution metadata, import paths, and local shadowing before the live driver is imported or used. The final workflow will require this gate, the installed Firebird client, target identity, and complete sanitized discovery evidence.
**Tech Stack:** Python 3.10+, installed `fdb` 2.0.4, installed Firebird 2.5.9.27139 client, `importlib.metadata`, pytest, existing structured catalog/mapping/fingerprint modules.
## Global Constraints
- `TRUSTED_PROCESS = TRUE`; malicious same-process introspection is outside this component boundary.
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened or queried.
- External ERP files under `D:\PROJETOS\GTISISTEMAS\` outside `Projeto` are `READ_ONLY_EXTERNAL_SYSTEM`.
- `APPLICATION_DDL_DML_WRITES = 0`, `OTHER_DATABASES_ACCESSED = 0`, and `ERP_FILES_MODIFIED = 0`.
- No public SQL execution or registry mutation API; procedures/UDFs are metadata-only.
- No dependency update, service/configuration change, arbitrary Firebird installation, or guessed credential.
- No database access before the provenance and safety gates pass.
- Never store credentials, original ERP database paths, raw BLOBs, full rows, or unnecessary personal/commercial data.
- Git identity must not be invented or configured; report `GIT_IDENTITY_NOT_CONFIGURED` when absent.
---
### Task 1: Formalize threat model and registry hardening
**Files:**
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/MISSION_00_REPORT.md`
- Modify: `src/gti_discovery/firebird_queries.py`
- Modify: `src/gti_discovery/catalog.py`
- Test: `tests/test_catalog.py`
- Test: `tests/test_read_only_boundary.py`
**Interfaces:**
- Preserves the structured public reader and private immutable operation registry.
- Produces no public operation-registration mutator and no raw SQL API.
- [ ] **Step 1: Write failing threat-model/registry tests**
Assert `TRUSTED_PROCESS = TRUE` is documented, package exports contain no raw execution, registry maps are immutable, no public registration function exists, and normal callers cannot mutate approved operation types. Do not test deliberate access to Python internals as a security requirement.
- [ ] **Step 2: Implement minimal registry hardening**
Keep operation registration module-private and centralized; use immutable mappings and exact approved reader/adapter types. Remove only accidental public aliases or mutators. Do not add cryptographic or sandbox-like mechanisms.
- [ ] **Step 3: Document actual boundary**
State which protections are runtime, which parser code is offline-only, and that a process already running arbitrary Python is outside scope. Remove claims that imply cryptographic isolation.
- [ ] **Step 4: Run focused/full tests**
Run `python -m pytest tests/test_catalog.py tests/test_read_only_boundary.py -q` and `python -m pytest tests -q`. Expected: PASS without database access.
### Task 2: Prove `fdb` module provenance and shadowing absence
**Files:**
- Create: `src/gti_discovery/provenance.py`
- Create: `tests/test_provenance.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-mission-00e/task-2-report.md`
**Interfaces:**
- Produces `inspect_fdb_provenance() -> dict` with `module_file`, `module_package`, module version, distribution name/version/location, interpreter, working directory, relevant `sys.path`/`PYTHONPATH`, and shadowing findings.
- Produces `validate_fdb_provenance(expected_version="2.0.4") -> dict` and raises a sanitized error on ambiguity/mismatch.
- [ ] **Step 1: Write failing provenance tests**
Test installed module metadata correlation, module file under expected `site-packages`, distribution/version agreement, detection of local `fdb.py`/`fdb` directory and unexpected `.pth`, and rejection of a shadowed/mismatched module. Use monkeypatched fixtures; never import or connect to another database.
- [ ] **Step 2: Implement provenance inspection**
Use `importlib.util.find_spec`, `importlib.metadata.distribution/version`, `fdb.__file__`, `fdb.__package__`, `sys.executable`, `Path.cwd()`, `sys.path`, and `PYTHONPATH`. Resolve the actual file and distribution location. Do not print credential values or scan external databases.
- [ ] **Step 3: Implement local shadowing scan**
Search only `Projeto` for exact `fdb.py`, `fdb` directory, and `.pth` candidates. Treat ambiguous duplicate locations as blocked. Record paths only for project-local shadowing evidence, not ERP database paths.
- [ ] **Step 4: Run provenance tests and write report**
Run `python -m pytest tests/test_provenance.py tests/test_read_only_boundary.py -q` and `python -m pytest tests -q`. Write sanitized provenance results and `GIT_IDENTITY_NOT_CONFIGURED` status to the task report.
### Task 3: Pin and validate the proven dependency/client pair
**Files:**
- Modify: `pyproject.toml`
- Modify: `scripts/run_firebird_discovery.py`
- Create: `tests/test_dependency_compatibility.py`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-mission-00e/task-3-report.md`
**Interfaces:**
- Produces `validate_firebird_runtime() -> dict` that requires the proven `fdb` distribution/version and installed Firebird 2.5.9 client evidence before driver use.
- Produces sanitized client path/version/architecture evidence.
- [ ] **Step 1: Write failing compatibility tests**
Test wrong distribution version, module/distribution mismatch, missing client, wrong client version, shadowing, unsupported read-only TPB, and valid installed `fdb 2.0.4` plus Firebird `2.5.9.27139` fixtures. Tests must fail before any driver connection.
- [ ] **Step 2: Record the existing dependency without unnecessary installation**
Add the appropriate exact `fdb` 2.0.4 dependency declaration only if the project dependency mechanism requires it; do not download or update packages. Document why this version matches the installed Firebird 2.5.9 client.
- [ ] **Step 3: Implement pre-import compatibility gate**
Validate distribution metadata, module provenance, client DLL existence/version, architecture evidence, and exact installed read-only isolation API before importing/using `fdb` for connection. Fail closed on ambiguity.
- [ ] **Step 4: Run tests and record evidence**
Run `python -m pytest tests/test_dependency_compatibility.py tests/test_provenance.py -q` and `python -m pytest tests -q`. Do not connect to the database.
### Task 4: Execute gated real discovery and finalize documentation
**Files:**
- Modify: `scripts/run_firebird_discovery.py`
- Modify: `src/gti_discovery/mapping.py`
- Modify: `docs/AI_PROJECT_STATE.md`
- Modify: `docs/DATABASE_DISCOVERY.md`
- Modify: `docs/SCHEMA_MAPPING.md`
- Modify: `docs/MISSION_00_REPORT.md`
- Create: `docs/superpowers/sdd/2026-09-16-firebird-mission-00e/task-4-report.md`
**Interfaces:**
- Consumes provenance/runtime validation and structured reader/sampling APIs from Tasks 1-3.
- Produces sanitized connection facts, full catalog discovery, product/UI fiscal mapping, image/incremental/trigger evidence, and a real fingerprint only when complete.
- [ ] **Step 1: Run the complete pre-connection gate**
Run all focused tests and `python -m pytest tests -q`. Confirm no public SQL API, immutable registry, provenance match, no shadowing, compatible client, and green suite. If any fails, write `MISSION_00_BLOCKED` and do not connect.
- [ ] **Step 2: Inspect installed Firebird evidence read-only**
Use services/processes/DLL/tools/configuration only as read-only evidence. Do not start/stop/restart services or modify external ERP files. Keep credentials and original database paths out of reports.
- [ ] **Step 3: Connect to the sole authorized target**
Require process credentials without logging them, validate the absolute allowlist path, configure Firebird transaction READ ONLY, issue the minimal identity query first, and compare returned identity. Never use another `.FDB`, write probe, procedure, UDF, generator, or DDL/DML.
- [ ] **Step 4: Run structured discovery**
Collect all available Firebird catalogs through operation keys, use catalog-issued bounded samples, and map product fields plus every UI fiscal field (`NCM`, CFOP/CST entrada/saída, ICMS, CEST, CSOSN, PIS/COFINS, IPI, origem, IAT, incidência, IPPT). Investigate Foto/image/BLOB subtype/relationship, incremental evidence, and trigger/procedure dependencies without execution.
- [ ] **Step 5: Produce fingerprint and current documents**
Generate a sanitized real fingerprint only for complete evidence and classify it. Update current-state documents with the threat model, provenance, client, connection, schema, mappings, gaps, and exact safety counters. Preserve `MISSION_00_BLOCKED` if any live prerequisite or discovery result is unavailable.
- [ ] **Step 6: Final verification and Git status**
Run `python -m pytest tests -q`, `git diff --check`, `git status --short --branch`, `git config user.name`, and `git config user.email`. Verify no external file changes and no other database access. Do not configure Git or commit if identity is absent.
## Plan Self-Review
- The specification's threat model, registry hardening, provenance, shadowing, dependency/client validation, pre-import gate, real connection, discovery, UI mapping, image/incremental/trigger review, documentation, tests, and Git requirements are covered by Tasks 1-4.
- No database access is permitted before Task 4 Step 1 passes.
- `TRUSTED_PROCESS = TRUE` avoids pretending Python private objects are cryptographic isolation while retaining accidental-use defenses.
- All runtime interfaces use structured operations; the old parser cannot authorize internal SQL.
- No placeholder instructions, arbitrary sampling, dependency download, service mutation, or external file write is permitted.
@@ -0,0 +1,64 @@
# Fiscal Data Foundation V1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add isolated, versioned official fiscal-data ingestion without product tax determination.
**Architecture:** A shared dataset lifecycle stores auditable metadata and retains active versions. Each domain parser/importer validates its own official payload and fails independently.
**Tech Stack:** Python, SQLAlchemy, Alembic, FastAPI, pytest.
## Global Constraints
- Use only official primary sources.
- Preserve existing NCM and CEST imports.
- Never write to Firebird or classify products.
- Failed imports cannot supersede active datasets.
---
### Task 1: Dataset Version Metadata
**Files:**
- Modify: `central/app/models.py`
- Create: `central/alembic/versions/0003_dataset_foundation_metadata.py`
- Test: `central/tests/test_dataset_lifecycle.py`
- [ ] Write a failing test asserting `parser_version`, effective dates, raw reference, and failure reason persist.
- [ ] Run `python -m pytest central/tests/test_dataset_lifecycle.py -q` and verify failure.
- [ ] Add the model fields and migration.
- [ ] Re-run the focused test and verify pass.
### Task 2: Isolated Lifecycle Service
**Files:**
- Modify: `central/app/datasets/service.py`
- Test: `central/tests/test_dataset_lifecycle.py`
- [ ] Write failing tests for a failed replacement retaining its prior active dataset and for independent domains.
- [ ] Run focused tests and verify failure.
- [ ] Implement pending, failed, and active promotion methods scoped by domain.
- [ ] Re-run focused tests and verify pass.
### Task 3: Official Domain Importers
**Files:**
- Create: `central/app/datasets/foundation.py`
- Create: `central/scripts/import_foundation.py`
- Test: `central/tests/test_foundation_parsers.py`
- [ ] Write failing parser tests for official structured payload validation and unsupported structures.
- [ ] Run focused tests and verify failure.
- [ ] Implement versioned fail-closed adapters for BA_ST_MVA, CFOP, CST_CSOSN, PIS_COFINS, TIPI_IPI, CCLASSTRIB_IBSCBS, FCP, and MONOFASICO_ICMS.
- [ ] Re-run focused tests and verify pass.
### Task 4: Active Dataset Smoke Matrix
**Files:**
- Modify: `central/app/api.py`
- Test: `central/tests/test_dataset_status.py`
- [ ] Write failing test for active-only dataset status with lifecycle metadata.
- [ ] Run focused test and verify failure.
- [ ] Add the active dataset status/smoke response.
- [ ] Run `python -m pytest tests central/tests agent/tests -q` and verify pass.
@@ -0,0 +1,523 @@
# GTI Fiscal Hub V1 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build and exercise a Docker-hosted FastAPI fiscal hub, official local NCM registry, and read-only Windows-ready Firebird agent for 20 real products.
**Architecture:** A modular FastAPI monolith owns versioned datasets, classification, determination placeholders, cache, and audit in PostgreSQL. A separate agent uses the existing Firebird read-only boundary plus SQLite outbox state to submit product batches and create comparison artifacts, without any ERP writes.
**Tech Stack:** Python 3.12, FastAPI, Pydantic, SQLAlchemy, Alembic, psycopg, PostgreSQL 16, httpx, Docker Compose, SQLite, pytest, `fdb==2.0.4`.
## Global Constraints
- The only canonical NCM source is `https://portalunico.siscomex.gov.br/classif/api/publico/nomenclatura/download/json`.
- Verify the downloaded JSON root has `Data_Ultima_Atualizacao_NCM`, `Ato`, and `Nomenclaturas`; each nomenclature must contain `Codigo`, `Descricao`, `Data_Inicio`, and `Data_Fim`.
- Store raw-payload SHA-256, provenance, record count, and version metadata; preserve prior dataset versions.
- Promote a dataset atomically from `READY` to `ACTIVE` while changing the former active version to `SUPERSEDED` in one transaction.
- No paid APIs, LLMs, photos, BLOB reads, Firebird DML/DDL, ERP schema changes, or Firebird writes.
- Central PostgreSQL is Hub-only; the agent communicates only through authenticated HTTP.
- Every response contains `engine_version`, `dataset_versions`, and `decision_version`.
- `NCM_VALIDATED` does not authorize fiscal-field writes; `can_auto_apply` is always `false` in Mission 02.
- Use the reference fingerprint `1198f89897532b643439f06b729e93b189018f56e7d478e95095f8f56977005b` as `REFERENCE_SCHEMA_V1`.
- Preserve existing worktree changes. Do not configure Git identity or create a commit unless an identity already exists.
## Planned File Structure
- `central/app/config.py`: environment-backed Hub settings.
- `central/app/db.py`: SQLAlchemy engine/session lifecycle.
- `central/app/models.py`: PostgreSQL ORM records.
- `central/app/schemas.py`: HTTP request and response contracts.
- `central/app/datasets/ncm.py`: verified source parsing, import validation, and normalization.
- `central/app/datasets/service.py`: dataset lifecycle and atomic promotion.
- `central/app/classification.py`: exact NCM validation and deterministic candidate ranking.
- `central/app/providers.py`: fiscal integration statuses and RTC provider interface.
- `central/app/auth.py`: bearer token verification and tenant/agent binding.
- `central/app/api.py`: versioned routes and dependencies.
- `central/app/main.py`: application construction and health checks.
- `central/alembic/`: migration environment and initial schema migration.
- `central/scripts/import_ncm.py`: Hub NCM update command.
- `agent/src/gti_fiscal_agent/`: Firebird reader, local state/outbox, client, and runner.
- `agent/scripts/run_agent.py`: CLI for the 20-product run.
- `central/tests/` and `agent/tests/`: focused unit/integration tests.
- `docker-compose.yml`, `central/Dockerfile`, `.env.example`: development operation.
- `docs/MISSION_02_PRODUCT_RESULTS.json`, `docs/MISSION_02_REPORT.md`, and `docs/FISCAL_DATASETS.md`: proof and source documentation.
---
### Task 1: Create Hub, Agent, and Docker Foundations
**Files:**
- Create: `central/pyproject.toml`
- Create: `central/__init__.py`
- Create: `central/app/__init__.py`
- Create: `central/app/config.py`
- Create: `central/app/main.py`
- Create: `central/Dockerfile`
- Create: `docker-compose.yml`
- Modify: `.env.example`
- Test: `central/tests/test_health.py`
**Interfaces:**
- Produces `Settings`, `create_app() -> FastAPI`, and a Compose service named `fiscal-hub`.
- Consumes no new internal components.
- [ ] **Step 1: Write the failing health test**
```python
from fastapi.testclient import TestClient
from central.app.main import create_app
def test_health_reports_database_dependency():
client = TestClient(create_app(database_probe=lambda: True))
assert client.get("/health").json() == {"status": "ok"}
```
- [ ] **Step 2: Run the focused test to verify it fails**
Run: `python -m pytest central/tests/test_health.py -q`
Expected: import failure because `central.app.main` does not exist.
- [ ] **Step 3: Implement the minimal configuration and health application**
```python
def create_app(database_probe: Callable[[], bool] = database_is_available) -> FastAPI:
app = FastAPI(title="GTI Fiscal Hub", version="0.1.0")
@app.get("/health")
def health():
if not database_probe():
raise HTTPException(status_code=503, detail="postgres unavailable")
return {"status": "ok"}
return app
```
Create Compose services for PostgreSQL 16 and `fiscal-hub`, each with a healthcheck. Configure `DATABASE_URL`, `HUB_PORT`, `AGENT_DEV_TOKEN`, `NCM_DATASET_URL`, `DATASET_STORAGE_PATH`, `LOG_LEVEL`, and `HTTP_TIMEOUT_SECONDS` in `.env.example` without values that are real secrets.
- [ ] **Step 4: Run focused tests and Compose validation**
Run: `python -m pytest central/tests/test_health.py -q`
Expected: PASS.
Run: `docker compose config`
Expected: rendered `postgres` and `fiscal-hub` services with healthchecks.
- [ ] **Step 5: Commit if Git identity exists**
Run: `git config user.name`
If it exits nonzero, record `GIT_IDENTITY_NOT_CONFIGURED` and do not commit. Otherwise: `git add central docker-compose.yml .env.example && git commit -m "feat: scaffold fiscal hub services"`.
### Task 2: Add PostgreSQL Schema and Dataset Lifecycle
**Files:**
- Create: `central/app/db.py`
- Create: `central/app/models.py`
- Create: `central/alembic.ini`
- Create: `central/alembic/env.py`
- Create: `central/alembic/versions/0001_initial.py`
- Create: `central/app/datasets/service.py`
- Test: `central/tests/test_dataset_lifecycle.py`
**Interfaces:**
- Produces `DatasetService.create_pending()`, `mark_ready()`, and `promote(dataset_id)`.
- Produces ORM tables `datasets`, `ncm_entries`, `tenants`, `agents`, `classification_requests`, `classification_results`, and `classification_cache`.
- `promote()` is consumed by the importer in Task 3.
- [ ] **Step 1: Write lifecycle tests**
```python
def test_promotion_supersedes_previous_active_in_one_commit(session):
old = datasets.create_ready(session, version="old", sha256="a" * 64)
datasets.promote(session, old.id)
new = datasets.create_ready(session, version="new", sha256="b" * 64)
datasets.promote(session, new.id)
assert datasets.status(session, old.id) == "SUPERSEDED"
assert datasets.status(session, new.id) == "ACTIVE"
```
- [ ] **Step 2: Run lifecycle test to verify failure**
Run: `python -m pytest central/tests/test_dataset_lifecycle.py -q`
Expected: import failure for `DatasetService`.
- [ ] **Step 3: Implement models, migration, and transaction-scoped promotion**
Define a unique dataset identity on `(provider, sha256)`. Scope `ncm_entries` to `dataset_id` with an index on `(dataset_id, code)`. Store only `token_hash` in `agents`. In `promote`, lock current active NCM dataset rows with `SELECT ... FOR UPDATE`, verify target is `READY`, update old active rows to `SUPERSEDED`, update target to `ACTIVE`, and commit once. Roll back on every exception.
- [ ] **Step 4: Run migration and tests against PostgreSQL**
Run: `docker compose up -d postgres`
Run: `docker compose exec fiscal-hub alembic upgrade head`
Run: `python -m pytest central/tests/test_dataset_lifecycle.py -q`
Expected: migration succeeds; lifecycle test passes.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: add fiscal hub persistence"`.
### Task 3: Implement Official NCM Download, Validation, and Import
**Files:**
- Create: `central/app/datasets/ncm.py`
- Create: `central/scripts/import_ncm.py`
- Create: `central/tests/fixtures/ncm_official_sample.json`
- Test: `central/tests/test_ncm_import.py`
**Interfaces:**
- Produces `parse_ncm_payload(payload: bytes) -> NcmPayload`, `import_ncm(session, payload, source_url, downloaded_at) -> Dataset`.
- `NcmPayload` contains `version_label`, `act`, and `entries: list[NcmEntryInput]`.
- Consumed by the import CLI and API dataset status route.
- [ ] **Step 1: Write parser and idempotency tests using the verified source shape**
```python
def test_parser_accepts_verified_siscomex_shape():
parsed = parse_ncm_payload(b'{"Data_Ultima_Atualizacao_NCM":"Vigente em 17/09/2026","Ato":"Resolução Gecex nº 926/2026","Nomenclaturas":[{"Codigo":"08043000","Descricao":"Abacaxis.","Data_Inicio":"01/04/2022","Data_Fim":"31/12/9999"}]}')
assert parsed.entries[0].code == "08043000"
def test_import_same_sha_returns_existing_dataset(session):
first = import_ncm(session, payload, SOURCE_URL, now)
second = import_ncm(session, payload, SOURCE_URL, now)
assert first.id == second.id
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest central/tests/test_ncm_import.py -q`
Expected: import failure for `parse_ncm_payload`.
- [ ] **Step 3: Implement strict parser and staged import**
Calculate `hashlib.sha256(payload).hexdigest()` before parsing. Require an object root with the known `Data_Ultima_Atualizacao_NCM`, `Ato`, and nonempty `Nomenclaturas` members; tolerate additional metadata but reject missing or incorrectly typed known members. Require string `Codigo`, `Descricao`, `Data_Inicio`, and `Data_Fim` per entry. Normalize code by removing non-digits; retain all hierarchy rows but validate product NCMs only against eight-digit entries. Reject duplicate normalized `(code, valid_from, valid_to)` values and implausible totals below 1,000 entries. Insert the dataset as `PENDING`, bulk-insert all entries, validate inserted count equals parsed count, mark `READY`, then call Task 2 promotion. Never delete rows from a prior dataset.
The CLI fetches with `httpx`, requires a successful response, writes the raw payload under the configured dataset storage path using its SHA-256 filename, then invokes the transactional importer.
- [ ] **Step 4: Run importer tests and actual source import**
Run: `python -m pytest central/tests/test_ncm_import.py -q`
Expected: PASS.
Run: `docker compose exec fiscal-hub python -m central.scripts.import_ncm`
Expected: one `ACTIVE` NCM dataset with recorded source URL, SHA-256, update label, act, downloaded timestamp, and count.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: import versioned official NCM data"`.
### Task 4: Add Deterministic Product Classification and Provider Boundaries
**Files:**
- Create: `central/app/classification.py`
- Create: `central/app/providers.py`
- Test: `central/tests/test_classification.py`
- Test: `central/tests/test_providers.py`
**Interfaces:**
- Produces `normalize_code(value) -> str`, `normalize_description(value) -> str`, and `Classifier.classify(request) -> ClassificationDecision`.
- Produces `RtcCalculatorProvider.determine(context) -> ProviderResult` and `NotConfiguredRtcCalculator`.
- Consumed by API routes in Task 5.
- [ ] **Step 1: Write exact-validation and candidate-ranking tests**
```python
def test_valid_existing_ncm_is_validated_without_reclassification(repository):
decision = Classifier(repository).classify(product(current_ncm="0804.30.00", description="anything"))
assert decision.status == "NCM_VALIDATED"
assert decision.classification.ncm == "08043000"
assert decision.can_auto_apply is False
def test_unmatched_description_is_not_invented(repository):
decision = Classifier(repository).classify(product(current_ncm=None, description="RISOLE DE CAMARAO"))
assert decision.status in {"HIGH_CONFIDENCE_CANDIDATE", "AMBIGUOUS", "NOT_FOUND"}
assert decision.can_auto_apply is False
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest central/tests/test_classification.py central/tests/test_providers.py -q`
Expected: import failure for `Classifier`.
- [ ] **Step 3: Implement normalizers, ranking, and NOT_CONFIGURED RTC adapter**
Normalize descriptions with Unicode decomposition, accent removal, uppercase conversion, punctuation-to-space conversion, and collapsed whitespace. Rank candidates with token overlap between normalized description tokens and active NCM descriptions; only return a candidate when its score is at least `0.90` and exceeds the second candidate by `0.15`. Otherwise return `AMBIGUOUS` or `NOT_FOUND`. Leave CEST and fiscal determination fields `null` with reason codes such as `FISCAL_DETERMINATION_NOT_CONFIGURED`.
Implement the RTC protocol to return `{status: "NOT_CONFIGURED", values: {}}`; do not call an external service.
- [ ] **Step 4: Run focused tests**
Run: `python -m pytest central/tests/test_classification.py central/tests/test_providers.py -q`
Expected: PASS.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: add deterministic fiscal classification"`.
### Task 5: Implement Authenticated, Cached `/v1` API
**Files:**
- Create: `central/app/schemas.py`
- Create: `central/app/auth.py`
- Create: `central/app/api.py`
- Modify: `central/app/main.py`
- Test: `central/tests/test_api.py`
**Interfaces:**
- Produces `POST /v1/validate/ncm`, `POST /v1/classify`, `POST /v1/classify/batch`, and `GET /v1/datasets/status`.
- Consumes `Classifier`, `DatasetService`, and a bearer-token `AgentPrincipal`.
- Returns `ClassificationResponse` containing provenance versions and `can_auto_apply=False`.
- [ ] **Step 1: Write API auth, isolation, cache, and idempotency tests**
```python
def test_classify_requires_agent_token(client):
assert client.post("/v1/classify", json=payload()).status_code == 401
def test_same_authenticated_request_uses_cache(client, token):
first = client.post("/v1/classify", headers={"Authorization": f"Bearer {token}"}, json=payload())
second = client.post("/v1/classify", headers={"Authorization": f"Bearer {token}"}, json=payload())
assert first.json()["request_fingerprint"] == second.json()["request_fingerprint"]
assert second.json()["provenance"]["cache_hit"] is True
```
- [ ] **Step 2: Run API tests to verify failure**
Run: `python -m pytest central/tests/test_api.py -q`
Expected: import failure for API routes.
- [ ] **Step 3: Implement models and route behavior**
Hash configured bootstrap tokens with SHA-256 at provisioning and compare bearer tokens with `hmac.compare_digest`. Resolve `tenant_id` and `agent_id` from the token; reject a payload whose supplied IDs differ. Build cache and request fingerprints from resolved tenant, normalized product/context, `ENGINE_VERSION`, `DECISION_VERSION`, and active dataset versions. Persist one request/result audit pair per new fingerprint and return cache metadata for repeats. Batch must preserve input order and return one decision per product. Health returns 503 if the database probe fails.
- [ ] **Step 4: Run API tests and smoke test**
Run: `python -m pytest central/tests/test_api.py -q`
Expected: PASS.
Run: `curl http://localhost:8000/health`
Expected: `{"status":"ok"}` after Compose is up.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add central && git commit -m "feat: expose authenticated fiscal API"`.
### Task 6: Build Agent State, Outbox, and HTTP Client
**Files:**
- Create: `agent/pyproject.toml`
- Create: `agent/src/gti_fiscal_agent/__init__.py`
- Create: `agent/src/gti_fiscal_agent/state.py`
- Create: `agent/src/gti_fiscal_agent/client.py`
- Create: `agent/src/gti_fiscal_agent/fingerprints.py`
- Test: `agent/tests/test_state.py`
- Test: `agent/tests/test_client.py`
**Interfaces:**
- Produces `AgentState.enqueue(product)`, `pending_batch()`, `mark_sent()`, `mark_result_received()`, and `product_fingerprint(product) -> str`.
- Produces `HubClient.classify_batch(products) -> list[dict]`.
- Consumed by the Firebird runner in Task 7.
- [ ] **Step 1: Write outbox and offline-client tests**
```python
def test_unchanged_product_with_current_versions_is_not_enqueued(state):
state.mark_result_received("000276", source_fingerprint="a", dataset_version="v1", engine_version="0.1.0")
assert state.enqueue("000276", "a", "v1", "0.1.0") is False
def test_offline_client_leaves_row_pending(state, offline_client):
state.enqueue("000276", "a", "v1", "0.1.0")
send_pending(state, offline_client)
assert state.status("000276") == "PENDING_API"
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest agent/tests/test_state.py agent/tests/test_client.py -q`
Expected: import failure for `AgentState`.
- [ ] **Step 3: Implement SQLite state machine and client**
Create a SQLite `product_state` table keyed by product ID and an `outbox` table keyed by product ID plus request fingerprint. Enforce transitions `DISCOVERED -> PENDING_API -> SENT -> RESULT_RECEIVED`; define but do not use `APPLIED`. Persist retry count and next-attempt timestamp. Build product fingerprints from description, GTIN, NCM, `CLASSIFICACAO_FISCAL`, CFOP, CST, aliquota, CEST, and CSOSN. On connection error or non-success HTTP status, increment retry metadata and restore `PENDING_API`; never raise a Firebird write path.
- [ ] **Step 4: Run focused tests**
Run: `python -m pytest agent/tests/test_state.py agent/tests/test_client.py -q`
Expected: PASS.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add agent && git commit -m "feat: add fiscal agent outbox"`.
### Task 7: Integrate Read-Only Firebird Agent and Generate the 20-Product Proof
**Files:**
- Create: `agent/src/gti_fiscal_agent/firebird_reader.py`
- Create: `agent/src/gti_fiscal_agent/runner.py`
- Create: `agent/scripts/run_agent.py`
- Test: `agent/tests/test_firebird_reader.py`
- Test: `agent/tests/test_runner.py`
- Create: `docs/MISSION_02_PRODUCT_RESULTS.json`
- Create: `docs/MISSION_02_REPORT.md`
**Interfaces:**
- Produces `read_products(catalog, codes) -> list[AgentProduct]` and `run_once() -> RunSummary`.
- Consumes `gti_discovery` closed read-only catalog APIs, `AgentState`, and `HubClient`.
- [ ] **Step 1: Write fingerprint gate and payload-mapping tests**
```python
def test_unknown_schema_does_not_read_or_send(reader, client):
result = run_once(reader.with_fingerprint("other"), client)
assert result.status == "SCHEMA_UNKNOWN"
assert client.calls == []
def test_payload_uses_classificacao_fiscal_when_ncm_is_empty(reader):
product = reader.product("000276")
assert to_hub_product(product).current_ncm == "08043000"
```
- [ ] **Step 2: Run tests to verify failure**
Run: `python -m pytest agent/tests/test_firebird_reader.py agent/tests/test_runner.py -q`
Expected: import failure for `run_once`.
- [ ] **Step 3: Implement SELECT-only reader and runner**
Reuse `connect_authorized_read_only`, catalog data methods, and existing fingerprint functions; do not create a raw SQL escape hatch. Verify the computed schema fingerprint equals the reference constant before selecting the 20 codes from `SAMPLE_PRODUCTS_V1.json`. Select only mapped fiscal scalar fields, never `IMG_PRODUTO` or `FOTO`. Map `current_ncm` from `NCM` when nonempty, otherwise `CLASSIFICACAO_FISCAL`. Send batch requests with the configured tenant, agent, and company context. Save exactly 20 comparison entries containing current fields, returned classification, proposed fields, provenance, reason codes, and divergences. Mark results received only after a valid response.
- [ ] **Step 4: Run focused tests and the real end-to-end CLI**
Run: `python -m pytest agent/tests/test_firebird_reader.py agent/tests/test_runner.py -q`
Expected: PASS.
Run: `python agent/scripts/run_agent.py --sample docs/SAMPLE_PRODUCTS_V1.json --output docs/MISSION_02_PRODUCT_RESULTS.json`
Expected: 20 results, zero Firebird writes, and a persisted SQLite outbox state.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add agent docs/MISSION_02_PRODUCT_RESULTS.json && git commit -m "feat: process products through fiscal hub"`.
### Task 8: Document Dataset and Provider Investigation Results
**Files:**
- Create: `docs/FISCAL_DATASETS.md`
- Modify: `docs/MISSION_02_REPORT.md`
- Test: `central/tests/test_dataset_status_documentation.py`
**Interfaces:**
- Produces a factual provider status registry consumed only by the Mission 02 report.
- [ ] **Step 1: Write a report consistency test**
```python
def test_report_contains_required_cost_and_safety_counters():
report = Path("docs/MISSION_02_REPORT.md").read_text(encoding="utf-8")
assert "PAID_API_CALLS=0" in report
assert "PAID_API_COST=R$0.00" in report
assert "APPLICATION_DDL_DML_WRITES=0" in report
```
- [ ] **Step 2: Run test to verify failure**
Run: `python -m pytest central/tests/test_dataset_status_documentation.py -q`
Expected: FAIL because the Mission 02 report does not exist.
- [ ] **Step 3: Document exact source and integrations without invented claims**
Record NCM as `IMPLEMENTED` with actual imported source URL, download timestamp, SHA-256, update label, act, and row count. Research primary/free sources for CEST, CFOP, CST/CSOSN, cClassTrib, IBS/CBS, and the official RTC calculator; assign only `IMPLEMENTED`, `PARTIAL`, `DISCOVERED`, or `NOT_AVAILABLE` supported by evidence. Include RTC installation/API/license/hosting/input/output evidence, or record the concrete absence/blocker and `NOT_CONFIGURED` adapter. Report the real 20-product metrics, timing, cache hits, and every required safety and cost counter.
- [ ] **Step 4: Run documentation test**
Run: `python -m pytest central/tests/test_dataset_status_documentation.py -q`
Expected: PASS.
- [ ] **Step 5: Commit if allowed**
Run the Task 1 identity check. If configured: `git add docs central/tests/test_dataset_status_documentation.py && git commit -m "docs: report fiscal hub POC coverage"`.
### Task 9: Verify Complete POC and Operational Resilience
**Files:**
- Modify: `docs/MISSION_02_REPORT.md`
- Test: `central/tests/test_health.py`
- Test: `central/tests/test_api.py`
- Test: `agent/tests/test_runner.py`
**Interfaces:**
- Consumes the complete Hub, importer, agent, report, and Docker Compose configuration.
- [ ] **Step 1: Add failure-mode regression tests**
```python
def test_health_is_503_when_postgres_is_down():
client = TestClient(create_app(database_probe=lambda: False))
assert client.get("/health").status_code == 503
def test_second_agent_run_skips_unchanged_products_and_counts_cache(agent):
first = agent.run_once()
second = agent.run_once()
assert first.total == 20
assert second.api_requests == 0
```
- [ ] **Step 2: Run these tests to verify their expected initial behavior**
Run: `python -m pytest central/tests/test_health.py agent/tests/test_runner.py -q`
Expected: PASS after Tasks 1-7; otherwise fix the implementation before continuing.
- [ ] **Step 3: Execute Compose and live proof**
Run: `docker compose up -d --build`
Run: `docker compose ps`
Expected: `postgres` and `fiscal-hub` report healthy.
Run: `docker compose exec fiscal-hub python -m central.scripts.import_ncm`
Run: `python agent/scripts/run_agent.py --sample docs/SAMPLE_PRODUCTS_V1.json --output docs/MISSION_02_PRODUCT_RESULTS.json`
Run the same agent command again and record API call count, cache behavior, total timing, per-product timing, and cache-hit timing in the report.
- [ ] **Step 4: Run full verification**
Run: `python -m pytest tests central/tests agent/tests -q`
Expected: all tests pass.
Run: `python -m json.tool docs/MISSION_02_PRODUCT_RESULTS.json > $null`
Expected: valid JSON with 20 entries.
Run: `git diff --check`
Expected: no whitespace errors; line-ending warnings may be reported without diff failures.
- [ ] **Step 5: Record final state without forcing Git identity**
Run: `git status --short`
Run: `git log --oneline -10`
Add branch, status, commit state if any, health evidence, test output, performance, and real gaps to `docs/MISSION_02_REPORT.md`. If `git config user.name` is unavailable, report `GIT_IDENTITY_NOT_CONFIGURED` and do not configure it.
@@ -0,0 +1,57 @@
# Firebird Installation Inventory
## Scope and Safety
- Inventory performed read-only on 2026-09-16.
- `D:\PROJETOS\GTISISTEMAS\` outside `Projeto` was treated as a read-only external system.
- No source code, ERP files, services, processes, configuration, database objects, or permissions were modified.
- `ERP_FILES_MODIFIED = 0`
- No target database was accessed during inventory. `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` was not opened, connected to, queried, or inspected through a database tool.
- Another database file was visible under the external ERP tree, but it was also not opened.
- Credentials, passwords, connection values, and other secret values are omitted as `<redacted>`.
## Firebird Installation
- Installation root: `C:\Program Files\Firebird\Firebird_2_5\`.
- Registry instance `DefaultInstance` points to that installation root.
- Server family/version: Firebird 2.5, build `2.5.9.27139` (`WI-V2.5.9.27139`).
- Present server/client binaries include `fbserver.exe`, `fbguard.exe`, `fbclient.dll`, `isql.exe`, `gbak.exe`, `gstat.exe`, `gfix.exe`, `fbsvcmgr.exe`, `fbtracemgr.exe`, and `instsvc.exe`; these report build `2.5.9.27139` where version metadata is available.
- Firebird ICU DLLs report version `3.0.0.0`.
- `security2.fdb` exists in the installation root; it was not opened.
## Services and Process
- `FirebirdGuardianDefaultInstance`: running, automatic start, `LocalSystem`; executable is `fbguard.exe -s DefaultInstance`.
- `FirebirdServerDefaultInstance`: running, manual start, `LocalSystem`; executable is `fbserver.exe -s DefaultInstance`.
- `fbserver.exe` is running and responsive; process ID was observed but is not recorded as a stable inventory fact.
- Firebird server is listening on `0.0.0.0:3050`.
- No service was started, stopped, restarted, or reconfigured.
## Configuration
- `firebird.conf` exists and contains an active `RemoteAuxPort = 3051` setting.
- `aliases.conf` exists; no active alias facts were recorded.
- `databases.conf` was not present.
- Configuration values that could identify credentials, database locations, or endpoints are intentionally `<redacted>` or omitted.
## ERP Artifacts
- ERP executables observed include `GtiVendas_NFCe.exe` version `3.5.5.475`, `SGV_5.0.6.124.exe` version `5.0.6.124`, `configpdv.exe` version `1.00`, `Envio NF-e.exe` version `1.0.1.1`, and `GtiServer_v2.0.1.18.exe` version `2.0.1.18`.
- ERP Firebird client libraries observed include `fbclient.dll` version `2.5.9.27139` and `GDS32.DLL` version `2.5.8.27089`.
- Additional ERP DLLs and support libraries were inventoried; missing or blank embedded version metadata was recorded as unavailable rather than inferred.
- ERP configuration/document file types found include `.ini`, `.xml`, `.cfg`, `.conf`, and `.txt`, including `com.ini`, `Config_GtiServer.ini`, `ConfigBoleto.ini`, and `FTPConfig.ini`.
- Configuration and document contents were not reproduced in this report. Secret-bearing and database-related values are `<redacted>`.
## Verification and Git
- Exact command: `python -m pytest tests -q`
- Reported result: `58 passed in 0.15s`.
- Git identity check: `GIT_IDENTITY_NOT_CONFIGURED`.
- No commit was created and Git configuration was not changed.
## Unavailable Checks and Concerns
- The process API did not expose a usable executable path or command line for the running `fbserver.exe`; service configuration was used for the executable path.
- `isql-fb` was not available as a distinct command; the installed Firebird 2.5 `isql.exe` was present.
- No database metadata, schema, connection test, or database-level configuration was collected because database access was explicitly prohibited for this task.
- The running Firebird 2.5 service and the presence of `GDS32.DLL` version `2.5.8.27089` should be considered when planning later read-only discovery work.
@@ -0,0 +1,28 @@
# Task 3 Report
Status: COMPLETE
Runtime evidence/status:
- Pre-import dependency/client gate: `UNKNOWN` unless provenance, client
version, client architecture, and Python/client architecture match are
proven before importing `fdb`.
- Runtime compatibility fixture: `VALID` for `fdb 2.0.4`, client `2.5.9.27139`,
matching x64 architecture, and the exact read-only TPB.
- `validate_firebird_runtime(fdb_module=None)`: `UNKNOWN` with
`compatibility_unproven`; read-only API proof is mandatory.
- Success output exposes only compatibility categories/statuses and excludes
client paths, database identities, and internal provenance evidence.
- Blocked output exposes only status, blocker, and safety counters; provenance
path fields are not serialized.
Safety:
- No database, external ERP file, DLL, service, registry, or system
configuration was accessed or modified.
- `GIT_IDENTITY_NOT_CONFIGURED`.
Tests:
- `python -m pytest tests/test_dependency_compatibility.py -q`: 6 passed.
- `python -m pytest tests -q`: 255 passed, 1 skipped.
@@ -0,0 +1,85 @@
# Mission 00: Firebird Connectivity and Discovery
## Goal
Use the Firebird installation associated with the locally installed ERP as
read-only evidence to connect only to the authorized copy
`D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`, perform complete schema
discovery, and document confirmed findings without implementing fiscal writes,
image writes, services, or schedulers.
## Scope and boundaries
- Files outside `D:\PROJETOS\GTISISTEMAS\Projeto` are read-only.
- Only `TESTESOFICIAIS.FDB` may be opened or queried.
- Other database files may be listed but must not be connected to, copied, or
processed.
- No DML, DDL, procedure execution, generator changes, metadata changes,
permission changes, or service lifecycle changes are allowed.
- Credentials and secrets found in ERP configuration are evidence of location
only and must never enter source, reports, logs, or commits.
## Connection strategy
Use the installed Firebird server and its compatible Python driver/client as
the primary path. First inspect services, processes, executable and DLL
versions, tools, and ERP configuration. Select the client only after its
version and architecture are evidenced. Use an embedded/local client only as a
documented fallback if the server path is unavailable and compatibility is
proven. Do not install an arbitrary Firebird version.
The connection layer will enforce the absolute authorized database path and
require database-level or transaction/driver-level read-only enforcement. The
first database operation will be a minimal identity query. The sanitized
connection description will include driver, client, architecture, host/port
form, and target path, but no secrets.
## Discovery flow
1. Detect Firebird capabilities and execute only safe catalog SELECT queries.
2. Collect engine version, dialect, ODS, charset, page size, database identity,
relation counts, fields, keys, indexes, dependencies, generators, triggers,
procedures, parameters, and compatible monitoring metadata.
3. Identify product candidates using names, fields, keys, indexes, counts,
relationships, types, and minimal masked scalar samples.
4. Map product, stock, fiscal, company/branch, image/file, audit, and
incremental concepts. Mark every mapping `CONFIRMED`, `INFERRED`, or
`UNKNOWN` based on direct evidence.
5. Inspect trigger and routine source only as metadata text, without invoking
routines or collecting source/blob contents that are sensitive or
unnecessary.
6. Build fingerprint V1 from engine context, product structure, fiscal fields,
relationships, image structure, and critical triggers.
## Safety and sanitization
The existing read-only SQL boundary remains mandatory. Catalog capability
failures are recorded as unavailable or query errors rather than treated as
negative findings. Samples are small, scalar, masked, and purpose-limited.
Raw BLOBs, credentials, customer data, and unnecessary commercial data are
excluded. A path guard prevents accidental access to the ERP's original
database even if configuration reveals it.
## Project changes and verification
Changes are limited to the authorized project directory: path guard/driver
integration, discovery tests and fixtures, fingerprint support, and the three
current-state documents plus sanitized mission evidence. The `.FDB` remains
ignored by Git. Verification includes focused safety/parser tests, full
`python -m pytest tests -q`, `git diff --check`, sensitive-data review, and
Git status/identity inspection. A single mission commit is created only when
Git identity is already configured; otherwise report
`GIT_IDENTITY_NOT_CONFIGURED`.
## Outcomes
`MISSION_00_COMPLETE` requires real connection, confirmed engine details,
evidence-backed product and fiscal mapping, image/incremental/trigger review,
fingerprint V1, green tests, zero application writes, and no external file
modification. Any unresolved connectivity or compatibility blocker produces
`MISSION_00_BLOCKED` with the evidence and remaining gaps documented.
## Explicit non-goals
Do not implement fiscal rules, APIs, image storage, product UPDATEs, Windows
services, production schedulers, migrations, triggers, or database changes.
@@ -0,0 +1,95 @@
# Mission 00D: Structured Firebird Read-Only Boundary
## Goal
Remove arbitrary SQL from the public safety boundary, preserve legitimate
Firebird metadata expressions internally, then connect read-only only to
`D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` and complete evidence-backed
discovery when credentials and compatibility permit.
## Root cause
The previous `ReadOnlyConnection.execute(sql)` accepted consumer-provided SQL
and used a blacklist/allowlist parser as its execution authority. Quoted
identifiers could hide callable names from the textual scan, allowing potential
procedure/UDF invocation. The same incomplete callable allowlist rejected
legitimate Firebird expressions such as `CAST` and aggregate functions. This
is a boundary-design failure, not a missing regex case.
## Architecture
- Remove `execute(sql)` and equivalent raw execution methods from the public
adapter consumed by collectors and mapping code.
- Keep driver execution in a private implementation that accepts only internal
query identifiers and internally owned static SQL templates.
- Expose structured catalog operations for database info, relations, columns,
primary/foreign keys, indexes, triggers, procedures, generators,
dependencies, and monitoring metadata.
- Expose relation sampling only after relation and column identities have been
returned by the catalog. Validate those identities structurally, quote them
as Firebird identifiers, and cap the integer row limit.
- Keep transaction-level Firebird read-only enforcement and fail closed if it
cannot be proven. Do not expose commit, rollback, DDL, DML, procedure
execution, or generator modification.
- `collect_metadata` must require the approved structured reader at runtime,
not merely by type annotation.
## Query ownership
All metadata SQL belongs to the implementation and is selected by fixed query
keys. Internal queries may use `CAST`, `COUNT`, `SUM`, `MIN`, `MAX`,
`COALESCE`, joins, grouping, and other required Firebird syntax without going
through the old arbitrary-SQL parser. Procedures and UDFs are queried only as
metadata and never invoked.
The old parser may remain as an offline diagnostic/test utility, but it has no
authority to decide whether runtime SQL is executable. If retained, its
documentation must clearly state that quoted identifiers and Firebird grammar
are not security guarantees.
## Sampling
Sampling accepts only a relation identity previously returned by catalog
metadata, a column set previously returned for that relation, and a validated
small integer limit. It rejects expressions, clauses, procedures, UDFs,
fragments, arbitrary `WHERE`/`ORDER BY`, unknown relations/columns, malformed
quoted identifiers, and alternate database paths. It returns sanitized scalar
evidence only; BLOB content, credentials, customer data, and unnecessary
commercial values are excluded.
## Verification gates
Before any database access, tests must prove that arbitrary SQL cannot reach
the driver, quoted identifiers cannot reintroduce callable execution,
procedures/UDFs cannot be invoked, legitimate internal Firebird expressions are
supported, and arbitrary adapters are rejected. Only after all focused tests
and the full suite pass may the installed Firebird 2.5.9 client be used.
The connection then targets only the authorized `.FDB`, performs a minimal
identity query first, and uses a read-only transaction. If credentials,
compatibility, or identity verification fail, the result remains
`MISSION_00_BLOCKED` and no discovery query continues.
## Discovery and documentation
After a successful connection, collect capability-aware catalog metadata,
identify the product relation using structural and masked sample evidence,
correlate the ERP UI fields to database fields with `CONFIRMED`, `INFERRED`,
or `UNKNOWN` classifications, inspect fiscal/image/incremental structures,
and analyze trigger/procedure metadata without execution. Generate a real
schema fingerprint only from sanitized evidence. Update the current-state
documents and final report to describe the actual adapter, parser role,
runtime protections, transaction behavior, connection facts, and remaining
gaps.
## Safety invariants
- `AUTHORIZED_DATABASE` is the sole file opened or queried.
- `OTHER_DATABASES_ACCESSED = 0`.
- `APPLICATION_DDL_DML_WRITES = 0`.
- `ERP_FILES_MODIFIED = 0`.
- Files outside `Projeto` are read-only.
- No credentials, raw BLOBs, full rows, or original ERP database paths enter
project documentation.
- No API fiscal, image API, service, scheduler, product UPDATE, or migration
is implemented.
@@ -0,0 +1,88 @@
# Mission 00E: Threat Model, FDB Provenance, and Real Discovery
## Goal
Formalize the trusted-process threat model, validate the actual `fdb` module
provenance and Firebird client compatibility, then run read-only discovery only
against `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`.
## Threat model
`TRUSTED_PROCESS = TRUE`: only versioned application code runs in the process;
there are no untrusted plugins, user-provided scripts, external `eval`/`exec`,
or externally selected dynamic imports. Deliberate introspection or monkey
patching by code that already has arbitrary Python execution is equivalent to
application compromise and is outside this component's boundary.
The boundary protects against implementation mistakes, accidental unauthorized
operations, arbitrary SQL exposed to normal callers, parameter-to-SQL injection,
routine execution, wrong database selection, unintended writes, and dependency
shadowing/import mistakes.
## API boundary
The public API exposes only structured catalog operations and catalog-issued
bounded sampling. It exposes no `execute(sql)`, `query(sql)`, `raw_execute`,
operation-registration mutator, procedure/UDF invocation, arbitrary clauses, or
alternate database path. The registry remains module-private, immutable, and
centrally constructed. Python internals are not treated as a cryptographic
sandbox.
The old SQL parser is offline-only diagnostic/test code. It has no authority to
approve runtime SQL. Internal Firebird query templates may use legitimate
`CAST`, aggregates, joins, and grouping because they are selected by fixed
operation keys owned by the project.
## Provenance gate
Before any database connection, inspect the actual `import fdb` module:
- `fdb.__file__` and `fdb.__package__`;
- module version and `importlib.metadata` distribution/version/location;
- interpreter, working directory, `sys.path`, and `PYTHONPATH`;
- local `fdb.py`, `fdb` directory, and unexpected `.pth` shadowing candidates.
The imported module must correspond to the expected installed distribution and
version, without ambiguous duplicate installations. The installed Firebird
client/server evidence must match the selected driver compatibility. Any
ambiguity blocks before import/use of the driver for database access.
## Connection and discovery
After the provenance and safety gates pass, connect only to the authorized FDB,
using the installed compatible client and a Firebird read-only transaction. Run
the minimal identity operation first and compare the engine-reported identity
to the authorized target. Then collect catalog metadata through structured
operations, use only catalog-issued sample identities, and inspect product,
stock, fiscal, image, incremental, trigger, and procedure metadata without
executing routines or reading BLOB contents.
Correlate the ERP UI fields `NCM`, `CFOP`, `CST ICMS`, ICMS rate, `CEST`,
`CSOSN`, PIS/COFINS, IPI, origin, IAT, incidence, and IPPT to table, column,
type, domain/FK, dependencies, triggers, evidence, and confidence. Classify
each as `CONFIRMED`, `INFERRED`, or `UNKNOWN`. Generate a real fingerprint only
from sanitized complete evidence; otherwise use `SCHEMA_UNKNOWN`.
## Evidence and safety
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened.
- External ERP files, services, registry, DLLs, and configuration are read-only.
- `APPLICATION_DDL_DML_WRITES = 0`.
- `OTHER_DATABASES_ACCESSED = 0`.
- `ERP_FILES_MODIFIED = 0`.
- No credentials, original ERP database path, raw BLOB/image, full rows, or
unnecessary customer/commercial data enters source or reports.
- No fiscal API, image API, Windows service, scheduler, product UPDATE, or
migration is implemented.
## Verification and outcome
Tests cover the public API, registry immutability, SQL non-authority,
quoted-identifier and routine cases, legitimate internal expressions, path
allowlisting, provenance/shadowing, client compatibility, structured reader,
sampling, fingerprint, and discovery. The final report includes provenance,
client/engine facts, connection result, schema mappings, image/incremental/
trigger evidence, tests, Git state, and real gaps. Missing credentials,
ambiguous provenance, compatibility failure, identity mismatch, or incomplete
discovery results in `MISSION_00_BLOCKED` without a database connection or
fallback to another database.
@@ -0,0 +1,19 @@
# Fiscal Data Foundation V1
## Goal
Provide independently versioned, official fiscal datasets for the Hub without determining taxes for products.
## Dataset lifecycle
Each domain runs independently: official download, raw SHA-256, versioned parser, structural validation, `PENDING` import, semantic/volume validation, then `ACTIVE` promotion. A parser failure records `FAILED` and `failure_reason`; the previous `ACTIVE` version remains queryable. No secondary source can promote a canonical dataset.
Every version records `domain`, `source_url`, `downloaded_at`, `raw_sha256`, `parser_version`, `effective_from`, `effective_to`, `record_count`, `status`, `failure_reason`, and `raw_payload_reference` when retention is allowed.
## Domains
Existing NCM/Siscomex and CEST/CONFAZ remain intact. New isolated importers cover BA ST/MVA, CFOP, ICMS CST/CSOSN/origin, PIS/COFINS, TIPI/IPI, RTC, FCP, and monophase ICMS/ANP. Each has a normalized operational table only when queried by Hub consumers.
## Verification
Parser tests prove fail-closed behavior and independent lifecycle behavior. The final smoke reads only `ACTIVE` versions and produces the dataset matrix. This mission does not classify products or propose tax fields.
@@ -0,0 +1,151 @@
# GTI Fiscal Hub V1 - Mission 02 Design
## Scope
Deliver one end-to-end vertical slice: a FastAPI fiscal hub backed by
PostgreSQL, a versioned official NCM dataset, and a Windows-ready Python agent
that reads the authorized Firebird database and submits 20 real products to the
hub. The agent never writes to Firebird in this mission.
The Hub and agent are separate applications. The Hub owns fiscal knowledge;
the agent owns ERP communication. The Hub never accesses a customer Firebird
database and the agent never accesses central PostgreSQL.
## Components
`central/app` is a modular FastAPI monolith with these conceptual modules:
- API and bearer-token authentication.
- Dataset registry and NCM importer.
- Product classification: normalized description, GTIN, NCM, and CEST.
- Fiscal determination: operation and company context with NCM/CEST, CFOP,
CST/CSOSN, PIS/COFINS, IPI, and future IBS/CBS/cClassTrib inputs.
- Deterministic classification cache.
- Audit and provenance.
- Provider interfaces, including `RtcCalculatorProvider`.
Product classification and fiscal determination remain separate modules inside
the same process. This makes an existing valid NCM distinct from a fully proven
tax treatment.
`agent/src` contains a Firebird reader, schema fingerprint gate, product
fingerprinting, SQLite state/outbox, HTTP client, retry/backoff handling, and
comparison/report generation.
`docker-compose.yml` starts `postgres` and `fiscal-hub` only. The Firebird
database remains installed locally on Windows and outside Docker.
## Hub Persistence
PostgreSQL stores `datasets`, `ncm_entries`, `tenants`, `agents`,
`classification_requests`, `classification_results`, and
`classification_cache`. Dataset metadata includes source URL, provider,
download time, SHA-256, source validity/version data when present, record
count, lifecycle status, and import status.
Agent tokens are supplied by environment. PostgreSQL stores only token hashes,
each associated with one tenant and agent. Tenant identity is resolved from the
token and must agree with request metadata.
The agent uses a separate SQLite database. It stores source and result
fingerprints, fiscal dataset versions, timestamps, reason codes, retry data,
and an outbox with this lifecycle:
`DISCOVERED -> PENDING_API -> SENT -> RESULT_RECEIVED -> APPLIED`
`APPLIED` is reserved for a future write-capable mission and is not reached in
Mission 02.
## NCM Dataset Lifecycle
The only canonical source in this mission is the Receita Federal / Portal
Unico Siscomex JSON endpoint:
`https://portalunico.siscomex.gov.br/classif/api/publico/nomenclatura/download/json`
The importer downloads raw content, calculates SHA-256, parses JSON, and
inspects its real schema before mapping fields. It must not assume field names
before this inspection.
The lifecycle is:
`DOWNLOAD -> HASH -> PENDING -> IMPORT -> VALIDATION -> READY -> ATOMIC PROMOTION -> ACTIVE`
The previous active version becomes `SUPERSEDED` only inside the same short
promotion transaction that activates the ready version. NCM entries are scoped
to their dataset version, so clients observe either the complete prior dataset
or the complete new dataset, never a mixture. Failed imports remain
non-active, and prior versions remain queryable for rollback and audit.
Validation requires a successful HTTP response, parseable JSON, recognized
schema, plausible nonzero record volume, persisted hash and provenance, and a
fully successful import. A scheduled updater runs this same flow and promotes
only a validated version.
## Classification and Decisions
`POST /v1/validate/ncm` normalizes a NCM code and validates it against the
active official dataset. `/v1/classify` and `/v1/classify/batch` accept a
minimal product plus extensible company and operation context.
If an existing NCM is valid, the classification result is `NCM_VALIDATED`.
This is not permission to populate or change CST, CFOP, CSOSN, rates, CEST, or
any other fiscal field. Fields without official deterministic support return
`null` with explicit reason codes.
Products without a valid NCM use deterministic normalized-text candidate
ranking. Outcomes are `HIGH_CONFIDENCE_CANDIDATE`, `AMBIGUOUS`, or `NOT_FOUND`;
description similarity never creates a definitive NCM classification.
Each response carries `engine_version`, `dataset_versions` (including `ncm`),
and `decision_version`. These versions participate in central cache keys and
agent incremental decisions. A product with unchanged source fields is
reprocessed when relevant dataset or engine versions change.
`can_auto_apply` is always false in Mission 02 because no Firebird writes are
implemented. `NCM_VALIDATED` is a classification status only.
## Cache, Idempotency, and Audit
The Hub derives a request fingerprint from tenant, normalized product identity,
relevant context, engine version, and active dataset versions. A matching cache
entry returns an equivalent decision and does not create uncontrolled duplicate
audit records. A changed tenant, context, engine, or dataset version cannot
reuse a result.
The agent derives its product fingerprint from description, GTIN, NCM and
relevant current fiscal fields. It avoids an API call only if this fingerprint,
the prior decision versions, and result state all remain current. Interrupted
or failed sends stay in the outbox for retry. If the API is offline, the agent
does not affect the ERP and continues processing other queued items.
## Provider Status
NCM is implemented. CEST, CFOP, CST/CSOSN, cClassTrib, IBS/CBS, and the RTC
calculator are investigated and recorded with one of `IMPLEMENTED`, `PARTIAL`,
`DISCOVERED`, or `NOT_AVAILABLE`. The RTC adapter implements the provider
interface but returns `NOT_CONFIGURED` unless a concrete official integration
is verified to be simple and safe within this scope. It never fabricates tax
results.
## End-to-End Proof
The agent reads the 20 sample products from `C000025` using only SELECTs after
its schema fingerprint equals the recorded `REFERENCE_SCHEMA_V1` fingerprint.
It sends them to the authenticated batch endpoint, persists results locally,
and writes `docs/MISSION_02_PRODUCT_RESULTS.json` plus
`docs/MISSION_02_REPORT.md`.
The report includes coverage and timing metrics, API request and cache counts,
paid API counters fixed at zero, Firebird safety counters, dataset provenance,
integration statuses, and real gaps. No BLOB, image, Firebird schema change,
or Firebird DML/DDL is allowed.
## Verification
Focused tests cover NCM import and lifecycle, normalization/ranking,
authentication and tenant isolation, endpoints, cache/idempotency, agent
payloads, read-only Firebird access, fingerprint gating, incremental outbox,
and offline API behavior. The complete project test suite runs afterward.
Docker health checks cover PostgreSQL and the Hub; Hub health reports failed
when PostgreSQL is unavailable.