diff --git a/docs/superpowers/plans/2026-09-22-autonomous-fiscal-completion-v1.md b/docs/superpowers/plans/2026-09-22-autonomous-fiscal-completion-v1.md new file mode 100644 index 0000000..e9b89d9 --- /dev/null +++ b/docs/superpowers/plans/2026-09-22-autonomous-fiscal-completion-v1.md @@ -0,0 +1,182 @@ +# Autonomous Fiscal Completion V1 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Reduce deterministic fiscal residuals across the five allowlisted databases and produce a measured autonomy report without AI calls or human-answer inference. + +**Architecture:** Add pure, data-driven requiredness and completeness modules first. Feed them sanitized read-only ERP evidence and conservative provenance-aware aggregation. Keep dry-run/write planning separate from guarded execution, and add provider/cache/budget primitives without enabling any external provider. + +**Tech Stack:** Python 3.12, pytest, Firebird read-only discovery through existing `firebird_reader`, existing guarded executor, JSON/Markdown artifacts, SQLAlchemy/PostgreSQL models only for central AI ledger. + +## Global Constraints + +- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` throughout. +- No Z.ai, OpenAI, or any AI/provider network call. +- Human COMPANY/GROUP/PRODUCT decisions remain unanswered and unpersisted. +- `NCM_CONFLICT` rows and the 73 historical `INSUFFICIENT_EVIDENCE` conflicts remain blocked. +- Firebird access is read-only unless an exact allowlisted path passes every existing guard and the deterministic dry-run proves a write. +- Never infer fiscal origin, ST withholding, regime facts, or tenant policy from weak/default evidence. +- `RETRIEVAL_V2` remains active; retrieval V3 remains unpromoted. +- Every new artifact reports provenance, unknowns, counts, and zero-call/write measurements. + +## File Map + +- Create `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json`: field/context requiredness rules. +- Create `central/app/fiscal_completeness.py`: pure field/product evaluation API. +- Create `central/tests/test_fiscal_completeness.py`: matrix and evaluator tests. +- Create `docs/ERP_INBOUND_FISCAL_EVIDENCE_MAP_V1.json`: discovered tables, relations, evidence capabilities, and counts. +- Create `scripts/discover_inbound_fiscal_evidence.py`: read-only schema/evidence discovery with sanitized output. +- Create `central/app/evidence_aggregation.py`: supplier/document observations and conservative consensus metrics. +- Create `central/tests/test_evidence_aggregation.py`: conflict and threshold tests. +- Create `docs/ORIGIN_EVIDENCE_RESOLUTION_V1.json`: origin statuses and provenance metrics. +- Create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json`: per-database completeness/autonomy metrics. +- Create `docs/LOCAL_PRODUCT_HEALTH_V2.json`: updated health summary. +- Create `docs/EXTERNAL_PRODUCT_DATA_PROVIDER_EVALUATION_V1.md`: researched provider comparison without purchase/query. +- Create `central/app/product_data_provider.py`: provider protocol and cache normalization types. +- Create `central/app/ai/budget_ledger.py`: persistent budget decision interface and atomic pre-call guard. +- Create `central/tests/test_product_data_provider.py` and `central/tests/test_ai_budget_ledger.py`. +- Modify `central/app/models.py` and migrations only if the existing migration/test pattern supports the ledger/cache tables. +- Modify `docs/AI_PROJECT_STATE.md` and create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.md` only after measured outputs exist. + +--- + +### Task 1: Requiredness Matrix Contract + +**Files:** Create `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json`; test in `central/tests/test_fiscal_completeness.py`. + +**Interfaces:** Matrix JSON has `artifact`, `version`, `fields`, `contexts`, and each field rule has `classification`, `when`, `validity`, `source`, and `human_decision_required`. The five classifications are exact strings: `REQUIRED`, `CONDITIONAL`, `OPTIONAL_REFERENCE`, `NOT_APPLICABLE`, `ERP_NOT_USED`. + +- [ ] Write tests that load the matrix, require all 23 contract fields exactly once, reject unknown classifications, and verify `NOT_APPLICABLE` does not imply missing data. +- [ ] Run `python -m pytest central/tests/test_fiscal_completeness.py -q`; expect failure because the matrix is absent. +- [ ] Create the matrix from `FISCAL_FIELD_CONTRACT_V1.json`, `ERP_OPERATION_PROFILES_V1.json`, and known company context keys. Mark unresolved PIS/COFINS and ICMS operator facts conditional, IPI reference-only for non-contributors, service/input-only fields as conditional, and unsupported ERP fields as `ERP_NOT_USED`. +- [ ] Run the focused tests and confirm they pass. + +### Task 2: Completeness Evaluator + +**Files:** Create `central/app/fiscal_completeness.py`; modify `central/app/field_contract.py` only if a shared field-name export is needed; test `central/tests/test_fiscal_completeness.py`. + +**Interfaces:** Implement `evaluate_field(field, value, context, matrix) -> dict` and `evaluate_product(product, context, matrix) -> dict`. Field statuses are `REQUIRED_AND_VALID`, `REQUIRED_MISSING`, `CONDITIONAL_AND_VALID`, `CONDITIONAL_NOT_TRIGGERED`, `OPTIONAL_VALID`, `OPTIONAL_EMPTY`, `NOT_APPLICABLE`, `ERP_NOT_SUPPORTED`; product output includes `FISCALLY_COMPLETE` and `fields`. + +- [ ] Add tests for required valid/missing, conditional triggered/untriggered, optional empty, not applicable empty, ERP unsupported, and invalid values. +- [ ] Run focused tests to confirm the new API fails before implementation. +- [ ] Implement pure evaluation with no database/network dependency; use existing contract validators where available and preserve explicit `None`/empty values. +- [ ] Add tests proving a blank `NOT_APPLICABLE` field does not make a product incomplete and a human-required company fact remains missing. +- [ ] Run focused tests and then `python -m pytest central/tests/test_field_contract.py central/tests/test_fiscal_completeness.py -q`. + +### Task 3: Read-Only Inbound Discovery + +**Files:** Create `scripts/discover_inbound_fiscal_evidence.py`, `docs/ERP_INBOUND_FISCAL_EVIDENCE_MAP_V1.json`; tests in `tests/test_inbound_evidence_discovery.py`. + +**Interfaces:** `discover_database(path, connection_factory) -> dict` must use read-only isolation and return sanitized table/column/relationship/evidence metadata. `classify_evidence_source(table, columns) -> set[str]` returns categories such as `HEADER`, `ITEM`, `SUPPLIER`, `KEY`, `XML`, `FISCAL_ITEM`, and `UNKNOWN`. + +- [ ] Write fake-cursor tests proving table names are discovered from system metadata, XML/BLOB columns are detected by type/name, and no DML/DDL SQL is issued. +- [ ] Run the focused tests and confirm failure before implementation. +- [ ] Implement metadata discovery using the existing Firebird reader/connection conventions and parameter-free system SELECTs only; do not assume NF-e table names. +- [ ] Run discovery against the five explicit paths in read-only mode, record unavailable databases/evidence as `UNAVAILABLE`, and sanitize credentials, CNPJ, XML payloads, and product descriptions from the artifact. +- [ ] Validate the JSON artifact contains database/path policy, discovered tables, candidate relationships, XML availability, and row counts. + +### Task 4: Inbound Observation Extraction + +**Files:** Modify `scripts/discover_inbound_fiscal_evidence.py`; create `central/app/evidence_aggregation.py`, `central/tests/test_evidence_aggregation.py`; create `docs/INBOUND_SUPPLIER_OBSERVATIONS_V1.json`. + +**Interfaces:** `Observation` contains product identity, supplier/document identity hash, observed fiscal values, UF/date, and provenance. `aggregate_observations(observations) -> dict` returns `observation_count`, `supplier_count`, `agreement_rate`, `latest_observation`, and `conflict_count` per product/attribute. + +- [ ] Add tests for unanimous multi-supplier evidence, split evidence, latest-date ordering, missing values, and supplier/document de-duplication. +- [ ] Run focused tests to verify failure before implementation. +- [ ] Implement conservative aggregation: agreement is strong only when at least two observations from at least two suppliers agree and no conflicting observation exists; never use simple majority as truth. +- [ ] Extract only available inbound facts; classify every observation as `SUPPLIER_OBSERVATION` and retain source table/document provenance. +- [ ] Produce counts for inbound items analyzed, XML documents available, and evidence-bearing products. + +### Task 5: Deterministic Origin Resolution + +**Files:** Create `central/app/origin_resolution.py`, `central/tests/test_origin_resolution.py`, `docs/ORIGIN_EVIDENCE_RESOLUTION_V1.json`. + +**Interfaces:** `resolve_origin(product_id, observations, kb_facts) -> dict` returns `status` in `ORIGIN_CONFIRMED`, `ORIGIN_CONFLICT`, `ORIGIN_NEEDS_EVIDENCE`, plus value, metrics, and provenance. + +- [ ] Test that explicit consistent inbound origin can confirm, disagreement creates conflict, one weak observation remains pending, and CST digits never determine origin. +- [ ] Implement only the conservative threshold from Task 4; do not globalize a lot/acquisition-specific fact without compatible identity evidence. +- [ ] Reconcile against the existing 20,995-style pending origin counts without changing company decision artifacts or current conflicts. +- [ ] Record before/resolved/conflict/after counts and exact reasons. + +### Task 6: Product Role, ST/CEST, and Dynamic CFOP + +**Files:** Create `central/app/deterministic_resolution.py`, `central/tests/test_deterministic_resolution.py`; update `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json` and `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json` through generated outputs. + +**Interfaces:** `resolve_product_role(signals)`, `resolve_st_status(signals)`, and `resolve_entry_scope(observations, company_uf)` return a value only with strong multi-signal evidence; otherwise return a typed pending result with reasons. Entry scope accepts only `INTRA`, `INTER`, `MIXED_OR_DYNAMIC`. + +- [ ] Test conflicting role signals, recipe/production evidence, ST subject versus withheld, CEST corroboration, and mixed supplier UFs. +- [ ] Implement evidence stacks using group, recipe, stock, supplier, purchases, sales, production, flags, and history; no single weak flag may resolve role. +- [ ] Keep `ST_WITHHELD` pending when purchase evidence does not prove retention; do not copy company-specific CFOP/CST/CSOSN across tenants. +- [ ] Determine CFOP entry behavior as operational default/last operation/preferred default only if ERP evidence proves the behavior; otherwise report dynamic scope. +- [ ] Re-run deterministic dry-run planning for all five databases and classify each candidate without writing. + +### Task 7: Guarded Deterministic Write Reprocessing + +**Files:** Modify only existing dry-run/write orchestration files required by tests; add `tests/test_autonomous_write_gate.py`; create `docs/AUTONOMOUS_WRITE_AUDIT_V1.json`. + +**Interfaces:** `eligible_write(candidate, guards) -> bool` and `reprocess_allowlisted(databases, dry_run) -> dict` must expose `writes_attempted`, `writes_verified`, `second_pass_writes`, skipped reasons, and changed columns. + +- [ ] Add tests proving non-allowlisted paths, missing provenance, human-required facts, conflicts, failed schema/type guards, rowcount mismatch, and post-write mismatch all fail closed. +- [ ] Run tests against mocked executor before changing orchestration. +- [ ] Wire only deterministic candidates that already satisfy the existing guarded executor contract; do not broaden field allowlists or AI flags. +- [ ] Execute first pass only after the dry-run artifact reports eligible changes; verify exact before/after and unchanged columns. +- [ ] Reprocess all five databases and require `SECOND_PASS_WRITES=0`; if any guard or verification fails, record pending and do not retry destructively. + +### Task 8: Completeness and Autonomy Metrics + +**Files:** Create `scripts/generate_autonomous_completion.py`, `central/tests/test_autonomy_metrics.py`, `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json`, `docs/LOCAL_PRODUCT_HEALTH_V2.json`. + +**Interfaces:** `build_database_metrics(database_snapshot, evaluator, evidence) -> dict` returns `TOTAL_PRODUCTS`, complete/incomplete counts, reason buckets, required instances, valid/missing counts, autonomous rate, KB reuse, and AI residual counts. + +- [ ] Add fixture tests for all incompleteness buckets: `MISSING_COMPANY_FACT`, `MISSING_PRODUCT_FACT`, `MISSING_EXTERNAL_PRODUCT_DATA`, `AI_SEMANTIC_CANDIDATE`, `DATA_CONFLICT`, `ERP_SCHEMA_LIMITATION`. +- [ ] Implement metrics from evaluator outputs and measured artifacts, not estimates or prior report text. +- [ ] Generate five-bank metrics, `AI_READY_BEFORE=511`, measured `AI_READY_AFTER`, GTIN split, and autonomous/external/AI projected rates without calling AI. +- [ ] Add a new-install simulation that excludes tenant-specific answers and reports `MANUAL_CONFIGURATION_REQUIRED_ON_INSTALL`. + +### Task 9: External Provider Adapter and Evaluation + +**Files:** Create `central/app/product_data_provider.py`, `central/tests/test_product_data_provider.py`, `docs/EXTERNAL_PRODUCT_DATA_PROVIDER_EVALUATION_V1.md`. + +**Interfaces:** Define `ProductDataProvider` protocol with `lookup_gtin(gtin) -> ProviderRecord | None`; `ProviderRecord` contains provider/version/retrieved_at/raw_hash/normalized_data/license metadata. Define `LocalBulkDatasetProvider` as an offline-only implementation; no network provider is enabled. + +- [ ] Test normalization, raw hashing, provider versioning, cache-key construction, and provider-unavailable behavior. +- [ ] Implement the protocol and offline cache boundary without adding a paid dependency. +- [ ] Document GS1, official/free sources, commercial API/dataset options, terms, cache/redistribution constraints, and cost scenarios for 35k/70-client, 100k, and 500k products. Mark unknown prices as `NOT_VERIFIED` rather than inventing them. +- [ ] Set recommendation to authorization-required unless an existing licensed local source is proven. + +### Task 10: Persistent AI Budget and Idempotency Controls + +**Files:** Create `central/app/ai/budget_ledger.py`, `central/tests/test_ai_budget_ledger.py`, `central/tests/test_ai_idempotency.py`; modify `central/app/models.py` and add a migration only if the repository migration conventions support it. + +**Interfaces:** `BudgetLimits`, `UsageEntry`, `BudgetLedger.check_and_reserve(request) -> BudgetDecision`, and `result_cache_key(product, candidates, datasets, prompt, model) -> str`. + +- [ ] Test per-run, daily, monthly, request-count, and new-product limits; boundary equality is allowed, exceeding a limit returns `AI_BUDGET_EXHAUSTED` before provider invocation. +- [ ] Test atomic reservation semantics with two concurrent reservations against the same budget window. +- [ ] Implement ledger persistence using the existing central database abstraction; keep all credentials out of the ledger. +- [ ] Test identical product/candidate/dataset/prompt/model inputs produce a cache hit and zero second call; changed model or dataset version produces a new key. +- [ ] Run these tests entirely offline with fake providers and assert no network calls. + +### Task 11: Final Artifacts and Project State + +**Files:** Create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.md`; modify `docs/AI_PROJECT_STATE.md`. + +- [ ] Generate the consolidated report from measured JSON artifacts, including per-bank autonomy, requiredness, inbound reduction, KB reuse, provider candidates/cost/licensing, AI residual, human residual, writes, second pass, tests, and blockers. +- [ ] Update project state with the actual mission status, preserving `AI_WRITE_ALLOWED=false`, `FIREBIRD_AI_WRITES=0`, retrieval V2, and all conflict counts. +- [ ] Add tests that load the final JSON and assert required keys and safety invariants. +- [ ] Run `python -m pytest tests -q` and `python -m pytest central/tests -q`. +- [ ] Run `git diff --check`, artifact invariant validation, and tracked-secret scan. +- [ ] Commit implementation in coherent milestone commits; do not tag or push. + +## Final Verification + +Run exactly: + +```text +python -m pytest tests -q +python -m pytest central/tests -q +git diff --check +``` + +The final report must state measured values, not expected values. A failed guard, +unavailable inbound evidence, or unavailable external provider is a documented +residual, never an inferred fiscal answer.