docs: plan autonomous fiscal completion v1
This commit is contained in:
gti
@@ -0,0 +1,182 @@
|
|||||||
|
# Autonomous Fiscal Completion V1 Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Reduce deterministic fiscal residuals across the five allowlisted databases and produce a measured autonomy report without AI calls or human-answer inference.
|
||||||
|
|
||||||
|
**Architecture:** Add pure, data-driven requiredness and completeness modules first. Feed them sanitized read-only ERP evidence and conservative provenance-aware aggregation. Keep dry-run/write planning separate from guarded execution, and add provider/cache/budget primitives without enabling any external provider.
|
||||||
|
|
||||||
|
**Tech Stack:** Python 3.12, pytest, Firebird read-only discovery through existing `firebird_reader`, existing guarded executor, JSON/Markdown artifacts, SQLAlchemy/PostgreSQL models only for central AI ledger.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` throughout.
|
||||||
|
- No Z.ai, OpenAI, or any AI/provider network call.
|
||||||
|
- Human COMPANY/GROUP/PRODUCT decisions remain unanswered and unpersisted.
|
||||||
|
- `NCM_CONFLICT` rows and the 73 historical `INSUFFICIENT_EVIDENCE` conflicts remain blocked.
|
||||||
|
- Firebird access is read-only unless an exact allowlisted path passes every existing guard and the deterministic dry-run proves a write.
|
||||||
|
- Never infer fiscal origin, ST withholding, regime facts, or tenant policy from weak/default evidence.
|
||||||
|
- `RETRIEVAL_V2` remains active; retrieval V3 remains unpromoted.
|
||||||
|
- Every new artifact reports provenance, unknowns, counts, and zero-call/write measurements.
|
||||||
|
|
||||||
|
## File Map
|
||||||
|
|
||||||
|
- Create `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json`: field/context requiredness rules.
|
||||||
|
- Create `central/app/fiscal_completeness.py`: pure field/product evaluation API.
|
||||||
|
- Create `central/tests/test_fiscal_completeness.py`: matrix and evaluator tests.
|
||||||
|
- Create `docs/ERP_INBOUND_FISCAL_EVIDENCE_MAP_V1.json`: discovered tables, relations, evidence capabilities, and counts.
|
||||||
|
- Create `scripts/discover_inbound_fiscal_evidence.py`: read-only schema/evidence discovery with sanitized output.
|
||||||
|
- Create `central/app/evidence_aggregation.py`: supplier/document observations and conservative consensus metrics.
|
||||||
|
- Create `central/tests/test_evidence_aggregation.py`: conflict and threshold tests.
|
||||||
|
- Create `docs/ORIGIN_EVIDENCE_RESOLUTION_V1.json`: origin statuses and provenance metrics.
|
||||||
|
- Create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json`: per-database completeness/autonomy metrics.
|
||||||
|
- Create `docs/LOCAL_PRODUCT_HEALTH_V2.json`: updated health summary.
|
||||||
|
- Create `docs/EXTERNAL_PRODUCT_DATA_PROVIDER_EVALUATION_V1.md`: researched provider comparison without purchase/query.
|
||||||
|
- Create `central/app/product_data_provider.py`: provider protocol and cache normalization types.
|
||||||
|
- Create `central/app/ai/budget_ledger.py`: persistent budget decision interface and atomic pre-call guard.
|
||||||
|
- Create `central/tests/test_product_data_provider.py` and `central/tests/test_ai_budget_ledger.py`.
|
||||||
|
- Modify `central/app/models.py` and migrations only if the existing migration/test pattern supports the ledger/cache tables.
|
||||||
|
- Modify `docs/AI_PROJECT_STATE.md` and create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.md` only after measured outputs exist.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Requiredness Matrix Contract
|
||||||
|
|
||||||
|
**Files:** Create `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json`; test in `central/tests/test_fiscal_completeness.py`.
|
||||||
|
|
||||||
|
**Interfaces:** Matrix JSON has `artifact`, `version`, `fields`, `contexts`, and each field rule has `classification`, `when`, `validity`, `source`, and `human_decision_required`. The five classifications are exact strings: `REQUIRED`, `CONDITIONAL`, `OPTIONAL_REFERENCE`, `NOT_APPLICABLE`, `ERP_NOT_USED`.
|
||||||
|
|
||||||
|
- [ ] Write tests that load the matrix, require all 23 contract fields exactly once, reject unknown classifications, and verify `NOT_APPLICABLE` does not imply missing data.
|
||||||
|
- [ ] Run `python -m pytest central/tests/test_fiscal_completeness.py -q`; expect failure because the matrix is absent.
|
||||||
|
- [ ] Create the matrix from `FISCAL_FIELD_CONTRACT_V1.json`, `ERP_OPERATION_PROFILES_V1.json`, and known company context keys. Mark unresolved PIS/COFINS and ICMS operator facts conditional, IPI reference-only for non-contributors, service/input-only fields as conditional, and unsupported ERP fields as `ERP_NOT_USED`.
|
||||||
|
- [ ] Run the focused tests and confirm they pass.
|
||||||
|
|
||||||
|
### Task 2: Completeness Evaluator
|
||||||
|
|
||||||
|
**Files:** Create `central/app/fiscal_completeness.py`; modify `central/app/field_contract.py` only if a shared field-name export is needed; test `central/tests/test_fiscal_completeness.py`.
|
||||||
|
|
||||||
|
**Interfaces:** Implement `evaluate_field(field, value, context, matrix) -> dict` and `evaluate_product(product, context, matrix) -> dict`. Field statuses are `REQUIRED_AND_VALID`, `REQUIRED_MISSING`, `CONDITIONAL_AND_VALID`, `CONDITIONAL_NOT_TRIGGERED`, `OPTIONAL_VALID`, `OPTIONAL_EMPTY`, `NOT_APPLICABLE`, `ERP_NOT_SUPPORTED`; product output includes `FISCALLY_COMPLETE` and `fields`.
|
||||||
|
|
||||||
|
- [ ] Add tests for required valid/missing, conditional triggered/untriggered, optional empty, not applicable empty, ERP unsupported, and invalid values.
|
||||||
|
- [ ] Run focused tests to confirm the new API fails before implementation.
|
||||||
|
- [ ] Implement pure evaluation with no database/network dependency; use existing contract validators where available and preserve explicit `None`/empty values.
|
||||||
|
- [ ] Add tests proving a blank `NOT_APPLICABLE` field does not make a product incomplete and a human-required company fact remains missing.
|
||||||
|
- [ ] Run focused tests and then `python -m pytest central/tests/test_field_contract.py central/tests/test_fiscal_completeness.py -q`.
|
||||||
|
|
||||||
|
### Task 3: Read-Only Inbound Discovery
|
||||||
|
|
||||||
|
**Files:** Create `scripts/discover_inbound_fiscal_evidence.py`, `docs/ERP_INBOUND_FISCAL_EVIDENCE_MAP_V1.json`; tests in `tests/test_inbound_evidence_discovery.py`.
|
||||||
|
|
||||||
|
**Interfaces:** `discover_database(path, connection_factory) -> dict` must use read-only isolation and return sanitized table/column/relationship/evidence metadata. `classify_evidence_source(table, columns) -> set[str]` returns categories such as `HEADER`, `ITEM`, `SUPPLIER`, `KEY`, `XML`, `FISCAL_ITEM`, and `UNKNOWN`.
|
||||||
|
|
||||||
|
- [ ] Write fake-cursor tests proving table names are discovered from system metadata, XML/BLOB columns are detected by type/name, and no DML/DDL SQL is issued.
|
||||||
|
- [ ] Run the focused tests and confirm failure before implementation.
|
||||||
|
- [ ] Implement metadata discovery using the existing Firebird reader/connection conventions and parameter-free system SELECTs only; do not assume NF-e table names.
|
||||||
|
- [ ] Run discovery against the five explicit paths in read-only mode, record unavailable databases/evidence as `UNAVAILABLE`, and sanitize credentials, CNPJ, XML payloads, and product descriptions from the artifact.
|
||||||
|
- [ ] Validate the JSON artifact contains database/path policy, discovered tables, candidate relationships, XML availability, and row counts.
|
||||||
|
|
||||||
|
### Task 4: Inbound Observation Extraction
|
||||||
|
|
||||||
|
**Files:** Modify `scripts/discover_inbound_fiscal_evidence.py`; create `central/app/evidence_aggregation.py`, `central/tests/test_evidence_aggregation.py`; create `docs/INBOUND_SUPPLIER_OBSERVATIONS_V1.json`.
|
||||||
|
|
||||||
|
**Interfaces:** `Observation` contains product identity, supplier/document identity hash, observed fiscal values, UF/date, and provenance. `aggregate_observations(observations) -> dict` returns `observation_count`, `supplier_count`, `agreement_rate`, `latest_observation`, and `conflict_count` per product/attribute.
|
||||||
|
|
||||||
|
- [ ] Add tests for unanimous multi-supplier evidence, split evidence, latest-date ordering, missing values, and supplier/document de-duplication.
|
||||||
|
- [ ] Run focused tests to verify failure before implementation.
|
||||||
|
- [ ] Implement conservative aggregation: agreement is strong only when at least two observations from at least two suppliers agree and no conflicting observation exists; never use simple majority as truth.
|
||||||
|
- [ ] Extract only available inbound facts; classify every observation as `SUPPLIER_OBSERVATION` and retain source table/document provenance.
|
||||||
|
- [ ] Produce counts for inbound items analyzed, XML documents available, and evidence-bearing products.
|
||||||
|
|
||||||
|
### Task 5: Deterministic Origin Resolution
|
||||||
|
|
||||||
|
**Files:** Create `central/app/origin_resolution.py`, `central/tests/test_origin_resolution.py`, `docs/ORIGIN_EVIDENCE_RESOLUTION_V1.json`.
|
||||||
|
|
||||||
|
**Interfaces:** `resolve_origin(product_id, observations, kb_facts) -> dict` returns `status` in `ORIGIN_CONFIRMED`, `ORIGIN_CONFLICT`, `ORIGIN_NEEDS_EVIDENCE`, plus value, metrics, and provenance.
|
||||||
|
|
||||||
|
- [ ] Test that explicit consistent inbound origin can confirm, disagreement creates conflict, one weak observation remains pending, and CST digits never determine origin.
|
||||||
|
- [ ] Implement only the conservative threshold from Task 4; do not globalize a lot/acquisition-specific fact without compatible identity evidence.
|
||||||
|
- [ ] Reconcile against the existing 20,995-style pending origin counts without changing company decision artifacts or current conflicts.
|
||||||
|
- [ ] Record before/resolved/conflict/after counts and exact reasons.
|
||||||
|
|
||||||
|
### Task 6: Product Role, ST/CEST, and Dynamic CFOP
|
||||||
|
|
||||||
|
**Files:** Create `central/app/deterministic_resolution.py`, `central/tests/test_deterministic_resolution.py`; update `docs/FISCAL_REQUIREDNESS_MATRIX_V1.json` and `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json` through generated outputs.
|
||||||
|
|
||||||
|
**Interfaces:** `resolve_product_role(signals)`, `resolve_st_status(signals)`, and `resolve_entry_scope(observations, company_uf)` return a value only with strong multi-signal evidence; otherwise return a typed pending result with reasons. Entry scope accepts only `INTRA`, `INTER`, `MIXED_OR_DYNAMIC`.
|
||||||
|
|
||||||
|
- [ ] Test conflicting role signals, recipe/production evidence, ST subject versus withheld, CEST corroboration, and mixed supplier UFs.
|
||||||
|
- [ ] Implement evidence stacks using group, recipe, stock, supplier, purchases, sales, production, flags, and history; no single weak flag may resolve role.
|
||||||
|
- [ ] Keep `ST_WITHHELD` pending when purchase evidence does not prove retention; do not copy company-specific CFOP/CST/CSOSN across tenants.
|
||||||
|
- [ ] Determine CFOP entry behavior as operational default/last operation/preferred default only if ERP evidence proves the behavior; otherwise report dynamic scope.
|
||||||
|
- [ ] Re-run deterministic dry-run planning for all five databases and classify each candidate without writing.
|
||||||
|
|
||||||
|
### Task 7: Guarded Deterministic Write Reprocessing
|
||||||
|
|
||||||
|
**Files:** Modify only existing dry-run/write orchestration files required by tests; add `tests/test_autonomous_write_gate.py`; create `docs/AUTONOMOUS_WRITE_AUDIT_V1.json`.
|
||||||
|
|
||||||
|
**Interfaces:** `eligible_write(candidate, guards) -> bool` and `reprocess_allowlisted(databases, dry_run) -> dict` must expose `writes_attempted`, `writes_verified`, `second_pass_writes`, skipped reasons, and changed columns.
|
||||||
|
|
||||||
|
- [ ] Add tests proving non-allowlisted paths, missing provenance, human-required facts, conflicts, failed schema/type guards, rowcount mismatch, and post-write mismatch all fail closed.
|
||||||
|
- [ ] Run tests against mocked executor before changing orchestration.
|
||||||
|
- [ ] Wire only deterministic candidates that already satisfy the existing guarded executor contract; do not broaden field allowlists or AI flags.
|
||||||
|
- [ ] Execute first pass only after the dry-run artifact reports eligible changes; verify exact before/after and unchanged columns.
|
||||||
|
- [ ] Reprocess all five databases and require `SECOND_PASS_WRITES=0`; if any guard or verification fails, record pending and do not retry destructively.
|
||||||
|
|
||||||
|
### Task 8: Completeness and Autonomy Metrics
|
||||||
|
|
||||||
|
**Files:** Create `scripts/generate_autonomous_completion.py`, `central/tests/test_autonomy_metrics.py`, `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.json`, `docs/LOCAL_PRODUCT_HEALTH_V2.json`.
|
||||||
|
|
||||||
|
**Interfaces:** `build_database_metrics(database_snapshot, evaluator, evidence) -> dict` returns `TOTAL_PRODUCTS`, complete/incomplete counts, reason buckets, required instances, valid/missing counts, autonomous rate, KB reuse, and AI residual counts.
|
||||||
|
|
||||||
|
- [ ] Add fixture tests for all incompleteness buckets: `MISSING_COMPANY_FACT`, `MISSING_PRODUCT_FACT`, `MISSING_EXTERNAL_PRODUCT_DATA`, `AI_SEMANTIC_CANDIDATE`, `DATA_CONFLICT`, `ERP_SCHEMA_LIMITATION`.
|
||||||
|
- [ ] Implement metrics from evaluator outputs and measured artifacts, not estimates or prior report text.
|
||||||
|
- [ ] Generate five-bank metrics, `AI_READY_BEFORE=511`, measured `AI_READY_AFTER`, GTIN split, and autonomous/external/AI projected rates without calling AI.
|
||||||
|
- [ ] Add a new-install simulation that excludes tenant-specific answers and reports `MANUAL_CONFIGURATION_REQUIRED_ON_INSTALL`.
|
||||||
|
|
||||||
|
### Task 9: External Provider Adapter and Evaluation
|
||||||
|
|
||||||
|
**Files:** Create `central/app/product_data_provider.py`, `central/tests/test_product_data_provider.py`, `docs/EXTERNAL_PRODUCT_DATA_PROVIDER_EVALUATION_V1.md`.
|
||||||
|
|
||||||
|
**Interfaces:** Define `ProductDataProvider` protocol with `lookup_gtin(gtin) -> ProviderRecord | None`; `ProviderRecord` contains provider/version/retrieved_at/raw_hash/normalized_data/license metadata. Define `LocalBulkDatasetProvider` as an offline-only implementation; no network provider is enabled.
|
||||||
|
|
||||||
|
- [ ] Test normalization, raw hashing, provider versioning, cache-key construction, and provider-unavailable behavior.
|
||||||
|
- [ ] Implement the protocol and offline cache boundary without adding a paid dependency.
|
||||||
|
- [ ] Document GS1, official/free sources, commercial API/dataset options, terms, cache/redistribution constraints, and cost scenarios for 35k/70-client, 100k, and 500k products. Mark unknown prices as `NOT_VERIFIED` rather than inventing them.
|
||||||
|
- [ ] Set recommendation to authorization-required unless an existing licensed local source is proven.
|
||||||
|
|
||||||
|
### Task 10: Persistent AI Budget and Idempotency Controls
|
||||||
|
|
||||||
|
**Files:** Create `central/app/ai/budget_ledger.py`, `central/tests/test_ai_budget_ledger.py`, `central/tests/test_ai_idempotency.py`; modify `central/app/models.py` and add a migration only if the repository migration conventions support it.
|
||||||
|
|
||||||
|
**Interfaces:** `BudgetLimits`, `UsageEntry`, `BudgetLedger.check_and_reserve(request) -> BudgetDecision`, and `result_cache_key(product, candidates, datasets, prompt, model) -> str`.
|
||||||
|
|
||||||
|
- [ ] Test per-run, daily, monthly, request-count, and new-product limits; boundary equality is allowed, exceeding a limit returns `AI_BUDGET_EXHAUSTED` before provider invocation.
|
||||||
|
- [ ] Test atomic reservation semantics with two concurrent reservations against the same budget window.
|
||||||
|
- [ ] Implement ledger persistence using the existing central database abstraction; keep all credentials out of the ledger.
|
||||||
|
- [ ] Test identical product/candidate/dataset/prompt/model inputs produce a cache hit and zero second call; changed model or dataset version produces a new key.
|
||||||
|
- [ ] Run these tests entirely offline with fake providers and assert no network calls.
|
||||||
|
|
||||||
|
### Task 11: Final Artifacts and Project State
|
||||||
|
|
||||||
|
**Files:** Create `docs/AUTONOMOUS_FISCAL_COMPLETION_V1.md`; modify `docs/AI_PROJECT_STATE.md`.
|
||||||
|
|
||||||
|
- [ ] Generate the consolidated report from measured JSON artifacts, including per-bank autonomy, requiredness, inbound reduction, KB reuse, provider candidates/cost/licensing, AI residual, human residual, writes, second pass, tests, and blockers.
|
||||||
|
- [ ] Update project state with the actual mission status, preserving `AI_WRITE_ALLOWED=false`, `FIREBIRD_AI_WRITES=0`, retrieval V2, and all conflict counts.
|
||||||
|
- [ ] Add tests that load the final JSON and assert required keys and safety invariants.
|
||||||
|
- [ ] Run `python -m pytest tests -q` and `python -m pytest central/tests -q`.
|
||||||
|
- [ ] Run `git diff --check`, artifact invariant validation, and tracked-secret scan.
|
||||||
|
- [ ] Commit implementation in coherent milestone commits; do not tag or push.
|
||||||
|
|
||||||
|
## Final Verification
|
||||||
|
|
||||||
|
Run exactly:
|
||||||
|
|
||||||
|
```text
|
||||||
|
python -m pytest tests -q
|
||||||
|
python -m pytest central/tests -q
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
The final report must state measured values, not expected values. A failed guard,
|
||||||
|
unavailable inbound evidence, or unavailable external provider is a documented
|
||||||
|
residual, never an inferred fiscal answer.
|
||||||
Reference in New Issue
Block a user