Files
GTI_GERENCIADOR_FISCAL/docs/AI_PROJECT_STATE.md
T

425 lines
32 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AI Project State
## Current status (real LLM evaluation V1 — 2026-09-21)
`BLOCKED_EXTERNAL_AI_CREDENTIAL` (implementation complete). The entire real-LLM
evaluation pipeline is built and tested offline: OpenAI-compatible provider
(`central/app/ai/llm_provider.py`, key only from Hub env, injectable transport),
strict per-item validator (candidate-list confinement, abstain semantics, 1:1
work_item_id, calibrated confidence in [0,1]), progressive batch probing (20/10/5),
post-AI validator (active-dataset membership), metrics + A/B/C error attribution +
confidence-bucket calibration (`central/app/ai/eval_metrics.py`), permanent result
cache (`ai_eval_cache`, migration 0014) with zero-call re-execution proof, and the
runner `central/scripts/run_real_llm_ncm_eval.py` (gold-first; silver/noisy only
after the gate: selection accuracy >= 0.95 AND high-confidence wrong rate <= 0.02;
artifacts REAL_LLM_NCM_EVALUATION_V1/_ERRORS_/_COST_). The runner refuses to call
anything without `AI_API_KEY` and exits BLOCKED_EXTERNAL_AI_CREDENTIAL with the exact
operator action. Zero LLM calls, zero Firebird writes, AI_WRITE_ALLOWED=false.
Operator action: set AI_API_KEY (optionally AI_BASE_URL/AI_MODEL) in the Hub .env.
## Semantic candidate retrieval V2 (2026-09-21)
`NCM_RETRIEVAL_V2_GATE_PASSED`. Deterministic hybrid retrieval (hierarchical official
corpus enriched with ancestor descriptions + KB commercial->NCM neighbor bridge +
KB-derived abbreviation lexicon, 644 entries; weights 0.3/0.7; leave-one-out discipline
after catching self-retrieval leakage) lifted candidate recall from 7.5%@50 (V1) to
HYBRID 92%@20 / 93.5%@50, GOLD 95.4%@20/@50, MRR 0.84 — meeting the LLM liberation gate
(GOLD@20 ≥ 0.80, @50 ≥ 0.90). Benchmark quality audit: GOLD 65 / SILVER 118 / NOISY 17.
GTIN audit: 8,559 valid, 251 cross-tenant matches, 90 GTIN-conflicting NCMs surfaced.
Reprocessing the 750 work items: 4 KB_EXACT + 16 KB_FINGERPRINT + 1 high-confidence
retrieval + 143 single-official-candidate (CEST/EX) resolved deterministically; 511
items (4,181 cells) still need LLM selection; projected cost $0.54 → $0.44. Zero LLM
calls, zero Firebird writes, AI_WRITE_ALLOWED remains false. Artifacts:
docs/NCM_RETRIEVAL_EVALUATION_V2.json, docs/NCM_RETRIEVAL_ARCHITECTURE_V2.md,
docs/GTIN_KB_AUDIT_V1.json, docs/AI_RESIDUAL_PLAN_V2.json.
## AI semantic layer V1 (2026-09-21)
`AI_SEMANTIC_LAYER_V1_IMPLEMENTED`. The deterministic gate was closed
(OFFICIAL_RULE_RESOLVABLE_REMAINING=0, ERP_DATA_RESOLVABLE_REMAINING=0) and the Hub now
owns a Product Knowledge Base plus a deduplicated AI work-item queue. entry_scope now
admits MIXED_OR_DYNAMIC (CFOP_ENTRADA becomes operation-dependent; no CFOP invented).
KB seeded read-only from all 5 databases: 30,801 global products (8,559 GTIN), 35,065
observations, 163 surfaced knowledge conflicts, idempotent load. 850 residual questions
deduped into 750 work items (20 answered by KB cache alone). Budget-guarded cost plan:
$0.54 estimated for 730 items (authorized, $0 spent — no AI credential exists here).
The 200-sample NCM benchmark proved the honest floor: candidate recall@10 = 5.5%
(ERP commercial wording vs official legal wording), so AI selection over closed
candidates is capped there — READY_FOR_AI_ASSISTED_WRITE_V2=false with candidate
generation as the named prerequisite. AI_WRITE_ALLOWED=false in V1: AI results stay
AI_CANDIDATE_VALIDATED, never Firebird writes. Artifacts:
docs/PRODUCT_KNOWLEDGE_BASE_V1.md, docs/AI_SEMANTIC_LAYER_V1.md,
docs/AI_EVALUATION_V1.json, docs/AI_COST_PLAN_V1.json, docs/AI_RESIDUAL_PLAN_V1.json.
`FISCAL_DATA_FOUNDATION_MOSTLY_COMPLETE`. Deploy/pilot work remains paused
per the current mission's scope. The operator manually placed the 6 official
files that were blocked by the Portal NF-e/SPED session gate into
`sources/raw/bootstrap_v1/`; all 6 were identified by content (never by
filename), hashed, parsed, validated, and promoted `ACTIVE` — no internet
access was attempted or needed this pass. **15 fiscal domains are now
`ACTIVE`**: NCM, TIPI_IPI, CEST, BA_ST_MVA, ORIGEM_MERCADORIA, CST_ICMS,
CSOSN, CST_PIS, CST_COFINS (from the prior pass), plus **CFOP, FCP, ANP,
NCM_UTRIB, RTC_BIOFUEL_INDEX, RTC_CCREDPRES** (new this pass) —
**42,593 fiscal records persisted total**. 11 domains remain
`MISSING_BOOTSTRAP_FILE` (ICMS monofásico, RTC cClassTrib, RTC alíquotas
CBS, and all 8 PIS/COFINS Tabela 4.3.10–4.3.17 files) — no file for them
exists locally; internet access was deliberately not attempted this
session. See `docs/SOURCE_REGISTRY_V2.md`. Read-only coverage over all 521
`TESTESOFICIAIS.FDB` products is in `docs/FISCAL_COVERAGE_FULL_V2.json`
(supersedes `_V1.json`, kept for history). Zero Firebird writes this
session (confirmed independently: `000026.IPI` still `3.25`, 109 non-null
`IPI` rows unchanged).
`PILOT_DEPLOYMENT_READY` (previous mission's state, unchanged this pass). The
Hub (central/) + Agent (agent/) system is
operational end-to-end via a real Job queue, deploy-ready via Docker
Compose, and the Agent is installable as a Windows Service. Formal Agent
registration exists (`central/scripts/register_agent.py`), each Agent has
its own token, and write authorization for a real client installation now
requires two explicit, independently-configured layers (path + schema
fingerprint) rather than the dev-only hardcoded path alone. See
`docs/OPERATIONS.md` for day-to-day commands and `docs/PILOT_DEPLOYMENT.md`
for standing up a new environment end to end.
Firebird discovery (Mission 00/01) remains read-only and unchanged; its
sanitized evidence lives in `docs/SAMPLE_PRODUCTS_V1.json` and
`docs/MISSION_01_REPORT.md`.
## Architecture
- **Hub** (`central/app`): FastAPI + PostgreSQL. Classifies NCM/CEST/TIPI/
BA_ST_MVA (`classification.py`), owns the Agent registry and Job queue
(`models.py`, `jobs.py`, `api.py`). Never touches Firebird, never stores
Firebird credentials.
- **Agent** (`agent/src/gti_fiscal_agent`, `agent/scripts`): reads Firebird
read-only, calls the Hub's `/v1/classify/batch`, runs the IPI auto-apply
decision (`ipi_auto_apply.py`) through the shared pipeline
(`pipeline.py`), and — only when `FIREBIRD_WRITES=1` and the database
path is exactly the authorized disposable test file — writes via the
guarded executor (`scripts/firebird_write_executor.py`). Long-running
entrypoint: `agent/scripts/run_agent_service.py` (heartbeat + Job poll
loop). One-shot CLI variant kept for manual use:
`agent/scripts/run_agent_auto_apply.py`.
- Local per-product audit: Agent's SQLite (`agent/agent_state.sqlite`,
table `ipi_write_log`). Central per-Job audit: Postgres `jobs.result_summary`.
## Firebird write policy
`TESTESOFICIAIS.FDB` is the only Firebird database ever authorized for
writes in development (see `CLAUDE.md`). For a real pilot installation, two
independent, explicitly-configured layers are both required before any
write connection opens — never inferred from a filename or `.FDB` extension:
`AUTHORIZED_WRITE_DATABASE_PATH` (must equal that installation's own
`FIREBIRD_DATABASE_PATH`) and `EXPECTED_SCHEMA_FINGERPRINT` (must equal the
live schema fingerprint that installation reports). Leaving either unset
falls back to the dev-only constant, so a real client database is refused
by default (`scripts/firebird_write_executor.py::assert_authorized_write_path`
/ `assert_schema_fingerprint_authorized`, wired into
`agent/scripts/run_agent_service.py::execute_job`). All other Firebird
databases (including `Banco/GTI.FDB`, found but never opened) remain
read-only. No real client database has been written to in any mission —
every write so far targeted `TESTESOFICIAIS.FDB`.
## Auto-apply scope
Only `C000025.IPI`, only when the Hub's TIPI match is `rate_kind=NUMERIC`
and unambiguous. Never CEST, EX_TIPI, CFOP, CST/CSOSN, PIS/COFINS, MVA, or
non-numeric/blank/legacy-NCM cases — those remain measurement-only
(`docs/DRY_RUN_PLAN_V4.json`, `docs/FULL_DATABASE_DRY_RUN_V1.json`).
## E2E proof (2026-09-18)
Real Job created via `POST /v1/jobs` (`REPROCESS_PRODUCTS`), claimed and
executed by `run_agent_service.py` (not the manual batch script):
manually-induced drift (`C000025.IPI` forced to NULL on product `000026`)
was detected and auto-restored to `3.25`, verified independently in
Firebird, `jobs.result_summary` on the Hub, and the Agent's local
`ipi_write_log`. A second run against the same now-correct product
produced `writes_attempted=0` (idempotent no-op). `REPROCESS_ALL` over all
521 products completed with `writes_attempted=0` (everything already
correct from the prior full-database batch). A job created with
`FIREBIRD_WRITES=0` against a genuinely eligible drifted product completed
with `writes_attempted=0` and the ERP value left untouched, with the
would-be write logged locally. The database was left in its correct state.
During this session Docker Desktop itself restarted mid-mission (external
event, not induced); the Hub came back healthy on its own once Docker
recovered, and all prior Job history was intact (Postgres volume
persisted) — an incidental real-world confirmation of "Hub temporarily
unavailable, Agent/data survive."
## Pilot readiness proof (2026-09-18)
Registered a real Agent via `central/scripts/register_agent.py` (found and
fixed a `DetachedInstanceError` bug in the process: the script read ORM
attributes after the session closed — fixed to capture plain values before
closing, with a regression test that mirrors the real CLI's usage pattern).
Ran the Agent under that fresh, individually-hashed token end to end:
startup validation, heartbeat (visible via `GET /v1/agents/me`, correctly
derives `ONLINE`), job creation → claim → `COMPLETED` with a real product,
restart with SQLite/central history intact, and a live path-guard test
(fake client path, `FIREBIRD_WRITES=1`, no explicit authorization) that
failed closed before any Firebird connection attempt and reported the job
`FAILED` with a sanitized error while the Agent kept running. Also
restarted the Hub and Postgres containers independently mid-session and
confirmed the Agent/Hub recovered and history persisted through both.
Also fixed a Docker Compose bug introduced while making the Hub
deploy-configurable: passing `DATABASE_URL` through from the shared root
`.env` leaked the host-side (`localhost`-pointing) value into the
container's network namespace, where `localhost` means the container
itself, breaking Postgres connectivity. Removed that passthrough; the
Hub's own `postgres:5432` default (from `central/app/config.py`) is used
instead.
## Fiscal data foundation (2026-09-18)
`docs/SOURCE_REGISTRY_V2.md` is now the current source registry (`V1` kept
for history). Summary:
- **Preserved, re-verified**: NCM (15,156 entries, hash-verified as
unchanged — no internet available to re-fetch, so this is "verified
against local record" not "re-fetched"), TIPI_IPI (11,103 entries,
hash-verified byte-identical against the local xlsx), CEST (1,051
entries, preserved — no internet to re-fetch, no local raw copy to
hash-check against).
- **Re-imported with updated provenance**: BA_ST_MVA — the local PDF's bytes
changed (new `raw_sha256`) but the parsed content is identical (303/303
records match exactly what was previously `ACTIVE`); promoted a new
dataset row pointing at the current file rather than leaving provenance
referencing a stale path.
- **New this session**: `fiscal_code_entries` table (migration `0007`)
holding ORIGEM_MERCADORIA (9), CST_ICMS (11), CSOSN (10), CST_PIS (33),
CST_COFINS (33) — extracted directly from the official
`moc7-anexo-i-leiaute-e-rv.pdf` (Receita Federal/ENCAT), one shared
`Dataset` row (`kind=NFE_MOC_TABLES`). Parser:
`central/app/datasets/nfe_moc_tables.py`. Nothing invented: entries built
from a combined official phrase (CST ICMS 40/41/50; CSOSN
102/103/300/400) keep that phrasing rather than being split further than
the source itself supports; the 6 CST_COFINS codes not separately
captured from the PDF's COFINS section (a page-boundary text-extraction
artifact, not a semantic gap) are filled from the CST_PIS wording after
confirming character-for-character identical wording for every code both
extractors did capture — each such row is labeled
`metadata_json.mirrored_from = "CST_PIS"`.
- **Second pass (bootstrap_v1 operator-supplied files, migrations `0008`/`0009`)**:
the operator placed 6 previously-blocked official files directly into
`sources/raw/bootstrap_v1/`. Each was identified by actual sheet
names/header rows (never by filename) and promoted `ACTIVE`:
**CFOP** (619 codes, Informe Técnico 2023.002, new dedicated
`cfop_entries` table with all 14 official columns including the RTC-era
`indExcIBSCBS`), **FCP** (27 UFs, `fcp_rates`), **ANP** (1031 `cProdANP`
codes — confirmed the official NF-e portal table genuinely has no
description column, not an omission — `anp_product_codes`),
**NCM_UTRIB** (10,516 rows, `ncm_utrib_entries`), **RTC_BIOFUEL_INDEX**
(23 rows, current + historical `pBioObrigatorio`, `rtc_biofuel_index_entries`),
**RTC_CCREDPRES** (13 codes, all 18 official LC 214/2025 fields preserved
— legal basis, CBS/IBS indicators, rates, vigências —
`rtc_cred_pres_entries`). Bug found mid-import: `cclass_nota_referenciada`
was `VARCHAR(16)` but some official rows carry free text there; widened
to `Text` in migration `0009` after a clean rollback (no orphan row, no
data loss). New read-only endpoints:
`GET /v1/reference/{cfop,ncm_utrib,anp,fcp}/{code}`.
- **Still `MISSING_BOOTSTRAP_FILE`** (not `FAILED` — simply no file
provided, internet deliberately not attempted this session): ICMS
monofásico (combustíveis), RTC cClassTrib, RTC alíquotas CBS, and all 8
PIS/COFINS Tabela 4.3.10–4.3.17 files.
- **Coverage matrix**: `docs/FISCAL_COVERAGE_FULL_V2.json` (supersedes
`_V1.json`), all 521 products in `TESTESOFICIAIS.FDB`, read-only (zero
Firebird writes). NCM 495 valid / 3 legacy / 23 invalid / 0 missing. IPI
107 numeric / 20 non-numeric / 297 source-blank / 71 ambiguous / 23
no-match. CEST: 462 matches, 0 confirmed (structural ambiguity,
unchanged), 15 candidates. BA_ST_MVA: 268 matches, 31 resolved, 237
candidates. **New**: CFOP match_found=498/521 (this product's own
`C000025.CFOP` value is an officially valid code); NCM_UTRIB
match_found=495/521 (tracks `ncm_valid` exactly, as expected). ANP/FCP/
RTC_CCREDPRES/RTC_BIOFUEL_INDEX are `DATA_AVAILABLE=true` but
`match_found=0` — stated explicitly as a data-shape fact (this grocery/
restaurant ERP has no ANP/fuel-product/UF join key on `C000025`), not a
gap or a bug. PIS/COFINS CST domain remains `DATA_AVAILABLE`,
`RULE_RESOLVED=0` for all products (same reason as before: no NCM-linked
determination table exists to acquire yet).
## Safety counters (cumulative across missions)
- `APPLICATION_DDL_DML_WRITES` against Firebird: only ever against
`TESTESOFICIAIS.FDB`, only `UPDATE C000025.IPI` / `UPDATE C000025.CEST`-class
statements via the guarded executor (CEST never actually written — no
`CEST_CONFIRMED` case existed with enough evidence).
- `OTHER_DATABASES_ACCESSED = 0` (including `Banco/GTI.FDB`, never opened).
- Postgres migrations: `0001`..`0009` applied (`0006` adds `jobs` table and
`agents` status/heartbeat columns; `0007` adds `fiscal_code_entries`;
`0008` adds `cfop_entries`/`fcp_rates`/`anp_product_codes`/
`ncm_utrib_entries`/`rtc_biofuel_index_entries`/`rtc_cred_pres_entries`;
`0009` widens `rtc_cred_pres_entries.cclass_nota_referenciada` to `Text`).
- This session: `FIREBIRD_WRITES=0` throughout — zero writes, zero DML/DDL
against any Firebird database (only read-only SELECTs to build the
coverage matrix); independently re-verified after the session (`000026.IPI`
still `3.25`, 109 non-null `IPI` rows unchanged).
- Total fiscal records persisted in Postgres across all domains: 42,593.
## Threat model (Firebird discovery boundary, unchanged since Mission 00)
The structured boundary protects normal callers against implementation mistakes,
accidental unauthorized operations, arbitrary SQL exposed to normal
callers, the wrong database, writes, procedures/UDFs, injection, and
dependency shadowing. `TRUSTED_PROCESS = TRUE`: arbitrary Python already
running in the same process, including deliberate introspection or monkey
patching, is outside scope. No artificial cryptographic controls are added.
## Known gaps / next steps
- No "list all agents" or "disable agent" HTTP endpoint yet — disabling an
agent is a direct `agents.enabled` update via `DATABASE_URL` access (see
`docs/PILOT_DEPLOYMENT.md`'s "Rollback"). Fine for pilot scale; a small
admin surface is a natural follow-up once there's more than a couple of
installations.
- The Windows Service wrapper (`agent/service/gti_agent_windows_service.py`)
was written and unit-verified (module imports cleanly, stop-flag mechanism
tested against `run_agent_service.run()`), but was **not** registered with
the Windows Service Control Manager on this machine — `sc.exe`/pywin32
`install` modifies system-wide service state, which this session treats as
outside what it does unprompted. An operator runs the documented
`install`/`start` commands themselves.
- No auto-update mechanism (deliberately out of scope this round) — updates
are a manual stop/replace/start, documented in `docs/PILOT_DEPLOYMENT.md`.
- CEST matching still cannot confirm any of the 5 known-ambiguous products
(structural NCM→multiple-CEST ambiguity in CONFAZ data, not a matcher bug
— see `docs/DRY_RUN_PLAN_V4.json`).
- `NT_STATUS_FIELD` (how a TIPI "NT" marker should map to `C000025.CST_IPI`)
remains unmapped — no reliable ERP evidence found for it.
- 11 fiscal domains are `MISSING_BOOTSTRAP_FILE` (not `FAILED` — no file was
ever provided for them): ICMS monofásico, RTC cClassTrib, RTC alíquotas
CBS, and all 8 PIS/COFINS Tabela 4.3.10–4.3.17 files. CFOP, FCP, ANP,
NCM×uTrib, RTC_CCREDPRES, RTC_BIOFUEL_INDEX — all previously `FAILED` —
are now `ACTIVE` (operator supplied the official files). See
`docs/SOURCE_REGISTRY_V2.md` for exact evidence and next steps.
- PIS/COFINS: only the CST code domain (Tabela 4.3.3/4.3.4-equivalent) is
`ACTIVE`. The NCM-linked determination tables (SPED Guia Prático
EFD-Contribuições Tabelas 4.3.10–4.3.17: alíquotas diferenciadas,
monofásicas, ST, zero, isenção, sem incidência, suspensão) were not
acquired — no file provided, internet deliberately not attempted.
- CST_ICMS/CSOSN/ORIGEM_MERCADORIA/CST_PIS/CST_COFINS are consultable
(`DatasetService.fiscal_codes(domain, code=None)`) but not yet wired into
`Classifier` — no per-product automatic determination exists for these
domains, by design (mission explicitly forbade inventing fiscal rules).
## Git State
No Git identity was configured and no Git configuration was modified this
session.
## Business scope (2026-09-19)
Clients: restaurants, markets, clothing/retail, workshops, commerce. No fuel
stations, no EFD-Contribuicoes. CORE_MVP still missing only RTC_CCLASSTRIB and
RTC_CBS_RATES (`MISSING_BOOTSTRAP_FILE`, awaiting operator files; see
SOURCE_REGISTRY_V2.md). ICMS_MONOPHASIC = OUT_OF_SCOPE_MVP; ANP and
RTC_BIOFUEL_INDEX = ACTIVE_REFERENCE_ONLY; PIS_COFINS_4_3_10..17 =
DEFERRED_BY_BUSINESS_SCOPE (not deploy blockers; existing support kept).
FISCAL_COVERAGE_FULL_V3 to be generated once the two core domains are ACTIVE.
## CORE MVP fiscal foundation COMPLETE (2026-09-19)
RTC_CCLASSTRIB (164 cClassTrib + 18 CST) and RTC_CBS_RATES (2 rows) are ACTIVE
(migration 0010, endpoints `/v1/reference/rtc_cclasstrib/{code}`,
`/v1/reference/rtc_cbs_rate/{year}`). All 14 CORE domains ACTIVE; nothing CORE
is missing. Total fiscal records 42,777. Coverage: `docs/FISCAL_COVERAGE_FULL_V3.json`
(V2 kept for history). cClassTrib/cCredPres have no ERP join key → DATA_AVAILABLE
only; no rule invented. Zero Firebird writes.
## Fiscal Decision Engine V1 (2026-09-19)
`central/app/fiscal_decision.py` (pure, `DECISION_ENGINE_VERSION=fiscal-decision-v1`, carried in every
provenance) + `scripts/run_fiscal_decision_v1.py` (read-only over the 521 products; `--apply` writes only
AUTO_APPLY_SAFE via the guarded executor). Artifacts: `docs/FISCAL_DECISION_MATRIX_V1.json` (per product/domain:
current/proposed/status/reason/confidence/provenance/write_eligible + ERP field map),
`docs/FISCAL_DECISION_COVERAGE_V1.json`, `docs/NEXT_FISCAL_AUTOMATION_PRIORITIES_V1.md`.
Company context read live: UF=BA, CRT=1 (C000004). ERP mappings: 11 confirmed, 10 MAPPING_UNCONFIRMED
(CFOP_ENTRADA, CST_ENTRADA, COD_*_PIS/COFINS, ALIQ_PIS/COFINS, CST_IPI, ORIGEM). Result: AUTO_APPLY_SAFE = 0
(IPI already correct on all 107 numeric rows; CEST never single-confirmed; a lone EX row per NCM is NOT
evidence for a product — needs description match). CFOP/CSOSN/CST/PIS/COFINS/FCP/RTC = OPERATION_CONTEXT_REQUIRED
or unresolved; CBS 0.009 kept as reference only, never applied universally. No Hub classify change, so no
classification_cache bump needed. Helper `scripts/audit_erp_fields.py` (read-only). New endpoint
`GET /v1/reference/fiscal_codes/{domain}`. Zero Firebird writes this mission.
## ERP Fiscal Context Discovery V1 (2026-09-19, read-only)
Artifacts: `docs/ERP_FISCAL_FIELD_MAP_V1.json`, `FISCAL_CONTEXT_REQUIREMENTS_V1.json`, `FISCAL_CONTEXT_COVERAGE_V1.json`,
`NEXT_FISCAL_FIELD_IMPLEMENTATION_V1.md`; raw evidence in `docs/runs/erp_*`. Scripts (all read-only):
`discover_erp_schema.py`, `inspect_erp_table.py`, `analyze_erp_fiscal_history*.py`, `generate_erp_context_docs.py`.
Key facts: NFCE_ITENS (34,946 emitted NFC-e items 2018-2026) + NFCE (1,502) are real fiscal history; C000030 = 590 naturezas
(CFOP/TIPO 1 entrada,2 saída); C000001.IND_CFOP_* company defaults; C000004: UF=BA, CRT=1, OPTANTE_SIMPLES=S, ICMS_CSOSN=102.
Vendor DDL/scripts under D:/PROJETOS/GTISISTEMAS/Diretorios (C000083 confirms CST = origem+CST 3 digits). C000025 CFOP/CST/CSOSN/CEST
are product defaults (99%/100%/95%/99% equal to emitted on identity-consistent joins). CFOP_ENTRADA/CST_ENTRADA/ORIGEM/PIS/COFINS
product fields are unused/legacy. MVA: NO_PRODUCT_FIELD proven (MARGEM_AGREGADA 0/521, unused in C000062). WARNING: NFC-e items
show product identity drift (57% item NCM != current product NCM for same CODPRODUTO) — validate history only on consistent joins.
GTI.FDB (Banco) was NOT opened. Zero Firebird writes.
## Product Fiscal Defaults Engine V1 (2026-09-19)
`central/app/product_defaults.py` (`product-defaults-v1`, pure) + `scripts/run_product_defaults_v1.py`,
`reconcile_ncm_history.py`, `profile_erp_operations.py`. Findings: of 1,676 NFC-e NCM mismatches 1,661 are BAD_JOIN
(97% of mismatches sit in 2022-12; CODPRODUTO does not identify the product there; 1,751 items have no product at all),
4 TRUE_HISTORICAL_DIFFERENCE, 11 UNKNOWN, 0 technical bugs. Trusted corpus = 1,146 identity-proven items. All 1,499 keyed
NFC-e are model 65 / UF 29 (BA): saída, intra-UF, consumidor final. CFOP: 222 consistent with ST status (already correct),
29 ST conflicts (review; 15 of them the ERP actually emitted the current CFOP), 244 context required, 26 unresolved.
CSOSN: never resolved (no ICMS-treatment or ST-retention source); blank CSOSN does NOT default to company 102 in emission
(history emitted 101). CST = reference only (CRT=1). AUTO_APPLY_SAFE=0, writes 0, no auto-repair proof (no safe rule).
Artifacts: PRODUCT_FISCAL_DEFAULTS_{MATRIX,DRY_RUN}_V1.json, NCM_HISTORICAL_RECONCILIATION_V1.json, ERP_OPERATION_PROFILES_V1.json,
FISCAL_CONTEXT_REQUIREMENTS_V2.json, NEXT_FISCAL_FIELD_IMPLEMENTATION_V2.md. No Hub classify change => no cache bump.
## ICMS Context Engine V1 (2026-09-19, read-only)
`central/app/icms_context.py` (`icms-context-v1`, pure) + `scripts/run_icms_context_v1.py`. Purchase-side evidence is thin:
only C000088 (12 items, ICMS_RETIDO='N', ST values 0 → 11 products WITHHELD_NOT_CONFIRMED, 0 CONFIRMED). Role evidence:
C000025.TIPO (7 MATERIA-PRIMA=INPUT), PROD_COZINHA/RECEITA_INGREDIENTES/C000032.TIPO=COMPOSICAO (own production), purchase docs
(resale); TIPO_ITEM/PRODUTO_SERVICO unused. ST: 0 confirmed (ERP CEST blank for 490/521 so no corroboration), 28 not applicable,
268 candidates, 225 unknown. CSOSN V2: 0 resolved, 5 candidates, 490 context required, 26 unresolved (NCM); nothing auto-appliable.
ST conflict audit (29): ERP_HISTORY_CONFLICT 14, DESCRIPTION_MISMATCH 8, MULTIPLE_CEST 4, BA_ST_RULE_CONFLICT 1, NO_EVIDENCE 2.
Artifacts: ICMS_PRODUCT_CONTEXT_V1, ICMS_CONTEXT_COVERAGE_V1, CSOSN_DRY_RUN_V2, FISCAL_CONTEXT_REQUIREMENTS_V3, NEXT_FISCAL_FIELD_IMPLEMENTATION_V3.
No Hub change => no cache bump. Zero Firebird writes.
## SGV product-screen mapping (2026-09-19)
Architecture rule (also in CLAUDE.md): GTI Fiscal writes ONLY the canonical product record C000025; transactional tables are
evidence only (executor allowlist is C000025-only; tests enforce refusal for NFCE/NFCE_ITENS/C000032/C000061/C000062/C000088/INVENTARIO).
Map: `docs/SGV_PRODUCT_FISCAL_UI_MAP_V1.json` (25 UI fields; 0 confirmed by UI diff yet; write_target=true for 6 by equivalent
strong evidence: CLASSIFICACAO_FISCAL, CFOP, CST, CEST, IPI, EX_TIPI). UI oracle tooling: `scripts/sgv_product_snapshot.py before|diff`
(read-only, ledger in docs/runs/sgv_ui_diff_ledger.json) + operator sheet `docs/SGV_UI_MAPPING_SESSION_V1.md`; regenerate the map with
`scripts/generate_sgv_ui_map.py` after diffs. Company UF/CRT now read via `gti_fiscal_agent.firebird_reader.read_company_context`
(fail-closed; runners no longer hardcode BA). Multi-DB read-only auditor: `python audit_firebird_database.py <FDB>`.
## SGV UI mapping session tool (2026-09-19)
`python scripts/map_product_ui_fields.py --product 001434` — interactive, read-only oracle: operator changes one field at a time in the
SGV, tool diffs the full C000025 row, classifies DIRECT_BINDING / DERIVED/COUPLED_FIELD / AUTOMATIC_METADATA / AMBIGUOUS (never picks among
several changed columns), persists `docs/SGV_PRODUCT_FISCAL_UI_DIFF_SESSION_V1.json` step by step (resumable, `--restart`), prints the
original values to restore at the end. `python scripts/generate_sgv_ui_map.py` then rebuilds `docs/SGV_PRODUCT_FISCAL_UI_MAP_V1.json`
(confirmed_by_ui_diff only for DIRECT_BINDING; a control bound to an unexpected column replaces the hypothesised one). Session pending.
## SGV product screen mapped from the executable's DFM (2026-09-19)
`scripts/extract_sgv_dfm_bindings.py <SGV.exe>` parses the Delphi binary DFM of `Tfrmproduto` (read-only): every fiscal control on
the Config. Impostos / Estoque tabs is a data-aware control on `dsproduto2` -> `qrproduto` (`select * from c000025`) with an explicit
DataField; label/control positions match the screenshots. Result in `docs/SGV_PRODUCT_FISCAL_UI_MAP_V1.json`: 23 UI controls,
21 CONFIRMED_STRONG, 1 PROBABLE (origem: RadioGroup + hidden edit bound to COD_ORIGEM_MERCADORIA), 1 NO_MATCH (CSOSN description
combo is not data-aware). Screen writes CFOP_ENTRADA/CST_ENTRADA/ORIGEM-code even though they are empty in the data. The T/I/N/S
selector next to CEST is SITUACAO_TRIBUTARIA (S = Substituição Tributária = the ERP's own product ST flag; 0/521 products use S).
Cálculo Preço tab controls (IPI/PIS/COFINS) bind to C000026 (price calc), not C000025. write_target = binding proven; the executor
allowlist and the fiscal rule engines still decide what may actually be written (executor: C000025.IPI only).
## Multi-database validation (2026-09-19, read-only, `Banco/*.FDB` git-ignored)
4 copies audited: CONSTRUCAO (construction, 3,712 products, BA/CRT1), MMMODAS (apparel, 20,997, BA/CRT2), RINCAO (restaurant, 894, C000004 has 3 company rows -> ambiguous), UNIAO (market, 8,942, BA/CRT3). All expose C000004/C000025 with the 21 write-target columns and no type variation except UNIAO CEST VARCHAR(10) vs (7). One generic Agent is viable; needs company selector + CRT source of truth (C000004.CRT disagrees with EMISSOR.EMITENTE_CRT in 3 of 4) and capability detection (`scripts/sgv_capabilities.py`). Tools: `scripts/audit_multi_databases.py`, `run_multi_db_coverage.py` (in-container bulk classifier `bulk_classify_in_container.py`), `generate_multi_db_findings.py`. Artifacts: MULTI_DATABASE_{SCHEMA_AUDIT,FISCAL_COVERAGE,FIELD_COMPATIBILITY}_V1.json, MULTI_DATABASE_FINDINGS_V1.md. No auto-repair DB provided. Zero writes.
## Company context resolution + cache fix (2026-09-19)
`agent/src/gti_fiscal_agent/company_context.py`: deterministic ACTIVE_COMPANY/UF/CRT resolver (EMISSOR CNPJ ↔ C000004 CNPJ ↔ emitted NFC-e key CNPJ; CRT authority for emission = EMISSOR because the emitter executables bind QrEmissor.EMITENTE_*; C000004.CRT is the ERP/SPED filial registry). Never uses first row; config `COMPANY_CODE`/`COMPANY_CNPJ`/`COMPANY_CRT_SOURCE`; preflight `agent/scripts/preflight_company_context.py`, enforced at Agent startup; `read_company_context` fails closed. 4 databases: CONSTRUCAO AUTO_RESOLVED; MMMODAS/RINCAO/UNIAO CRT_SOURCE_CONFLICT (need COMPANY_CRT_SOURCE). Rules: `docs/AGENT_COMPANY_CONTEXT_RULES_V1.md`; evidence: COMPANY_CONTEXT_RESOLUTION_V1.json, MULTI_DATABASE_COMPANY_CONTEXT_V1.json.
Hub `classification_cache` fixed: fingerprint (tenant + input + ENGINE_VERSION + dataset_versions()) is computed first and a HIT returns without running the Classifier (test: `central/tests/test_classification_cache.py`). Live benchmark (100 products): MISS 337 ms/product, HIT 15 ms/product, 100/100 hits, identical results. Fingerprint inputs unchanged, so existing cache entries stay valid and version bumps still invalidate naturally.
## Release Candidate 1.0.0-rc1 (2026-09-19) — COMPLETE_LOCAL_RC
Frozen: `release/VERSION`, `central/app/version.py` (`GET /version`), `release/RELEASE_MANIFEST.json` (versions + ACTIVE dataset hashes + migrations 0001..0010). See `docs/RELEASE_CANDIDATE_V1.md`.
Architecture: Hub central (Docker Hub+PostgreSQL, `deploy/hub`) <-HTTPS- Agent Windows (`deploy/agent-windows`, service GTIFiscalAgent) -local read-only-> SGV Firebird. The Hub never accesses Firebird.
Agent: `FISCAL_MODE` DRY_RUN|AUTO_APPLY (DRY_RUN = technically no writes), `COMPANY_CODE/COMPANY_CNPJ/COMPANY_CRT_SOURCE(AUTO|EMISSOR|C000004)`, `COMPANY_UF` explicit override only; single preflight `agent/scripts/preflight.py` (also enforced at service start); REPROCESS_ALL job result includes the coverage report + company context (`gti_fiscal_agent/coverage.py`, shared with multi-db tooling); `scripts/prove_zero_write.py` (DML counters + table digests). Packaging: `scripts/build_agent_package.py` (secret-scanned zip). Hub prod: `deploy/hub/docker-compose.prod.yml` (Postgres internal only), backup/restore, dataset seed export/import; `APP_ENV=production` refuses AGENT_DEV_TOKEN.
Pilot profile: CONSTRUCAO (`pilot-construcao.env.example`); planned, not installed: MMMODAS/RINCAO/UNIAO with `COMPANY_CRT_SOURCE=EMISSOR`. No real host was accessed: a local simulation ran the Agent (registered agent, DRY_RUN) end-to-end on the CONSTRUCAO copy. Follow-up: memoize `dataset_versions()`.
## Local Completion V1 (2026-09)
- Contrato: `docs/FISCAL_FIELD_CONTRACT_V1.json` (23 campos, gerado por `scripts/generate_field_contract.py`); resolvedor puro `central/app/field_contract.py`.
- Contexto do operador (fatos que o ERP não conhece): tabela `fiscal_context_entries` (migration 0011), `POST /v1/context/import`, `scripts/import_fiscal_context.py` (CSV/JSON). Hierarquia PRODUCT > GROUP > NCM > COMPANY.
- Escrita: somente as 5 cópias descartáveis em `docs/LOCAL_WRITE_ALLOWLIST_V1.json` (caminho exato), só C000025, via `execute_guarded_product_update`.
- Runner: `scripts/local_completion.py` (estado em `docs/runs/local_completion/`, git-ignored); relatórios: `scripts/local_completion_report.py`.
- Resultado: 41.354 células corrigidas, 100% verificadas, 2ª passada 0 escritas; 255.643 células aguardam resposta (`docs/LOCAL_FISCAL_CONTEXT_QUESTIONS_V1.*`, 1.054 perguntas). Fontes externas pendentes: `docs/LOCAL_COMPLETION_MANUAL_FILES_V1.md`.
## Local Completion V2 (2026-09-20)
Audit outcome in `docs/LOCAL_COMPLETION_V2_REPORT.md`: origin rule invalid → rolled back (35,048); CFOP/IPPT weak-evidence writes rolled back; role no longer inferred from PROD_COZINHA alone; SPED 4.3.x dataset in `docs/sources/`; PIS/COFINS Simples rule from ERP emission evidence; questionnaire V2 (`docs/LOCAL_FISCAL_CONTEXT_QUESTIONS_V2.*`). Rollback tool: `scripts/rollback_field.py`; audit: `scripts/audit_local_writes.py`.