chore: establish project baseline
This commit is contained in:
@@ -0,0 +1,375 @@
|
||||
# AI Project State
|
||||
|
||||
## Current status
|
||||
|
||||
`FISCAL_DATA_FOUNDATION_MOSTLY_COMPLETE`. Deploy/pilot work remains paused
|
||||
per the current mission's scope. The operator manually placed the 6 official
|
||||
files that were blocked by the Portal NF-e/SPED session gate into
|
||||
`sources/raw/bootstrap_v1/`; all 6 were identified by content (never by
|
||||
filename), hashed, parsed, validated, and promoted `ACTIVE` — no internet
|
||||
access was attempted or needed this pass. **15 fiscal domains are now
|
||||
`ACTIVE`**: NCM, TIPI_IPI, CEST, BA_ST_MVA, ORIGEM_MERCADORIA, CST_ICMS,
|
||||
CSOSN, CST_PIS, CST_COFINS (from the prior pass), plus **CFOP, FCP, ANP,
|
||||
NCM_UTRIB, RTC_BIOFUEL_INDEX, RTC_CCREDPRES** (new this pass) —
|
||||
**42,593 fiscal records persisted total**. 11 domains remain
|
||||
`MISSING_BOOTSTRAP_FILE` (ICMS monofásico, RTC cClassTrib, RTC alíquotas
|
||||
CBS, and all 8 PIS/COFINS Tabela 4.3.10–4.3.17 files) — no file for them
|
||||
exists locally; internet access was deliberately not attempted this
|
||||
session. See `docs/SOURCE_REGISTRY_V2.md`. Read-only coverage over all 521
|
||||
`TESTESOFICIAIS.FDB` products is in `docs/FISCAL_COVERAGE_FULL_V2.json`
|
||||
(supersedes `_V1.json`, kept for history). Zero Firebird writes this
|
||||
session (confirmed independently: `000026.IPI` still `3.25`, 109 non-null
|
||||
`IPI` rows unchanged).
|
||||
|
||||
`PILOT_DEPLOYMENT_READY` (previous mission's state, unchanged this pass). The
|
||||
Hub (central/) + Agent (agent/) system is
|
||||
operational end-to-end via a real Job queue, deploy-ready via Docker
|
||||
Compose, and the Agent is installable as a Windows Service. Formal Agent
|
||||
registration exists (`central/scripts/register_agent.py`), each Agent has
|
||||
its own token, and write authorization for a real client installation now
|
||||
requires two explicit, independently-configured layers (path + schema
|
||||
fingerprint) rather than the dev-only hardcoded path alone. See
|
||||
`docs/OPERATIONS.md` for day-to-day commands and `docs/PILOT_DEPLOYMENT.md`
|
||||
for standing up a new environment end to end.
|
||||
|
||||
Firebird discovery (Mission 00/01) remains read-only and unchanged; its
|
||||
sanitized evidence lives in `docs/SAMPLE_PRODUCTS_V1.json` and
|
||||
`docs/MISSION_01_REPORT.md`.
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Hub** (`central/app`): FastAPI + PostgreSQL. Classifies NCM/CEST/TIPI/
|
||||
BA_ST_MVA (`classification.py`), owns the Agent registry and Job queue
|
||||
(`models.py`, `jobs.py`, `api.py`). Never touches Firebird, never stores
|
||||
Firebird credentials.
|
||||
- **Agent** (`agent/src/gti_fiscal_agent`, `agent/scripts`): reads Firebird
|
||||
read-only, calls the Hub's `/v1/classify/batch`, runs the IPI auto-apply
|
||||
decision (`ipi_auto_apply.py`) through the shared pipeline
|
||||
(`pipeline.py`), and — only when `FIREBIRD_WRITES=1` and the database
|
||||
path is exactly the authorized disposable test file — writes via the
|
||||
guarded executor (`scripts/firebird_write_executor.py`). Long-running
|
||||
entrypoint: `agent/scripts/run_agent_service.py` (heartbeat + Job poll
|
||||
loop). One-shot CLI variant kept for manual use:
|
||||
`agent/scripts/run_agent_auto_apply.py`.
|
||||
- Local per-product audit: Agent's SQLite (`agent/agent_state.sqlite`,
|
||||
table `ipi_write_log`). Central per-Job audit: Postgres `jobs.result_summary`.
|
||||
|
||||
## Firebird write policy
|
||||
|
||||
`TESTESOFICIAIS.FDB` is the only Firebird database ever authorized for
|
||||
writes in development (see `CLAUDE.md`). For a real pilot installation, two
|
||||
independent, explicitly-configured layers are both required before any
|
||||
write connection opens — never inferred from a filename or `.FDB` extension:
|
||||
`AUTHORIZED_WRITE_DATABASE_PATH` (must equal that installation's own
|
||||
`FIREBIRD_DATABASE_PATH`) and `EXPECTED_SCHEMA_FINGERPRINT` (must equal the
|
||||
live schema fingerprint that installation reports). Leaving either unset
|
||||
falls back to the dev-only constant, so a real client database is refused
|
||||
by default (`scripts/firebird_write_executor.py::assert_authorized_write_path`
|
||||
/ `assert_schema_fingerprint_authorized`, wired into
|
||||
`agent/scripts/run_agent_service.py::execute_job`). All other Firebird
|
||||
databases (including `Banco/GTI.FDB`, found but never opened) remain
|
||||
read-only. No real client database has been written to in any mission —
|
||||
every write so far targeted `TESTESOFICIAIS.FDB`.
|
||||
|
||||
## Auto-apply scope
|
||||
|
||||
Only `C000025.IPI`, only when the Hub's TIPI match is `rate_kind=NUMERIC`
|
||||
and unambiguous. Never CEST, EX_TIPI, CFOP, CST/CSOSN, PIS/COFINS, MVA, or
|
||||
non-numeric/blank/legacy-NCM cases — those remain measurement-only
|
||||
(`docs/DRY_RUN_PLAN_V4.json`, `docs/FULL_DATABASE_DRY_RUN_V1.json`).
|
||||
|
||||
## E2E proof (2026-09-18)
|
||||
|
||||
Real Job created via `POST /v1/jobs` (`REPROCESS_PRODUCTS`), claimed and
|
||||
executed by `run_agent_service.py` (not the manual batch script):
|
||||
manually-induced drift (`C000025.IPI` forced to NULL on product `000026`)
|
||||
was detected and auto-restored to `3.25`, verified independently in
|
||||
Firebird, `jobs.result_summary` on the Hub, and the Agent's local
|
||||
`ipi_write_log`. A second run against the same now-correct product
|
||||
produced `writes_attempted=0` (idempotent no-op). `REPROCESS_ALL` over all
|
||||
521 products completed with `writes_attempted=0` (everything already
|
||||
correct from the prior full-database batch). A job created with
|
||||
`FIREBIRD_WRITES=0` against a genuinely eligible drifted product completed
|
||||
with `writes_attempted=0` and the ERP value left untouched, with the
|
||||
would-be write logged locally. The database was left in its correct state.
|
||||
|
||||
During this session Docker Desktop itself restarted mid-mission (external
|
||||
event, not induced); the Hub came back healthy on its own once Docker
|
||||
recovered, and all prior Job history was intact (Postgres volume
|
||||
persisted) — an incidental real-world confirmation of "Hub temporarily
|
||||
unavailable, Agent/data survive."
|
||||
|
||||
## Pilot readiness proof (2026-09-18)
|
||||
|
||||
Registered a real Agent via `central/scripts/register_agent.py` (found and
|
||||
fixed a `DetachedInstanceError` bug in the process: the script read ORM
|
||||
attributes after the session closed — fixed to capture plain values before
|
||||
closing, with a regression test that mirrors the real CLI's usage pattern).
|
||||
Ran the Agent under that fresh, individually-hashed token end to end:
|
||||
startup validation, heartbeat (visible via `GET /v1/agents/me`, correctly
|
||||
derives `ONLINE`), job creation → claim → `COMPLETED` with a real product,
|
||||
restart with SQLite/central history intact, and a live path-guard test
|
||||
(fake client path, `FIREBIRD_WRITES=1`, no explicit authorization) that
|
||||
failed closed before any Firebird connection attempt and reported the job
|
||||
`FAILED` with a sanitized error while the Agent kept running. Also
|
||||
restarted the Hub and Postgres containers independently mid-session and
|
||||
confirmed the Agent/Hub recovered and history persisted through both.
|
||||
|
||||
Also fixed a Docker Compose bug introduced while making the Hub
|
||||
deploy-configurable: passing `DATABASE_URL` through from the shared root
|
||||
`.env` leaked the host-side (`localhost`-pointing) value into the
|
||||
container's network namespace, where `localhost` means the container
|
||||
itself, breaking Postgres connectivity. Removed that passthrough; the
|
||||
Hub's own `postgres:5432` default (from `central/app/config.py`) is used
|
||||
instead.
|
||||
|
||||
## Fiscal data foundation (2026-09-18)
|
||||
|
||||
`docs/SOURCE_REGISTRY_V2.md` is now the current source registry (`V1` kept
|
||||
for history). Summary:
|
||||
|
||||
- **Preserved, re-verified**: NCM (15,156 entries, hash-verified as
|
||||
unchanged — no internet available to re-fetch, so this is "verified
|
||||
against local record" not "re-fetched"), TIPI_IPI (11,103 entries,
|
||||
hash-verified byte-identical against the local xlsx), CEST (1,051
|
||||
entries, preserved — no internet to re-fetch, no local raw copy to
|
||||
hash-check against).
|
||||
- **Re-imported with updated provenance**: BA_ST_MVA — the local PDF's bytes
|
||||
changed (new `raw_sha256`) but the parsed content is identical (303/303
|
||||
records match exactly what was previously `ACTIVE`); promoted a new
|
||||
dataset row pointing at the current file rather than leaving provenance
|
||||
referencing a stale path.
|
||||
- **New this session**: `fiscal_code_entries` table (migration `0007`)
|
||||
holding ORIGEM_MERCADORIA (9), CST_ICMS (11), CSOSN (10), CST_PIS (33),
|
||||
CST_COFINS (33) — extracted directly from the official
|
||||
`moc7-anexo-i-leiaute-e-rv.pdf` (Receita Federal/ENCAT), one shared
|
||||
`Dataset` row (`kind=NFE_MOC_TABLES`). Parser:
|
||||
`central/app/datasets/nfe_moc_tables.py`. Nothing invented: entries built
|
||||
from a combined official phrase (CST ICMS 40/41/50; CSOSN
|
||||
102/103/300/400) keep that phrasing rather than being split further than
|
||||
the source itself supports; the 6 CST_COFINS codes not separately
|
||||
captured from the PDF's COFINS section (a page-boundary text-extraction
|
||||
artifact, not a semantic gap) are filled from the CST_PIS wording after
|
||||
confirming character-for-character identical wording for every code both
|
||||
extractors did capture — each such row is labeled
|
||||
`metadata_json.mirrored_from = "CST_PIS"`.
|
||||
- **Second pass (bootstrap_v1 operator-supplied files, migrations `0008`/`0009`)**:
|
||||
the operator placed 6 previously-blocked official files directly into
|
||||
`sources/raw/bootstrap_v1/`. Each was identified by actual sheet
|
||||
names/header rows (never by filename) and promoted `ACTIVE`:
|
||||
**CFOP** (619 codes, Informe Técnico 2023.002, new dedicated
|
||||
`cfop_entries` table with all 14 official columns including the RTC-era
|
||||
`indExcIBSCBS`), **FCP** (27 UFs, `fcp_rates`), **ANP** (1031 `cProdANP`
|
||||
codes — confirmed the official NF-e portal table genuinely has no
|
||||
description column, not an omission — `anp_product_codes`),
|
||||
**NCM_UTRIB** (10,516 rows, `ncm_utrib_entries`), **RTC_BIOFUEL_INDEX**
|
||||
(23 rows, current + historical `pBioObrigatorio`, `rtc_biofuel_index_entries`),
|
||||
**RTC_CCREDPRES** (13 codes, all 18 official LC 214/2025 fields preserved
|
||||
— legal basis, CBS/IBS indicators, rates, vigências —
|
||||
`rtc_cred_pres_entries`). Bug found mid-import: `cclass_nota_referenciada`
|
||||
was `VARCHAR(16)` but some official rows carry free text there; widened
|
||||
to `Text` in migration `0009` after a clean rollback (no orphan row, no
|
||||
data loss). New read-only endpoints:
|
||||
`GET /v1/reference/{cfop,ncm_utrib,anp,fcp}/{code}`.
|
||||
- **Still `MISSING_BOOTSTRAP_FILE`** (not `FAILED` — simply no file
|
||||
provided, internet deliberately not attempted this session): ICMS
|
||||
monofásico (combustíveis), RTC cClassTrib, RTC alíquotas CBS, and all 8
|
||||
PIS/COFINS Tabela 4.3.10–4.3.17 files.
|
||||
- **Coverage matrix**: `docs/FISCAL_COVERAGE_FULL_V2.json` (supersedes
|
||||
`_V1.json`), all 521 products in `TESTESOFICIAIS.FDB`, read-only (zero
|
||||
Firebird writes). NCM 495 valid / 3 legacy / 23 invalid / 0 missing. IPI
|
||||
107 numeric / 20 non-numeric / 297 source-blank / 71 ambiguous / 23
|
||||
no-match. CEST: 462 matches, 0 confirmed (structural ambiguity,
|
||||
unchanged), 15 candidates. BA_ST_MVA: 268 matches, 31 resolved, 237
|
||||
candidates. **New**: CFOP match_found=498/521 (this product's own
|
||||
`C000025.CFOP` value is an officially valid code); NCM_UTRIB
|
||||
match_found=495/521 (tracks `ncm_valid` exactly, as expected). ANP/FCP/
|
||||
RTC_CCREDPRES/RTC_BIOFUEL_INDEX are `DATA_AVAILABLE=true` but
|
||||
`match_found=0` — stated explicitly as a data-shape fact (this grocery/
|
||||
restaurant ERP has no ANP/fuel-product/UF join key on `C000025`), not a
|
||||
gap or a bug. PIS/COFINS CST domain remains `DATA_AVAILABLE`,
|
||||
`RULE_RESOLVED=0` for all products (same reason as before: no NCM-linked
|
||||
determination table exists to acquire yet).
|
||||
|
||||
## Safety counters (cumulative across missions)
|
||||
|
||||
- `APPLICATION_DDL_DML_WRITES` against Firebird: only ever against
|
||||
`TESTESOFICIAIS.FDB`, only `UPDATE C000025.IPI` / `UPDATE C000025.CEST`-class
|
||||
statements via the guarded executor (CEST never actually written — no
|
||||
`CEST_CONFIRMED` case existed with enough evidence).
|
||||
- `OTHER_DATABASES_ACCESSED = 0` (including `Banco/GTI.FDB`, never opened).
|
||||
- Postgres migrations: `0001`..`0009` applied (`0006` adds `jobs` table and
|
||||
`agents` status/heartbeat columns; `0007` adds `fiscal_code_entries`;
|
||||
`0008` adds `cfop_entries`/`fcp_rates`/`anp_product_codes`/
|
||||
`ncm_utrib_entries`/`rtc_biofuel_index_entries`/`rtc_cred_pres_entries`;
|
||||
`0009` widens `rtc_cred_pres_entries.cclass_nota_referenciada` to `Text`).
|
||||
- This session: `FIREBIRD_WRITES=0` throughout — zero writes, zero DML/DDL
|
||||
against any Firebird database (only read-only SELECTs to build the
|
||||
coverage matrix); independently re-verified after the session (`000026.IPI`
|
||||
still `3.25`, 109 non-null `IPI` rows unchanged).
|
||||
- Total fiscal records persisted in Postgres across all domains: 42,593.
|
||||
|
||||
## Threat model (Firebird discovery boundary, unchanged since Mission 00)
|
||||
|
||||
The structured boundary protects normal callers against implementation mistakes,
|
||||
accidental unauthorized operations, arbitrary SQL exposed to normal
|
||||
callers, the wrong database, writes, procedures/UDFs, injection, and
|
||||
dependency shadowing. `TRUSTED_PROCESS = TRUE`: arbitrary Python already
|
||||
running in the same process, including deliberate introspection or monkey
|
||||
patching, is outside scope. No artificial cryptographic controls are added.
|
||||
|
||||
## Known gaps / next steps
|
||||
|
||||
- No "list all agents" or "disable agent" HTTP endpoint yet — disabling an
|
||||
agent is a direct `agents.enabled` update via `DATABASE_URL` access (see
|
||||
`docs/PILOT_DEPLOYMENT.md`'s "Rollback"). Fine for pilot scale; a small
|
||||
admin surface is a natural follow-up once there's more than a couple of
|
||||
installations.
|
||||
- The Windows Service wrapper (`agent/service/gti_agent_windows_service.py`)
|
||||
was written and unit-verified (module imports cleanly, stop-flag mechanism
|
||||
tested against `run_agent_service.run()`), but was **not** registered with
|
||||
the Windows Service Control Manager on this machine — `sc.exe`/pywin32
|
||||
`install` modifies system-wide service state, which this session treats as
|
||||
outside what it does unprompted. An operator runs the documented
|
||||
`install`/`start` commands themselves.
|
||||
- No auto-update mechanism (deliberately out of scope this round) — updates
|
||||
are a manual stop/replace/start, documented in `docs/PILOT_DEPLOYMENT.md`.
|
||||
- CEST matching still cannot confirm any of the 5 known-ambiguous products
|
||||
(structural NCM→multiple-CEST ambiguity in CONFAZ data, not a matcher bug
|
||||
— see `docs/DRY_RUN_PLAN_V4.json`).
|
||||
- `NT_STATUS_FIELD` (how a TIPI "NT" marker should map to `C000025.CST_IPI`)
|
||||
remains unmapped — no reliable ERP evidence found for it.
|
||||
- 11 fiscal domains are `MISSING_BOOTSTRAP_FILE` (not `FAILED` — no file was
|
||||
ever provided for them): ICMS monofásico, RTC cClassTrib, RTC alíquotas
|
||||
CBS, and all 8 PIS/COFINS Tabela 4.3.10–4.3.17 files. CFOP, FCP, ANP,
|
||||
NCM×uTrib, RTC_CCREDPRES, RTC_BIOFUEL_INDEX — all previously `FAILED` —
|
||||
are now `ACTIVE` (operator supplied the official files). See
|
||||
`docs/SOURCE_REGISTRY_V2.md` for exact evidence and next steps.
|
||||
- PIS/COFINS: only the CST code domain (Tabela 4.3.3/4.3.4-equivalent) is
|
||||
`ACTIVE`. The NCM-linked determination tables (SPED Guia Prático
|
||||
EFD-Contribuições Tabelas 4.3.10–4.3.17: alíquotas diferenciadas,
|
||||
monofásicas, ST, zero, isenção, sem incidência, suspensão) were not
|
||||
acquired — no file provided, internet deliberately not attempted.
|
||||
- CST_ICMS/CSOSN/ORIGEM_MERCADORIA/CST_PIS/CST_COFINS are consultable
|
||||
(`DatasetService.fiscal_codes(domain, code=None)`) but not yet wired into
|
||||
`Classifier` — no per-product automatic determination exists for these
|
||||
domains, by design (mission explicitly forbade inventing fiscal rules).
|
||||
|
||||
## Git State
|
||||
|
||||
No Git identity was configured and no Git configuration was modified this
|
||||
session.
|
||||
|
||||
## Business scope (2026-09-19)
|
||||
Clients: restaurants, markets, clothing/retail, workshops, commerce. No fuel
|
||||
stations, no EFD-Contribuicoes. CORE_MVP still missing only RTC_CCLASSTRIB and
|
||||
RTC_CBS_RATES (`MISSING_BOOTSTRAP_FILE`, awaiting operator files; see
|
||||
SOURCE_REGISTRY_V2.md). ICMS_MONOPHASIC = OUT_OF_SCOPE_MVP; ANP and
|
||||
RTC_BIOFUEL_INDEX = ACTIVE_REFERENCE_ONLY; PIS_COFINS_4_3_10..17 =
|
||||
DEFERRED_BY_BUSINESS_SCOPE (not deploy blockers; existing support kept).
|
||||
FISCAL_COVERAGE_FULL_V3 to be generated once the two core domains are ACTIVE.
|
||||
|
||||
## CORE MVP fiscal foundation COMPLETE (2026-09-19)
|
||||
RTC_CCLASSTRIB (164 cClassTrib + 18 CST) and RTC_CBS_RATES (2 rows) are ACTIVE
|
||||
(migration 0010, endpoints `/v1/reference/rtc_cclasstrib/{code}`,
|
||||
`/v1/reference/rtc_cbs_rate/{year}`). All 14 CORE domains ACTIVE; nothing CORE
|
||||
is missing. Total fiscal records 42,777. Coverage: `docs/FISCAL_COVERAGE_FULL_V3.json`
|
||||
(V2 kept for history). cClassTrib/cCredPres have no ERP join key → DATA_AVAILABLE
|
||||
only; no rule invented. Zero Firebird writes.
|
||||
|
||||
## Fiscal Decision Engine V1 (2026-09-19)
|
||||
`central/app/fiscal_decision.py` (pure, `DECISION_ENGINE_VERSION=fiscal-decision-v1`, carried in every
|
||||
provenance) + `scripts/run_fiscal_decision_v1.py` (read-only over the 521 products; `--apply` writes only
|
||||
AUTO_APPLY_SAFE via the guarded executor). Artifacts: `docs/FISCAL_DECISION_MATRIX_V1.json` (per product/domain:
|
||||
current/proposed/status/reason/confidence/provenance/write_eligible + ERP field map),
|
||||
`docs/FISCAL_DECISION_COVERAGE_V1.json`, `docs/NEXT_FISCAL_AUTOMATION_PRIORITIES_V1.md`.
|
||||
Company context read live: UF=BA, CRT=1 (C000004). ERP mappings: 11 confirmed, 10 MAPPING_UNCONFIRMED
|
||||
(CFOP_ENTRADA, CST_ENTRADA, COD_*_PIS/COFINS, ALIQ_PIS/COFINS, CST_IPI, ORIGEM). Result: AUTO_APPLY_SAFE = 0
|
||||
(IPI already correct on all 107 numeric rows; CEST never single-confirmed; a lone EX row per NCM is NOT
|
||||
evidence for a product — needs description match). CFOP/CSOSN/CST/PIS/COFINS/FCP/RTC = OPERATION_CONTEXT_REQUIRED
|
||||
or unresolved; CBS 0.009 kept as reference only, never applied universally. No Hub classify change, so no
|
||||
classification_cache bump needed. Helper `scripts/audit_erp_fields.py` (read-only). New endpoint
|
||||
`GET /v1/reference/fiscal_codes/{domain}`. Zero Firebird writes this mission.
|
||||
|
||||
## ERP Fiscal Context Discovery V1 (2026-09-19, read-only)
|
||||
Artifacts: `docs/ERP_FISCAL_FIELD_MAP_V1.json`, `FISCAL_CONTEXT_REQUIREMENTS_V1.json`, `FISCAL_CONTEXT_COVERAGE_V1.json`,
|
||||
`NEXT_FISCAL_FIELD_IMPLEMENTATION_V1.md`; raw evidence in `docs/runs/erp_*`. Scripts (all read-only):
|
||||
`discover_erp_schema.py`, `inspect_erp_table.py`, `analyze_erp_fiscal_history*.py`, `generate_erp_context_docs.py`.
|
||||
Key facts: NFCE_ITENS (34,946 emitted NFC-e items 2018-2026) + NFCE (1,502) are real fiscal history; C000030 = 590 naturezas
|
||||
(CFOP/TIPO 1 entrada,2 saída); C000001.IND_CFOP_* company defaults; C000004: UF=BA, CRT=1, OPTANTE_SIMPLES=S, ICMS_CSOSN=102.
|
||||
Vendor DDL/scripts under D:/PROJETOS/GTISISTEMAS/Diretorios (C000083 confirms CST = origem+CST 3 digits). C000025 CFOP/CST/CSOSN/CEST
|
||||
are product defaults (99%/100%/95%/99% equal to emitted on identity-consistent joins). CFOP_ENTRADA/CST_ENTRADA/ORIGEM/PIS/COFINS
|
||||
product fields are unused/legacy. MVA: NO_PRODUCT_FIELD proven (MARGEM_AGREGADA 0/521, unused in C000062). WARNING: NFC-e items
|
||||
show product identity drift (57% item NCM != current product NCM for same CODPRODUTO) — validate history only on consistent joins.
|
||||
GTI.FDB (Banco) was NOT opened. Zero Firebird writes.
|
||||
|
||||
## Product Fiscal Defaults Engine V1 (2026-09-19)
|
||||
`central/app/product_defaults.py` (`product-defaults-v1`, pure) + `scripts/run_product_defaults_v1.py`,
|
||||
`reconcile_ncm_history.py`, `profile_erp_operations.py`. Findings: of 1,676 NFC-e NCM mismatches 1,661 are BAD_JOIN
|
||||
(97% of mismatches sit in 2022-12; CODPRODUTO does not identify the product there; 1,751 items have no product at all),
|
||||
4 TRUE_HISTORICAL_DIFFERENCE, 11 UNKNOWN, 0 technical bugs. Trusted corpus = 1,146 identity-proven items. All 1,499 keyed
|
||||
NFC-e are model 65 / UF 29 (BA): saída, intra-UF, consumidor final. CFOP: 222 consistent with ST status (already correct),
|
||||
29 ST conflicts (review; 15 of them the ERP actually emitted the current CFOP), 244 context required, 26 unresolved.
|
||||
CSOSN: never resolved (no ICMS-treatment or ST-retention source); blank CSOSN does NOT default to company 102 in emission
|
||||
(history emitted 101). CST = reference only (CRT=1). AUTO_APPLY_SAFE=0, writes 0, no auto-repair proof (no safe rule).
|
||||
Artifacts: PRODUCT_FISCAL_DEFAULTS_{MATRIX,DRY_RUN}_V1.json, NCM_HISTORICAL_RECONCILIATION_V1.json, ERP_OPERATION_PROFILES_V1.json,
|
||||
FISCAL_CONTEXT_REQUIREMENTS_V2.json, NEXT_FISCAL_FIELD_IMPLEMENTATION_V2.md. No Hub classify change => no cache bump.
|
||||
|
||||
## ICMS Context Engine V1 (2026-09-19, read-only)
|
||||
`central/app/icms_context.py` (`icms-context-v1`, pure) + `scripts/run_icms_context_v1.py`. Purchase-side evidence is thin:
|
||||
only C000088 (12 items, ICMS_RETIDO='N', ST values 0 → 11 products WITHHELD_NOT_CONFIRMED, 0 CONFIRMED). Role evidence:
|
||||
C000025.TIPO (7 MATERIA-PRIMA=INPUT), PROD_COZINHA/RECEITA_INGREDIENTES/C000032.TIPO=COMPOSICAO (own production), purchase docs
|
||||
(resale); TIPO_ITEM/PRODUTO_SERVICO unused. ST: 0 confirmed (ERP CEST blank for 490/521 so no corroboration), 28 not applicable,
|
||||
268 candidates, 225 unknown. CSOSN V2: 0 resolved, 5 candidates, 490 context required, 26 unresolved (NCM); nothing auto-appliable.
|
||||
ST conflict audit (29): ERP_HISTORY_CONFLICT 14, DESCRIPTION_MISMATCH 8, MULTIPLE_CEST 4, BA_ST_RULE_CONFLICT 1, NO_EVIDENCE 2.
|
||||
Artifacts: ICMS_PRODUCT_CONTEXT_V1, ICMS_CONTEXT_COVERAGE_V1, CSOSN_DRY_RUN_V2, FISCAL_CONTEXT_REQUIREMENTS_V3, NEXT_FISCAL_FIELD_IMPLEMENTATION_V3.
|
||||
No Hub change => no cache bump. Zero Firebird writes.
|
||||
|
||||
## SGV product-screen mapping (2026-09-19)
|
||||
Architecture rule (also in CLAUDE.md): GTI Fiscal writes ONLY the canonical product record C000025; transactional tables are
|
||||
evidence only (executor allowlist is C000025-only; tests enforce refusal for NFCE/NFCE_ITENS/C000032/C000061/C000062/C000088/INVENTARIO).
|
||||
Map: `docs/SGV_PRODUCT_FISCAL_UI_MAP_V1.json` (25 UI fields; 0 confirmed by UI diff yet; write_target=true for 6 by equivalent
|
||||
strong evidence: CLASSIFICACAO_FISCAL, CFOP, CST, CEST, IPI, EX_TIPI). UI oracle tooling: `scripts/sgv_product_snapshot.py before|diff`
|
||||
(read-only, ledger in docs/runs/sgv_ui_diff_ledger.json) + operator sheet `docs/SGV_UI_MAPPING_SESSION_V1.md`; regenerate the map with
|
||||
`scripts/generate_sgv_ui_map.py` after diffs. Company UF/CRT now read via `gti_fiscal_agent.firebird_reader.read_company_context`
|
||||
(fail-closed; runners no longer hardcode BA). Multi-DB read-only auditor: `python audit_firebird_database.py <FDB>`.
|
||||
|
||||
## SGV UI mapping session tool (2026-09-19)
|
||||
`python scripts/map_product_ui_fields.py --product 001434` — interactive, read-only oracle: operator changes one field at a time in the
|
||||
SGV, tool diffs the full C000025 row, classifies DIRECT_BINDING / DERIVED/COUPLED_FIELD / AUTOMATIC_METADATA / AMBIGUOUS (never picks among
|
||||
several changed columns), persists `docs/SGV_PRODUCT_FISCAL_UI_DIFF_SESSION_V1.json` step by step (resumable, `--restart`), prints the
|
||||
original values to restore at the end. `python scripts/generate_sgv_ui_map.py` then rebuilds `docs/SGV_PRODUCT_FISCAL_UI_MAP_V1.json`
|
||||
(confirmed_by_ui_diff only for DIRECT_BINDING; a control bound to an unexpected column replaces the hypothesised one). Session pending.
|
||||
|
||||
## SGV product screen mapped from the executable's DFM (2026-09-19)
|
||||
`scripts/extract_sgv_dfm_bindings.py <SGV.exe>` parses the Delphi binary DFM of `Tfrmproduto` (read-only): every fiscal control on
|
||||
the Config. Impostos / Estoque tabs is a data-aware control on `dsproduto2` -> `qrproduto` (`select * from c000025`) with an explicit
|
||||
DataField; label/control positions match the screenshots. Result in `docs/SGV_PRODUCT_FISCAL_UI_MAP_V1.json`: 23 UI controls,
|
||||
21 CONFIRMED_STRONG, 1 PROBABLE (origem: RadioGroup + hidden edit bound to COD_ORIGEM_MERCADORIA), 1 NO_MATCH (CSOSN description
|
||||
combo is not data-aware). Screen writes CFOP_ENTRADA/CST_ENTRADA/ORIGEM-code even though they are empty in the data. The T/I/N/S
|
||||
selector next to CEST is SITUACAO_TRIBUTARIA (S = Substituição Tributária = the ERP's own product ST flag; 0/521 products use S).
|
||||
Cálculo Preço tab controls (IPI/PIS/COFINS) bind to C000026 (price calc), not C000025. write_target = binding proven; the executor
|
||||
allowlist and the fiscal rule engines still decide what may actually be written (executor: C000025.IPI only).
|
||||
|
||||
## Multi-database validation (2026-09-19, read-only, `Banco/*.FDB` git-ignored)
|
||||
4 copies audited: CONSTRUCAO (construction, 3,712 products, BA/CRT1), MMMODAS (apparel, 20,997, BA/CRT2), RINCAO (restaurant, 894, C000004 has 3 company rows -> ambiguous), UNIAO (market, 8,942, BA/CRT3). All expose C000004/C000025 with the 21 write-target columns and no type variation except UNIAO CEST VARCHAR(10) vs (7). One generic Agent is viable; needs company selector + CRT source of truth (C000004.CRT disagrees with EMISSOR.EMITENTE_CRT in 3 of 4) and capability detection (`scripts/sgv_capabilities.py`). Tools: `scripts/audit_multi_databases.py`, `run_multi_db_coverage.py` (in-container bulk classifier `bulk_classify_in_container.py`), `generate_multi_db_findings.py`. Artifacts: MULTI_DATABASE_{SCHEMA_AUDIT,FISCAL_COVERAGE,FIELD_COMPATIBILITY}_V1.json, MULTI_DATABASE_FINDINGS_V1.md. No auto-repair DB provided. Zero writes.
|
||||
|
||||
## Company context resolution + cache fix (2026-09-19)
|
||||
`agent/src/gti_fiscal_agent/company_context.py`: deterministic ACTIVE_COMPANY/UF/CRT resolver (EMISSOR CNPJ ↔ C000004 CNPJ ↔ emitted NFC-e key CNPJ; CRT authority for emission = EMISSOR because the emitter executables bind QrEmissor.EMITENTE_*; C000004.CRT is the ERP/SPED filial registry). Never uses first row; config `COMPANY_CODE`/`COMPANY_CNPJ`/`COMPANY_CRT_SOURCE`; preflight `agent/scripts/preflight_company_context.py`, enforced at Agent startup; `read_company_context` fails closed. 4 databases: CONSTRUCAO AUTO_RESOLVED; MMMODAS/RINCAO/UNIAO CRT_SOURCE_CONFLICT (need COMPANY_CRT_SOURCE). Rules: `docs/AGENT_COMPANY_CONTEXT_RULES_V1.md`; evidence: COMPANY_CONTEXT_RESOLUTION_V1.json, MULTI_DATABASE_COMPANY_CONTEXT_V1.json.
|
||||
Hub `classification_cache` fixed: fingerprint (tenant + input + ENGINE_VERSION + dataset_versions()) is computed first and a HIT returns without running the Classifier (test: `central/tests/test_classification_cache.py`). Live benchmark (100 products): MISS 337 ms/product, HIT 15 ms/product, 100/100 hits, identical results. Fingerprint inputs unchanged, so existing cache entries stay valid and version bumps still invalidate naturally.
|
||||
|
||||
## Release Candidate 1.0.0-rc1 (2026-09-19) — COMPLETE_LOCAL_RC
|
||||
Frozen: `release/VERSION`, `central/app/version.py` (`GET /version`), `release/RELEASE_MANIFEST.json` (versions + ACTIVE dataset hashes + migrations 0001..0010). See `docs/RELEASE_CANDIDATE_V1.md`.
|
||||
Architecture: Hub central (Docker Hub+PostgreSQL, `deploy/hub`) <-HTTPS- Agent Windows (`deploy/agent-windows`, service GTIFiscalAgent) -local read-only-> SGV Firebird. The Hub never accesses Firebird.
|
||||
Agent: `FISCAL_MODE` DRY_RUN|AUTO_APPLY (DRY_RUN = technically no writes), `COMPANY_CODE/COMPANY_CNPJ/COMPANY_CRT_SOURCE(AUTO|EMISSOR|C000004)`, `COMPANY_UF` explicit override only; single preflight `agent/scripts/preflight.py` (also enforced at service start); REPROCESS_ALL job result includes the coverage report + company context (`gti_fiscal_agent/coverage.py`, shared with multi-db tooling); `scripts/prove_zero_write.py` (DML counters + table digests). Packaging: `scripts/build_agent_package.py` (secret-scanned zip). Hub prod: `deploy/hub/docker-compose.prod.yml` (Postgres internal only), backup/restore, dataset seed export/import; `APP_ENV=production` refuses AGENT_DEV_TOKEN.
|
||||
Pilot profile: CONSTRUCAO (`pilot-construcao.env.example`); planned, not installed: MMMODAS/RINCAO/UNIAO with `COMPANY_CRT_SOURCE=EMISSOR`. No real host was accessed: a local simulation ran the Agent (registered agent, DRY_RUN) end-to-end on the CONSTRUCAO copy. Follow-up: memoize `dataset_versions()`.
|
||||
|
||||
## Local Completion V1 (2026-09)
|
||||
- Contrato: `docs/FISCAL_FIELD_CONTRACT_V1.json` (23 campos, gerado por `scripts/generate_field_contract.py`); resolvedor puro `central/app/field_contract.py`.
|
||||
- Contexto do operador (fatos que o ERP não conhece): tabela `fiscal_context_entries` (migration 0011), `POST /v1/context/import`, `scripts/import_fiscal_context.py` (CSV/JSON). Hierarquia PRODUCT > GROUP > NCM > COMPANY.
|
||||
- Escrita: somente as 5 cópias descartáveis em `docs/LOCAL_WRITE_ALLOWLIST_V1.json` (caminho exato), só C000025, via `execute_guarded_product_update`.
|
||||
- Runner: `scripts/local_completion.py` (estado em `docs/runs/local_completion/`, git-ignored); relatórios: `scripts/local_completion_report.py`.
|
||||
- Resultado: 41.354 células corrigidas, 100% verificadas, 2ª passada 0 escritas; 255.643 células aguardam resposta (`docs/LOCAL_FISCAL_CONTEXT_QUESTIONS_V1.*`, 1.054 perguntas). Fontes externas pendentes: `docs/LOCAL_COMPLETION_MANUAL_FILES_V1.md`.
|
||||
|
||||
## Local Completion V2 (2026-09-20)
|
||||
Audit outcome in `docs/LOCAL_COMPLETION_V2_REPORT.md`: origin rule invalid → rolled back (35,048); CFOP/IPPT weak-evidence writes rolled back; role no longer inferred from PROD_COZINHA alone; SPED 4.3.x dataset in `docs/sources/`; PIS/COFINS Simples rule from ERP emission evidence; questionnaire V2 (`docs/LOCAL_FISCAL_CONTEXT_QUESTIONS_V2.*`). Rollback tool: `scripts/rollback_field.py`; audit: `scripts/audit_local_writes.py`.
|
||||
Reference in New Issue
Block a user