Missão 5:
Fase 1 (Reference Data Registry): migration 0034 adds licence_terms, valid_from/
valid_to, file_size_bytes, raw_file_path, parse_errors, submitted_by/approved_by/
approved_at to the existing `datasets` table (additive only - the 9 pre-existing
dataset kinds keep their current auto-activate behavior in
central/app/datasets/service.py, completely untouched, so this changes nothing
about the currently-running engine). New central/app/reference_registry.py is a
separate, NEW ingestion workflow: register_draft() always lands as DRAFT (never
auto-activates), diff_against_vigente() summarizes a draft against the current
VIGENTE of the same kind for admin review, approve_draft()/reject_draft() are
the only way to promote a DRAFT (refuses non-DRAFT rows and refuses a draft
with recorded parse_errors). New admin screen "Tabelas oficiais"
(/admin/reference-datasets, ADMIN only, CSRF, every decision audited as
REFERENCE_DATASET_APPROVE/REJECT) lists the registry with a diff view and
approve/reject buttons. New CLI (scripts/ingest_reference_dataset.py) downloads
an official source (currently: NCM/Siscomex, no login/captcha), validates
(minimum row count, a known-sample spot check), and registers a DRAFT - --dry-run
never writes anything.
P1 (NCM) validated for real: downloaded the live official Siscomex NCM JSON
(2026-09-28, sha256 8a3bf89a..., 15,156 entries) with an identifiable User-Agent,
one request. Its hash is IDENTICAL to the NCM dataset already ACTIVE in the Hub -
confirms the currently loaded table is already current against the official
source today. Raw file kept out of git (.gitignore:
sources/raw/reference_registry/); manifest (URL, date, hash, size) recorded in
docs/REFERENCE_DATASET_MANIFEST_V1.md, including P2-P6 sources still pending URL
confirmation.
Fase 5 (feasibility studies, no code): docs/VIABILIDADE_CCG_DFE_V1.md - CCG/SEFAZ
GTIN lookup and Distribuição DF-e both require the client's own ICP-Brasil
certificate; recommendation is NOT to implement now (would require either the
Hub holding a client certificate - breaks the project's core security model - or
the Agent making new outbound internet calls, against this mission's fixed
rules). Reading the already-authorized purchase-NFe XML from disk via the Agent
(path already stored in XML_MES) is recommended as the next concrete step
instead: no certificate, no new internet call, opt-in per installation, result
always tagged DECLARADO (never auto-promoted - consensus across purchase
invoices is not fiscal truth). Open Food Facts stays identity-only, unchanged.
21 new tests, all passing. Full suite: 1147 passed, 1 skipped.
No write to any Firebird database anywhere. No Agent code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Missão 4 / Bloco E.1-E.2, on feature/cfop-ippt-write only (never main):
- Migration 0033_cfop_ippt_write_mode: Installation.cfop_ippt_write_mode
(DISABLED|DRY_RUN|APPROVAL_REQUIRED|ENABLED, DEFAULT 'DISABLED' for every
installation, existing and new) + append-only cfop_ippt_write_audits table.
- central/app/cfop_ippt_audit.py mirrors rtc_audit.py for this narrower
capability: job_authorizations(installation) (delivered inside the
authenticated job/heartbeat authorizations, never a local Agent switch) and
record_cfop_ippt_write_audits (persists applied/aborted cells; only CFOP/
IPPT are ever accepted, everything else silently ignored).
- central/app/web.py: POST /agents/{id}/cfop-ippt-write - ADMIN only (SSO
Gerenciador), CSRF, typing GRAVAR required for any mode other than
DISABLED, every change audited (CFOP_IPPT_WRITE_MODE). Small toggle UI in
agent_detail.html next to Contexto Fiscal.
- Wired into the existing job/heartbeat authorization payload and job
completion handling in central/app/api.py, alongside (never mixed with)
the RTC wiring.
- 18 new tests (central/tests/test_cfop_ippt_write_plumbing.py): default
DISABLED everywhere (job payload AND /v1/agents/me/authorizations); a
forged completion payload can never change the mode (read fresh from the
Installation row, never from Agent-supplied JSON); PADRAO role gets 403
with no audit row; ADMIN + wrong/missing CSRF is refused; ADMIN without
typing GRAVAR is refused (422); disabling never requires GRAVAR; invalid
mode is refused; the audit recorder only accepts CFOP/IPPT, silently
drops any other field (e.g. a stray CCLASSTRIB row).
- Two existing RTC tests updated for the additive authorizations-dict key
(cfop_ippt_write_mode) they now also carry.
Still NOT wired into agent/scripts/run_agent_service.py::execute_job - no
Agent code calls select_candidates/apply_candidates yet, so nothing writes
to Firebird from this branch. That wiring, an Agent version bump, a new
installer and Windows Sandbox validation are the next step, deliberately
left for after review (see docs/CFOP_IPPT_WRITE_CAPABILITY_V1.md) - this
mission explicitly asked not to publish this as an active release yet.
FULL_SUITE on this branch: 1 known, expected failure -
test_manifest_records_the_real_migration_head - because this branch adds
migration 0033 that was never merged to master/deployed; RELEASE_MANIFEST.json
intentionally still describes master/production (head 0032). Every other
test passes, including all CFOP/IPPT and RTC-adjacent tests.
REAL_PRODUCT_FISCAL_WRITES=0 (no write path is reachable from any Agent
code on any branch).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Missão 4 / Bloco A.1 (prioridade máxima): central.app.field_contract.infer_role
treated ANY C000032.TIPO=COMPOSICAO row for a product as proof of own
production, with no ratio check - a beer sold 19 times and used once as a
recipe component (Agent 2, real data) was classified OWN_PRODUCTION from
that single occurrence, proposing to change its CFOP and IPPT.
New deterministic threshold in field_contract.py: COMPOSICAO must be a
majority (>=70%) of that product's own C000032 movement history, with at
least 5 total events, to count as own-production evidence. Below 30% the
evidence is discarded entirely (falls through to other role evidence,
e.g. TIPO=VENDA -> RESALE). Between 30% and 70% (with enough events to be
meaningful) is genuinely conflicting evidence -> UNKNOWN, never guessed
either way. RECEITA_INGREDIENTES/PROD_COZINHA-based evidence (a direct
recipe-table linkage, not a statistical ratio) is untouched.
agent/src/gti_fiscal_agent/firebird_reader.py now counts COMPOSICAO vs other
C000032 rows per product (read_product_evidence: single indexed query;
read_evidence: one grouped query for the whole catalog) into a new
ev["composed_ratio"] key, alongside the existing "composed" set (kept for
other, non-role consumers like coverage.py). central/app/enrichment/api.py's
sanitize_product_evidence and fiscal_completion.py's empty-ev fallback both
carry the new key through.
New tests (central/tests/test_field_contract.py) cover the 5 real cases named
in the mission: the beer (1 vs 19, 5% - discarded), a second beer (3 vs 4,
42.9% - MIXED/UNKNOWN), the liqueur (34 vs 22, 60.7% - still MIXED/UNKNOWN,
below the 70% bar), the juice with PROD_COZINHA=1+RECEITA_INGREDIENTES
(unaffected, stays OWN_PRODUCTION), and the tilapia/corvina fish (no
composed evidence at all, stays RESALE) - plus a strong 90% majority case
and an insufficient-sample (<5 events) case. A new end-to-end test in
agent/tests/test_cfop_ippt_auto_apply.py (committed separately, see the
feature/cfop-ippt-write branch) proves the weak-evidence beer never reaches
select_candidates as a write candidate, independent of any write flag.
Re-running the dry-run over Agent 2's real database (GTI.FDB, 527 products)
after this fix: AUTO_FILL_SAFE CFOP/IPPT cells drop from 12 to 3 (the two
fish CFOP fills + the juice's IPPT, all with strong, non-statistical
evidence) - docs/runs/GTI_dryrun_universe_summary_after_A1.json. This
corrects the Missão 3 report's 12/9(downgraded)/4(sustained) manual
judgment call with a deterministic, tested engine rule.
No schema/migration change. Zero Firebird writes (read-only analysis only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Hub-only version bump for the static dossie_contador.md anonymization fix
and the Bloco B PIS/COFINS-source evidence update committed just before.
No schema/migration change, no Agent file touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Missao 2 (fechamento de lacunas fiscais, read-only + UI/engine gated):
- field_contract: pis_cofins_profile now also accepts the literal sentinels
NOT_APPLICABLE/OPERATION_TIME, resolving to NA only from an audited
FiscalContextEntry - never inferred from emission statistics. CFOP_ENTRADA
under an already-decided entry_scope=MIXED_OR_DYNAMIC now resolves to NA
instead of MANUAL, so it stops blocking product completeness and stops
re-asking an already-answered question. Regression tests prove: no ADMIN
decision => unchanged old behavior; decision present => gated bucket, no write.
- context_keys: validate_entry accepts the two sentinels for pis_cofins_profile.
- Contexto Fiscal UI: PIS/COFINS and CFOP_ENTRADA cards show the Bloco 2
cross-database evidence (5 installations, CRT 1 and 3: emission never reads
the PIS/COFINS cadastral fields anywhere) and a link to an ADMIN-facing,
anonymized dossie-for-the-accountant; entry_scope is now a dropdown.
- release/tests: decoupled the Hub's GTI_FISCAL_VERSION from the Agent's own
version (test_versions_are_coherent_across_hub_and_packages vs the new
test_agent_version_is_internally_coherent) - a Hub-only release must not
require touching, rebuilding or republishing the Agent.
- New read-only evidence: docs/runs/GTI_dryrun_universe_summary.json (527
products, Agent 2's real database), docs/runs/BLOCO2_PIS_COFINS_CROSS_DB_V1.json
(5 installations), docs/fiscal_completeness_matrix.md, docs/dossie_contador.md
(internal, real names) + central/static/docs/dossie_contador.md (public,
anonymized - no CNPJ/razao social/product names), docs/AGENT2_UPGRADE_RUNBOOK_V1.md.
- CLAUDE.md: recorded the permanent fiscal-mission rules (no AI/scraping, no
Firebird schema changes, deploy only via deploy_hub.ps1, never reenroll,
no remote access to client Agent hosts, no client data in unauthenticated
static routes).
No schema/migration change. Zero Firebird writes (new scripts never open a
write connection). Hub only - Agent code untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an ADMIN-only "Contexto Fiscal" section to the Agent detail page: confirm/alter/clear the fiscal decisions
field_contract.resolve_product genuinely cannot derive from the ERP alone (icms_treatment, pis_cofins_profile,
entry_scope). Persists through the existing FiscalContextEntry table (the same one blocker_resolution already
writes operator answers into) - no new persistence model, no opaque JSON box. CSRF + RBAC + audit reuse the exact
mechanism already proven for /agents/{id}/features. Confirming or clearing flags every already-enriched product on
the installation for re-evaluation.
The list was reduced from 5 candidates to 3 by auditing real production data (Agent 2, 2026-09-27/28):
icms_rate/credit_rate and rounding were dropped - both proven NOT_APPLICABLE / ALREADY_DERIVED for this CRT=1
company from real field_contract dry-runs, not assumption. Real evidence surfaced in the UI is a dated, one-time
manual read-only measurement (disclosed in the card copy itself as such) - the Hub has no direct Firebird access,
so this is not a live query; a follow-up should pipe it through the heartbeat the same way CRT/CONTRIBUINTE_IPI
were wired.
Also found, during the same audit, that this client's NFC-e emission ignores the product's cadastral PIS/COFINS
CST/rate fields entirely (verified on the 3 real products that DO have them populated - the emitted documents never
carry the cadastral value) - documented as a strong OPEN recommendation for a future field_contract change, not
applied here, since changing canonical fiscal semantics needs accountant sign-off, not just my own inference.
1129 tests passed (tests + central/tests + agent/tests), 0 failed. Version 1.3.9 (single GTI_FISCAL_VERSION). Hub
only - Agent unchanged, no new installer this release.
REPROCESS_PRODUCTS is not in SYNC_JOB_KINDS, so it fell to the else branch of execute_job and hardcoded company=None
for _unified_fiscal - a real gap, not a missing feature: the same deterministic, fail-closed company_context.resolve()
every sync job already uses is cheap (no full-table scan, confirmed against the real database) and applies just as
well to a one-off reprocess job. Now resolves the real company the same way; falls back to None only when the
company genuinely cannot be resolved yet, never a silent guess. Does not touch RTC rule logic.
Verified end-to-end against the real production database (dry-run, zero writes) that this was the last domain still
carrying the old bug - field_contract's post-NCM fiscal completion already bypassed it entirely by reading
Installation.company_uf/crt/company_contribuinte_ipi directly (populated via the 1.3.7 heartbeat fix).
1123 tests passed (tests + central/tests + agent/tests), 0 failed. Version 1.3.8 (single GTI_FISCAL_VERSION).
field_contract.resolve_product was only exercised by its own unit tests. /v1/products/resolve now calls it for every product whose NCM
resolves, merging CFOP/CST/CSOSN/PIS-COFINS/CST_IPI/SITUACAO_TRIBUTARIA/IAT/IPPT decisions alongside CLASSIFICACAO_FISCAL/CEST/IPI - reusing
the same engine, the same FiscalContextEntry table the blocker-resolution workflow already writes to, no duplicated rules.
Closes three real transport/discovery gaps found while wiring it in:
- Agent heartbeat never sent company_crt (env-var UF only) or company_contribuinte_ipi, so Installation.company_crt stayed permanently empty
even though company_context.resolve() already proves both from C000004/EMISSOR for every coverage job. Resolved once, cached, retried only
while unresolved. Migration 0032 adds Installation.company_contribuinte_ipi (NULL=unknown, never coerced to "N").
- The Agent's enrichment payload only ever reported 3 old fields (CLASSIFICACAO_FISCAL/CEST/IPI) as "current", so the Hub could never see the
product's own CFOP/CST/CSOSN/... nor the classification columns (TIPO/PROD_COZINHA/PRODUTO_SERVICO/CODGRUPO) infer_role needs - separated
into a read-only CURRENT_REPORT_FIELDS list, the write allowlist (PRODUCT_ENRICHMENT_WRITE_V1) untouched.
- Added a targeted, product_code-scoped evidence query (firebird_reader.read_product_evidence: recipes/composed/purchases/stock, indexed WHERE,
never a full-table scan) so st_withheld/product_role can be proven for a single ad-hoc resolve, not only during a full sync job.
Verified read-only against the real database (D:\PROJETOS\GTISISTEMAS\Banco\GTI.FDB): CRT=1, CONTRIBUINTE_IPI=N, company_context.resolve()
reaches HIGH confidence; infer_role correctly resolved RESALE/OWN_PRODUCTION for 5/8 sample real products with zero operator questions.
1121 tests passed (tests + central/tests + agent/tests), 0 failed. Version 1.3.7 (single GTI_FISCAL_VERSION shared by Hub and Agent per
central/app/version.py's own design - never independently versioned).
Sandbox test: a 26 s outage exhausted the previous 15 s retry budget and failed a REPROCESS_ALL.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by uninstall + fresh install on the clean Windows Sandbox: the old agent_state.sqlite identity conflicted with the new enrollment and the service sat idle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found in the clean-Windows recovery test: a single TLS handshake failure mid-job failed a whole sync.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- resolver (rules RTC-R0..R4/A1..A3/G1..G2) over the official cClassTrib table 2026-06-22 and LC 214/2025 annexes; local vs official catalog comparison
- ONE root-cause RTC blocker per product (missing/unresolved/ambiguous/conflict/official mismatch/catalog outdated/evidence missing)
- auto-apply: C000025.CCLASSTRIB only, three switches (FIREBIRD_WRITES + RTC_AUTO_APPLY_ENABLED + Hub authorization), per-product re-read and re-decision, never overwrites, rowcount 1, audit table rtc_write_audits (migration 0027)
- copy-only write E2E (280/280 applied, only CCLASSTRIB changed, 2nd run 0 writes, stale decision 0 writes); real databases untouched
- docs: cclasstrib_resolution, rtc_auto_apply, unified_fiscal_completeness
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- RTC: Agent reads ALL PARAMETROS rows every sync (conflict => off), reports rtc_source/rtc_observed_at/rtc_param_rows; legal_effective_context is informational only
- PIS/COFINS: applicability from CRT + PISCOFINS flag + incidence only (never from empty fields); Simples w/o special treatment NOT_APPLICABLE, taxed regime with explicit N OPTIONAL, unreadable declaration NEEDS_REVIEW
- CST: one digit with an official zero-padded form is VALID_AFTER_NORMALIZATION (non-blocking warning), nothing is written to the ERP
- tests: RTC toggle 0->1->0 across syncs (Agent + Hub stale closing), non-circularity, normalization, panel provenance
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- central/app/unified_fiscal_coverage.py: pure rules (PIS/COFINS NOT_APPLICABLE/REQUIRED/OPTIONAL/NEEDS_REVIEW, RTC only when the ERP switch is on)
- agent: read-only loader (unified_fiscal.py) replaces the legacy/expanded module; findings feed the single operational state, coverage and blocker queue
- hub: BLOCKED_FISCAL_DATA blockers (PIS/COFINS/RTC) share the queue; stale ones close on the next full pass; dashboard breakdown gains CEST/PIS/COFINS/RTC
- panel: legacy/expanded coverage block removed; discreet 'Reforma Tributaria: Ativa/Desativada'
- docs: unified_fiscal_completeness, pis_cofins_applicability, rtc_pis_cofins_model + official source manifests; read-only E2E script
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gti
