Missão 5:
Fase 1 (Reference Data Registry): migration 0034 adds licence_terms, valid_from/
valid_to, file_size_bytes, raw_file_path, parse_errors, submitted_by/approved_by/
approved_at to the existing `datasets` table (additive only - the 9 pre-existing
dataset kinds keep their current auto-activate behavior in
central/app/datasets/service.py, completely untouched, so this changes nothing
about the currently-running engine). New central/app/reference_registry.py is a
separate, NEW ingestion workflow: register_draft() always lands as DRAFT (never
auto-activates), diff_against_vigente() summarizes a draft against the current
VIGENTE of the same kind for admin review, approve_draft()/reject_draft() are
the only way to promote a DRAFT (refuses non-DRAFT rows and refuses a draft
with recorded parse_errors). New admin screen "Tabelas oficiais"
(/admin/reference-datasets, ADMIN only, CSRF, every decision audited as
REFERENCE_DATASET_APPROVE/REJECT) lists the registry with a diff view and
approve/reject buttons. New CLI (scripts/ingest_reference_dataset.py) downloads
an official source (currently: NCM/Siscomex, no login/captcha), validates
(minimum row count, a known-sample spot check), and registers a DRAFT - --dry-run
never writes anything.
P1 (NCM) validated for real: downloaded the live official Siscomex NCM JSON
(2026-09-28, sha256 8a3bf89a..., 15,156 entries) with an identifiable User-Agent,
one request. Its hash is IDENTICAL to the NCM dataset already ACTIVE in the Hub -
confirms the currently loaded table is already current against the official
source today. Raw file kept out of git (.gitignore:
sources/raw/reference_registry/); manifest (URL, date, hash, size) recorded in
docs/REFERENCE_DATASET_MANIFEST_V1.md, including P2-P6 sources still pending URL
confirmation.
Fase 5 (feasibility studies, no code): docs/VIABILIDADE_CCG_DFE_V1.md - CCG/SEFAZ
GTIN lookup and Distribuição DF-e both require the client's own ICP-Brasil
certificate; recommendation is NOT to implement now (would require either the
Hub holding a client certificate - breaks the project's core security model - or
the Agent making new outbound internet calls, against this mission's fixed
rules). Reading the already-authorized purchase-NFe XML from disk via the Agent
(path already stored in XML_MES) is recommended as the next concrete step
instead: no certificate, no new internet call, opt-in per installation, result
always tagged DECLARADO (never auto-promoted - consensus across purchase
invoices is not fiscal truth). Open Food Facts stays identity-only, unchanged.
21 new tests, all passing. Full suite: 1147 passed, 1 skipped.
No write to any Firebird database anywhere. No Agent code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Missão 4 / Bloco E.1-E.2, on feature/cfop-ippt-write only (never main):
- Migration 0033_cfop_ippt_write_mode: Installation.cfop_ippt_write_mode
(DISABLED|DRY_RUN|APPROVAL_REQUIRED|ENABLED, DEFAULT 'DISABLED' for every
installation, existing and new) + append-only cfop_ippt_write_audits table.
- central/app/cfop_ippt_audit.py mirrors rtc_audit.py for this narrower
capability: job_authorizations(installation) (delivered inside the
authenticated job/heartbeat authorizations, never a local Agent switch) and
record_cfop_ippt_write_audits (persists applied/aborted cells; only CFOP/
IPPT are ever accepted, everything else silently ignored).
- central/app/web.py: POST /agents/{id}/cfop-ippt-write - ADMIN only (SSO
Gerenciador), CSRF, typing GRAVAR required for any mode other than
DISABLED, every change audited (CFOP_IPPT_WRITE_MODE). Small toggle UI in
agent_detail.html next to Contexto Fiscal.
- Wired into the existing job/heartbeat authorization payload and job
completion handling in central/app/api.py, alongside (never mixed with)
the RTC wiring.
- 18 new tests (central/tests/test_cfop_ippt_write_plumbing.py): default
DISABLED everywhere (job payload AND /v1/agents/me/authorizations); a
forged completion payload can never change the mode (read fresh from the
Installation row, never from Agent-supplied JSON); PADRAO role gets 403
with no audit row; ADMIN + wrong/missing CSRF is refused; ADMIN without
typing GRAVAR is refused (422); disabling never requires GRAVAR; invalid
mode is refused; the audit recorder only accepts CFOP/IPPT, silently
drops any other field (e.g. a stray CCLASSTRIB row).
- Two existing RTC tests updated for the additive authorizations-dict key
(cfop_ippt_write_mode) they now also carry.
Still NOT wired into agent/scripts/run_agent_service.py::execute_job - no
Agent code calls select_candidates/apply_candidates yet, so nothing writes
to Firebird from this branch. That wiring, an Agent version bump, a new
installer and Windows Sandbox validation are the next step, deliberately
left for after review (see docs/CFOP_IPPT_WRITE_CAPABILITY_V1.md) - this
mission explicitly asked not to publish this as an active release yet.
FULL_SUITE on this branch: 1 known, expected failure -
test_manifest_records_the_real_migration_head - because this branch adds
migration 0033 that was never merged to master/deployed; RELEASE_MANIFEST.json
intentionally still describes master/production (head 0032). Every other
test passes, including all CFOP/IPPT and RTC-adjacent tests.
REAL_PRODUCT_FISCAL_WRITES=0 (no write path is reachable from any Agent
code on any branch).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
field_contract.resolve_product was only exercised by its own unit tests. /v1/products/resolve now calls it for every product whose NCM
resolves, merging CFOP/CST/CSOSN/PIS-COFINS/CST_IPI/SITUACAO_TRIBUTARIA/IAT/IPPT decisions alongside CLASSIFICACAO_FISCAL/CEST/IPI - reusing
the same engine, the same FiscalContextEntry table the blocker-resolution workflow already writes to, no duplicated rules.
Closes three real transport/discovery gaps found while wiring it in:
- Agent heartbeat never sent company_crt (env-var UF only) or company_contribuinte_ipi, so Installation.company_crt stayed permanently empty
even though company_context.resolve() already proves both from C000004/EMISSOR for every coverage job. Resolved once, cached, retried only
while unresolved. Migration 0032 adds Installation.company_contribuinte_ipi (NULL=unknown, never coerced to "N").
- The Agent's enrichment payload only ever reported 3 old fields (CLASSIFICACAO_FISCAL/CEST/IPI) as "current", so the Hub could never see the
product's own CFOP/CST/CSOSN/... nor the classification columns (TIPO/PROD_COZINHA/PRODUTO_SERVICO/CODGRUPO) infer_role needs - separated
into a read-only CURRENT_REPORT_FIELDS list, the write allowlist (PRODUCT_ENRICHMENT_WRITE_V1) untouched.
- Added a targeted, product_code-scoped evidence query (firebird_reader.read_product_evidence: recipes/composed/purchases/stock, indexed WHERE,
never a full-table scan) so st_withheld/product_role can be proven for a single ad-hoc resolve, not only during a full sync job.
Verified read-only against the real database (D:\PROJETOS\GTISISTEMAS\Banco\GTI.FDB): CRT=1, CONTRIBUINTE_IPI=N, company_context.resolve()
reaches HIGH confidence; infer_role correctly resolved RESALE/OWN_PRODUCTION for 5/8 sample real products with zero operator questions.
1121 tests passed (tests + central/tests + agent/tests), 0 failed. Version 1.3.7 (single GTI_FISCAL_VERSION shared by Hub and Agent per
central/app/version.py's own design - never independently versioned).
- resolver (rules RTC-R0..R4/A1..A3/G1..G2) over the official cClassTrib table 2026-06-22 and LC 214/2025 annexes; local vs official catalog comparison
- ONE root-cause RTC blocker per product (missing/unresolved/ambiguous/conflict/official mismatch/catalog outdated/evidence missing)
- auto-apply: C000025.CCLASSTRIB only, three switches (FIREBIRD_WRITES + RTC_AUTO_APPLY_ENABLED + Hub authorization), per-product re-read and re-decision, never overwrites, rowcount 1, audit table rtc_write_audits (migration 0027)
- copy-only write E2E (280/280 applied, only CCLASSTRIB changed, 2nd run 0 writes, stale decision 0 writes); real databases untouched
- docs: cclasstrib_resolution, rtc_auto_apply, unified_fiscal_completeness
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Hub: installations.processing_enabled / sync_interval_seconds / last_sync_* (migration 0026).
The first heartbeat of an installation without coverage queues exactly one INITIAL_SYNC;
afterwards one SYNC_INCREMENTAL per interval (idempotent, backs off on failure). One fiscal
job at a time per Agent (FIFO, stale RUNNING recovery). Job completion now updates the
coverage snapshot, blockers and JOB_STARTED/COMPLETED/FAILED events. Panel: Sincronizar
agora / Reprocessar tudo / Pausar / Retomar + status; pause is enforced Hub-side.
Agent: local AGENT_JOBS_ENABLED is ignored (central control); SYNC_INCREMENTAL re-classifies
only new/changed products via a per-product fingerprint+decision cache and rebuilds coverage
from it; results carry operational summary and blockers; Firebird credentials come from a
DPAPI firebird.bin. Setup: read-only SGV Firebird discovery (auto-select only when
unambiguous, wizard chooser otherwise), starts the service, waits for the first heartbeat,
upgrade-in-place mode. All jobs stay read-only (FIREBIRD_WRITES=0). Version 1.1.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the native list with a combobox (name/code/CNPJ search, accent- and
case-insensitive, keyboard + mouse, custom dropdown, hidden client_id = the
Gerenciador's real CLIENTE.ID). The backend resolves client_id against the
active GTI_GERENCIADOR mirror and rejects free text/unknown ids. Adds
Tenant.external_cnpj (migration 0025) mirrored via the internal endpoint.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tenant gains source (LOCAL|GTI_GERENCIADOR), external_client_id, external_code,
active and synced_at (migration 0024). central/app/client_sync.py fetches from
the panel's new GET /internal/fiscal/clients (dedicated bearer token, never
MySQL credentials) and upserts Tenant rows, never touching a LOCAL row.
/agents/new now offers only active GTI_GERENCIADOR tenants, with a client-side
search box; the POST validates the same rule server-side, not just in the
dropdown. central/scripts/sync_clients_from_gerenciador.py is the operator-run
sync + reconciliation report.
GTI_FISCAL_VERSION/AGENT_VERSION bumped to 1.1.2 (release/VERSION single
source of truth), incorporating the installation_id persistence fix. Rebuilt
the EXE installer trio at 1.1.2 (scripts/build_windows_installer.ps1).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One universal Windows Agent package per version (agent_releases: version/
platform/architecture/sha256/status DRAFT|PUBLISHED|REVOKED; the binary
lives on a local release-storage volume, never in PostgreSQL). An operator
creates a pending installation (agent_enrollments) from Agents / Servidores
-> + Nova instalação, then either downloads it directly or generates a
one-time HTTPS link (short TTL, single use). Either path streams the
package plus a freshly minted enrollment.json whose token is never written
to storage in plaintext, only its SHA-256 hash. On the client host,
gti-agent.ps1 install-auto verifies the bundle's SHA-256, refuses a non-
HTTPS Hub, enrolls via POST /v1/agents/enroll (deleting the enrollment
token immediately after, success or failure), writes an ACL-restricted
.env with the newly issued permanent Agent token, and registers the
Windows Service without starting it (Firebird stays unconfigured on
purpose). Idempotent, replay-proof, rate-limited per IP, fully audited,
never logs a secret.
Migration 0022_agent_releases_enrollment (verified fresh-DB ==
upgrade-from-0021 by pg_dump diff). GTI_FISCAL_VERSION/AGENT_VERSION
bumped to 1.1.0 as the single source of truth across code, packages, the
release manifest and every published AgentRelease row.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Collaborators logged into GTI Gerenciador open GTI Fiscal without a
second password. The panel signs a 60 s JWT (Ed25519; the Hub holds only
the public key); the Hub checks signature, iss/aud/exp/nbf, maximum
lifetime, nonce against a one-time state cookie set when the flow starts
at GET /login (login-CSRF protection) and a single-use jti
(sso_used_assertions). Collaborators are provisioned in admin_users with
auth_source=GTI_GERENCIADOR and no password; a CHECK constraint makes a
passwordless local account or a federated account with a password
impossible (migration 0021). Sessions last FEDERATED_SESSION_SECONDS
(1 h) and are renewed silently through the panel.
The local admin stays as break-glass at /login/local, audited. Logout ends
only the Hub session. Every SSO success/rejection (stable reason code),
logout and local login is audited with the real client IP, never the
token. verify_password now fails closed on a missing hash instead of
raising.
Visual identity: gti-theme.css is a verbatim copy of the panel's
design system (tokens, sidebar, light/dark theme) plus its tema.js,
sidebar.js and logo; control-panel.css keeps only Hub components, all
colours from the panel tokens. Page structure and features unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audit of models vs fresh DB vs production (all at 0019): 20 columns are
NOT NULL in the models but nullable in the database, all timestamps with
DEFAULT now() created by 0011/0012/0014/0017 without nullable=False.
Classified DB_CONSTRAINT_MISSING: no application path writes NULL to
them (inserts omit them, updates set now; no raw SQL inserts), and there
were zero NULLs in production and in the development database (246,682
rows checked). The upgrade aborts without changing anything if any NULL
is present, instead of inventing a value.
After this revision a fresh database matches the models on nullability,
column types, PKs, FKs, uniqueness and checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0001 called Base.metadata.create_all(), building the schema from the
current models, so on a fresh PostgreSQL 0002 collided on
datasets.raw_sha256 and `alembic upgrade head` could never run from zero.
0001 is now an explicit, frozen baseline: the seven base tables exactly as
the real history created them, before 0002/0006 extended them. Databases
already past 0001 never re-run it. Proof: a fresh `upgrade head` produces a
schema identical (pg_dump --schema-only) to the long-lived dev database.
0019 repairs a latent bug found by that comparison: 0006 used
server_default="now()" as a string, which PostgreSQL froze at the instant
the column was created, so every Agent and Job got the same created_at.
It also adds three indexes the models declare but no migration created.
central/tests/test_migration_baseline.py guards (via AST) against
migrations importing the models or using string timestamp defaults.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gti
OpenCode