GTI_GERENCIADOR_FISCAL/docs/product_enrichment.md

GTI Fiscal — Product Enrichment V1 (produto novo → EAN/nome → perfil fiscal → preenchimento seguro)

Living document. Same Hub / FastAPI / PostgreSQL / Agent as the rest of GTI Fiscal: no new server, service, panel or product, no change in the SGV, no Firebird trigger/table/procedure.

Hard rules (verified by tests)

• AI_ENABLED=false, LLM_ENABLED=false, EMBEDDINGS_ENABLED=false. Everything here is deterministic.
• External cost R$ 0: only free providers; ALLOW_PAID_PROVIDERS=false (a paid provider makes build_providers raise). Agents never call providers – only the Hub.
• Per-installation functions live in the Hub (Agent detail → Funcionalidades): Corretor fiscal on/off and Enriquecimento DISABLED | DRY_RUN | ENABLED. ADMIN only, CSRF, audited (INSTALLATION_FEATURES), typed confirmation PREENCHER for ENABLED. A new installation is WAITING_FEATURE_CONFIGURATION (features_configured=false): it heartbeats but claim_next_job/schedule_automatic_sync hand it nothing and the enrichment worker does zero Firebird work, until features are chosen (at creation in Nova instalação, or later). Existing installations keep the previous behaviour (migration 0029).
• Scope = new products only. When the function is first active the Agent stores a baseline of the existing C000025.CODIGO set (local SQLite); historical products are never sent anywhere.

Flow

1. Detection (Agent, ≤ 5 s) – SELECT COUNT(*) FROM C000025 every 2 s (adaptive up to 30 s if the count is slow); code-set diff only when the count moved (+ a 60 s safety diff). No trigger. Discovery on the five lab databases (docs/PRODUCT_NEW_DETECTION_DISCOVERY_V1.json): PK CODIGO char(6), DATA_CADASTRO/ULTIMA_ALTERACAO not indexed, count 1–9 ms, full code list 8–241 ms, the ERP already owns triggers on the table. The EAN typed by the operator is in CODBARRA (fiscal GTIN is mostly empty): the first VALID of GTIN/CODBARRA is sent.
2. Resolve (Hub) POST /v1/products/resolve {product_code, gtin, description, current} → per-field decisions. Hub time budget 4.5 s; beyond it (or on quota/offline) the state is PENDING_* and the Agent retries with backoff (30 s … 30 min); results are cached, so the retry is usually free.
3. Write (Agent) typed capability PRODUCT_ENRICHMENT_WRITE_V1, allowlist CLASSIFICACAO_FISCAL, CEST, IPI; fixed statement UPDATE C000025 SET f=? WHERE CODIGO=? AND (f IS NULL OR TRIM(f)=''), one short transaction per field, guarded re-select inside the transaction, post-write verification. Before every field: fresh central authorization (fail closed). COD_ORIGEM_MERCADORIA is never written; cClassTrib stays with the existing RTC gates (the Agent only requests REPROCESS_PRODUCTS for the product; nothing is decided here).
4. Audit POST /v1/products/enrichment/applied → product_field_audits (tenant, agent, installation, code, GTIN, description snapshot, field, old/new, rule, KB status, evidence, dataset versions, authorization origin, job, result). No secrets.

Identity and states

• GTIN: 8/12/13/14 digits + check digit → VALID_GTIN | INVALID_GTIN | MISSING_GTIN (+ RESTRICTED_GTIN for store-internal prefixes 02/04/20–29). GTIN has absolute priority over the description. An invalid GTIN never reaches a provider.
• Resolution states (discrete, no scores): EXACT_GTIN_VERIFIED, EXACT_KB_NAME_VERIFIED, MULTI_SOURCE_MATCH, SINGLE_WEAK_SOURCE, AMBIGUOUS, CONFLICT, NOT_FOUND (+ operational INVALID_GTIN, PENDING_PROVIDER_QUOTA, PENDING_PROVIDERS_OFFLINE, FEATURE_DISABLED).
• Field decisions: AUTO_FILL_SAFE | SUGGESTION_ONLY | CONFLICT | INSUFFICIENT_EVIDENCE | ALREADY_SET | NOT_APPLICABLE | HUMAN_OVERRIDDEN. A field that is not resolved never blocks the others.

Knowledge base (same PostgreSQL: kb_products, kb_knowledge with per-field status)

UNVERIFIED / EVIDENCE_ONLY / VERIFIED_KB / CONFLICT / STALE. NCM becomes VERIFIED_KB only when it is valid and vigente in the official NCM dataset, has ≥ 2 independent evidence families (each provider = 1 family; client observations = 1 family and only when ≥ 2 distinct clients) and no conflicting value. Client consensus alone is never authority (measured on the lab databases: 241 GTINs shared by ≥ 2 clients, 93 with different NCMs). Company-specific decisions (CSOSN, CFOP, …) never enter the KB. Human edits never train the KB. NCM auto-fill: only VERIFIED_KB; a single non-official source is SUGGESTION_ONLY; conflicts write nothing. CEST and IPI are deterministic functions of an official NCM through the existing engine (Classifier): they are filled when the NCM is being written (AUTO) or is already present and official, only for a unique official CEST mapping / a numeric TIPI rate. Name-only: only an exact fingerprint (sorted normalized tokens, quantity/unit normalized) of a specific description (≥ 2 informative tokens + quantity, or ≥ 4) that maps to a unique VERIFIED_KB product may fill; generic names never do. ERP observations (scripts/enrichment_lab_kb.py, read-only) enter as CLIENT_OBSERVATION evidence only.

Human override protection

After writing, the Agent watches the value. Any value different from what it wrote — including a deleted value — marks the field HUMAN_OVERRIDDEN (local SQLite + reported to the Hub). The Hub then answers HUMAN_OVERRIDDEN for that field, the enrichment never rewrites it, and enrichment.is_human_overridden(state_path, code, field) is the hook every automatic writer must call. No write loops: second run = 0 writes.

Providers (Hub only; central/app/enrichment/providers.py, single interface lookup_gtin / search_description / health)

Provider	Access	Limit used	Gives
Cosmos (Bluesoft)	COSMOS_TOKEN in the Hub env (optional)	25/day	description, brand, NCM, CEST — NOT_CONFIGURED without a token (not an error)
UPCitemdb (trial)	no key	100/day, burst 15 / 30 s (headers X-RateLimit-*)	title, brand, category
OSCBR / RSC Sistemas GTIN	free plan account made by the operator; OSCBR_USER / OSCBR_PASSWORD in the Hub env; 1 h bearer renewed automatically	20 req/min	name, brand, category, NCM (CEST when present) — NOT_CONFIGURED without credentials
Open Food Facts / Open Products Facts / Open Beauty Facts	open (ODbL), identifiable User-Agent	15 req/min	title, brand, quantity
Quota is central (ProviderState, ProviderUsageDaily, `quota_kind = REMOTE_QUOTA	LOCAL_ESTIMATE	DOCUMENTED_RATE), with positive (30 d) and negative (24 h) GtinLookupCache`, single-flight per (provider, GTIN), circuit breaker
(3 failures → 60 s doubling up to 15 min; 429 → until reset; 401/403 → 1 h). Quota exhausted → PENDING_PROVIDER_QUOTA + retry after reset.
Benchmark on real lab GTINs (docs/PRODUCT_PROVIDER_BENCHMARK_V1.json): identity found rate is low for Brazilian goods (Open Food Facts 4/30, UPCitemdb rate-limited after 6 calls) and only Cosmos returns NCM. Consequently
NCM auto-fill depends on the KB (Cosmos + client observations) and the product delivers CEST/IPI immediately whenever the operator already typed a valid NCM. Cosmos without a token is not a blocker.

Official data

NCM (Siscomex JSON) is refreshed daily by central/app/datasets/updater.py (staging in memory → schema validation → plausibility ≥ 90 % of the active count → atomic promotion by DatasetService.import_payload; on any failure the last valid dataset stays). Every attempt is a source_refresh_runs row (url, time, sha256, version, count, status, error). TIPI/CEST/CFOP/CST/CSOSN/PIS-COFINS/RTC datasets are reused as they are. The dashboard shows Fontes oficiais (records, version, sha, last update, status).

Panel

Dashboard: Fontes de produtos (status, used/limit/remaining/reset, last success/error, latency, cache-hit rate, configurado: sim/não — never a token), Fontes oficiais, small Enriquecimento coverage. Agent detail: Funcionalidades (admin) and Enriquecimento (counters + product lookup with per-field provenance). Problems appear through the existing blockers (field=ENRICHMENT, PT-BR reasons).

Coverage metrics (observability.coverage_metrics)

GTIN_VALID_RATE, KB_HIT_RATE, EXTERNAL_LOOKUP_RATE, PRODUCT_FOUND_RATE, NCM_AUTO_FILL_SAFE_RATE, NAME_ONLY_RESOLUTION_RATE, AMBIGUOUS_RATE, CONFLICT_RATE, PROVIDER_QUOTA_BLOCK_RATE (+ cache stats).

Evidence

• Offline tests: central/tests/test_product_enrichment_resolver.py (12 resolution scenarios, quota/circuit/cache, provider contract fixtures), test_product_enrichment_api.py (feature gates, audit, override, blockers, multi-installation), test_official_refresh.py, agent/tests/test_enrichment.py.
• Real Firebird copy-only E2E scripts/enrichment_copy_e2e.py → docs/runs/enrichment_copy_e2e.json: detect→written P50 2.8 s / P95 3.9 s / max 4.1 s; only the 3 allowed columns; other tables changed only by the SGV's own triggers for the test rows; second run 0 writes; DRY_RUN 0 writes; WAITING/DISABLED 0 Firebird activity and no baseline; human override incl. deleted value; Hub offline → pending → recovers.
• Migration 0029: fresh-vs-upgrade pg_dump -s identical; upgrade → downgrade → upgrade clean.
• Live smoke scripts/enrichment_live_smoke.py (1 lookup/provider, real internet, separate from the suite).

Provider evaluation (2026-09-26)

• DotCompany — NOT integrated. The free query (25/day per IP, no signup) is the website demo for human use; the API (/api/v1/catalogo/gtin/{ean}) is prepaid credits (4 credits = R$ 0,04 per query, packages from R$ 120); the only free API access is a dc_test_ sandbox with test data. Automated free use is not offered, so no adapter (would violate zero cost / permitted automation).
• OSCBR / RSC Sistemas — adapter implemented, optional. Free initial plan without card, 20 req/min, token via account registration (Basic auth on /oauth/token, 1 h Bearer), GET /api/gtin/infor/{gtin} (fields ncm, nome, marca, categoria, pais). Terms are shown only inside the authenticated panel, so the operator must create the account and read them; the Hub never creates accounts. Without credentials it is NOT_CONFIGURED and makes no request.
• Cosmos stays optional (COSMOS_TOKEN). Nothing else qualified (GS1 CNP is a paid membership; Verified by GS1 is not an automation API).
• PAID_PROVIDER_CALLS=0; ALLOW_PAID_PROVIDERS=false.

Field audit (docs/PRODUCT_ENRICHMENT_FIELD_AUDIT_V1.json)

24 fields checked against field_contract; only CLASSIFICACAO_FISCAL, CEST, IPI have a deterministic, company-independent rule. All others need company/operator context (product role, ICMS treatment, origin, PIS/COFINS profile, entry scope, rounding) or description evidence (EX_TIPI) and are excluded on purpose; CST_IPI is NOT_APPLICABLE for non-IPI companies; CCLASSTRIB stays with the RTC gates.

Re-baseline after inactivity (1.3.1)

While the function is DISABLED / WAITING_FEATURE_CONFIGURATION the Agent marks a re-baseline; on reactivation the current catalogue becomes history, so a long-disabled installation never sends a burst of products to the providers.

Production KB seed strategy

Production KB starts empty (0 products, head 0028 before deploy). The lab databases are not imported into production: their observations are CLIENT_OBSERVATION evidence only and cannot verify anything alone (93/241 shared GTINs disagree on NCM). The KB grows from provider evidence on real new products; scripts/enrichment_lab_kb.py --seed-url exists for a dev/staging seed only.

Coverage V2 (1.3.2, Hub-only, migration 0030) - historical evidence + KB model

• Miner (scripts/enrichment_history_miner.py, READ-ONLY, server-side GROUP BY): C000025 master, NFCE_ITENS (NCM/CEST per document line, dated), C000062 (NF-e entry items: CLASSIFICACAO_FISCAL, CODBARRA, DESCRICAO2), C000032/C000057 (EAN seen in movements). Five databases in ~17 s, peak 28 MB Python, 7 MB of output (docs/runs/enrichment_history_miner_report.json). No XML stores exist; there is no NF-e header/item table with dates beyond the ones above.
• Evidence model (central/app/enrichment/history.py, kb.py): per (GTIN, value) one lineage per client (master flag, document count, documents per year). Master data and the fiscal documents of the same ERP are ONE family (CLIENT_DATA); each NCM-bearing provider is another family. CLIENT_DATA counts only when strong (>= 20 documents over >= 2 years, or >= 2 distinct clients) and even then reaches at most CORROBORATED.
• KB states per NCM: OBSERVED, CORROBORATED, EXTERNAL_CORROBORATED, VERIFIED_KB, CONFLICT, STALE. Temporal state per NCM: HISTORICAL_STABLE_CURRENT, HISTORICAL_CHANGED, HISTORICAL_EXPIRED, HISTORICAL_INVALID (+ HISTORICAL_CONFLICT), judged against the ACTIVE official dataset today.
• Promotion to VERIFIED_KB (rule kb-promotion-v2): exact valid GTIN + NCM vigente in the official dataset + >= 2 independent families (strong CLIENT_DATA + >= 1 NCM-bearing provider, or >= 2 providers) + zero conflicting current value. Every promotion writes kb_promotion_audits (families, clients/documents/years/providers, official dataset version+sha, rule version). Three clients agreeing NEVER verify by themselves. Two different current NCMs = CONFLICT (no majority), an expired code that is as recent as the current one blocks "stable".
• Order per product: VERIFIED_KB (zero external requests) -> local CONFLICT (zero external, no write) -> locally corroborated (only NCM-bearing providers may still be asked) -> provider cache -> provider (single-flight per GTIN) -> fiscal validation. Backfill calls can never use the operational reserve (PROVIDER_OPERATIONAL_RESERVE_PERCENT, default 20). No mass provider use for the historical seed: it is local-only.
• Revalidation: after a new official NCM dataset, kb.revalidate never deletes; codes that left the dataset become STALE (STALE_REVIEW_REQUIRED, audited when they were VERIFIED_KB).
• Aliases: descriptions seen over the years are stored as non-fiscal aliases + fingerprints (marketing words such as ORIGINAL/TRADICIONAL ignored) to improve name-only matching; name-only still fills only from a unique VERIFIED_KB product.
• Import: python -m central.scripts.import_kb_history --staging DIR [--apply] (dry-run by default, idempotent, replaces a client's lineage instead of accumulating). Tenant labels are kept; CRT/CFOP/CSOSN or any client tax attribute is never stored.
• Panel (same dashboard): provider columns FOUND_TODAY, NCM_FOUND_TODAY, CACHE_HITS, RATE_LIMITED, NEXT_RESET and a compact Knowledge Base block (known products, verified, corroborated, conflicts, KB hits today, latest conflicts with clients/documents/providers).

Providers added / evaluated in V2

• Kodebar (adapter, optional, identity only): KODEBAR_API_KEY, free "Teste" plan 50/day (+10% tolerance not used), no card, GET /gtin/lookup?gtin= with X-API-Key; terms allow integration in our own system and internal caching, forbid redistribution; 429 above quota, no overage. Returns name/brand/thumbnail, no NCM.
• OSCBR: adapter limits set to the current free plan (50/day, 5/min). NOT_CONFIGURED without OSCBR_USER/OSCBR_PASSWORD.
• UPCitemdb: after a 429 the circuit stays open >= 1 h (never insist); the observed trial limit is stricter than the documented burst.
• Tabelas Fiscais: NOT integrated at runtime. Our NCM/TIPI/CEST datasets already come from the official sources and live locally; the site is a third-party mirror, so it would add no authority. If ever needed, use its downloadable CSV/JSON (local dataset mode) instead of per-product requests.

Measured (local data only; docs/runs/enrichment_history_analysis.json)

See the V2 report in the mission summary; key findings: cross-client NCM agreement is poor (a candidate taken from the other clients matches the held-out client's own NCM only 62 % of the time, and only 42 % even when >= 2 clients corroborate it), which is exactly why client data can never verify an NCM on its own.

TECHNICAL_PLACEHOLDER (1.3.4)

The SGV refuses to save a product without NCM, so operators type a filler. An empty-only rule therefore never fires for that product. Contract:

• Per-field allowlist PRODUCT_ENRICHMENT_PLACEHOLDERS = {CLASSIFICACAO_FISCAL: {"00000000"}} (dots ignored: 0000.00.00 is the same filler). 0, 99999999, SEM NCM... are NOT placeholders (no evidence of SGV behaviour); nothing else is ever overwritten.
• A placeholder is "not classified yet", never HUMAN_OVERRIDDEN. A real NCM (even a wrong one) is preserved, whatever the KB says. cClassTrib typed by the operator is never touched (RTC gates own it).
• Where it is decided: the Hub treats a placeholder like an empty field when it builds the per-field decisions (resolver._empty, audit note technical_placeholder); the Agent must ALSO accept it in the typed write (write_field: compare-and-swap UPDATE ... WHERE TRIM(f)=<exact value just read>), so this needs Agent >= 1.3.4. Both tables are identical (parity test). The audit records old = 00000000.
• Scope reminders: products created while the function was DISABLED / WAITING stay history (re-baseline); REPROCESS_ALL/SYNC do not push products through the enrichment (only NEW products after the baseline do); a product WITHOUT EAN is resolved by name only from a unique VERIFIED_KB match.