GTI_GERENCIADOR_FISCAL/docs/product_enrichment.md

# GTI Fiscal — Product Enrichment V1 (produto novo → EAN/nome → perfil fiscal → preenchimento seguro)

Living document. Same Hub / FastAPI / PostgreSQL / Agent as the rest of GTI Fiscal: **no new server, service, panel or product**, no change in the SGV, no Firebird trigger/table/procedure.

## Hard rules (verified by tests)
- `AI_ENABLED=false`, `LLM_ENABLED=false`, `EMBEDDINGS_ENABLED=false`. Everything here is deterministic.
- External cost R$ 0: only free providers; `ALLOW_PAID_PROVIDERS=false` (a paid provider makes `build_providers` raise). **Agents never call providers – only the Hub.**
- Per-installation functions live in the Hub (Agent detail → *Funcionalidades*): **Corretor fiscal** on/off and **Enriquecimento** `DISABLED | DRY_RUN | ENABLED`. ADMIN only, CSRF, audited (`INSTALLATION_FEATURES`),
  typed confirmation `PREENCHER` for ENABLED. A **new** installation is `WAITING_FEATURE_CONFIGURATION` (`features_configured=false`): it heartbeats but `claim_next_job`/`schedule_automatic_sync`
  hand it nothing and the enrichment worker does zero Firebird work, until features are chosen (at creation in *Nova instalação*, or later). Existing installations keep the previous behaviour (migration 0029).
- Scope = **new products only**. When the function is first active the Agent stores a baseline of the existing `C000025.CODIGO` set (local SQLite); historical products are never sent anywhere.

## Flow
1. **Detection (Agent, ≤ 5 s)** – `SELECT COUNT(*) FROM C000025` every 2 s (adaptive up to 30 s if the count is slow); code-set diff only when the count moved (+ a 60 s safety diff). No trigger. Discovery on the five
   lab databases (`docs/PRODUCT_NEW_DETECTION_DISCOVERY_V1.json`): PK `CODIGO` char(6), `DATA_CADASTRO/ULTIMA_ALTERACAO` **not indexed**, count 1–9 ms, full code list 8–241 ms, the ERP already owns triggers on the table.
   The EAN typed by the operator is in **`CODBARRA`** (fiscal `GTIN` is mostly empty): the first VALID of `GTIN`/`CODBARRA` is sent.
2. **Resolve (Hub)** `POST /v1/products/resolve {product_code, gtin, description, current}` → per-field decisions. Hub time budget 4.5 s; beyond it (or on quota/offline) the state is `PENDING_*` and the Agent retries with backoff (30 s … 30 min); results are cached, so the retry is usually free.
3. **Write (Agent)** typed capability `PRODUCT_ENRICHMENT_WRITE_V1`, allowlist `CLASSIFICACAO_FISCAL, CEST, IPI`; fixed statement `UPDATE C000025 SET f=? WHERE CODIGO=? AND (f IS NULL OR TRIM(f)='')`, one short transaction per field,
   guarded re-select inside the transaction, post-write verification. Before every field: fresh central authorization (fail closed). `COD_ORIGEM_MERCADORIA` is never written; cClassTrib stays with the existing RTC gates
   (the Agent only requests `REPROCESS_PRODUCTS` for the product; nothing is decided here).
4. **Audit** `POST /v1/products/enrichment/applied` → `product_field_audits` (tenant, agent, installation, code, GTIN, description snapshot, field, old/new, rule, KB status, evidence, dataset versions, authorization origin, job, result). No secrets.

## Identity and states
- GTIN: 8/12/13/14 digits + check digit → `VALID_GTIN | INVALID_GTIN | MISSING_GTIN` (+ `RESTRICTED_GTIN` for store-internal prefixes 02/04/20–29). **GTIN has absolute priority over the description.** An invalid GTIN never reaches a provider.
- Resolution states (discrete, no scores): `EXACT_GTIN_VERIFIED, EXACT_KB_NAME_VERIFIED, MULTI_SOURCE_MATCH, SINGLE_WEAK_SOURCE, AMBIGUOUS, CONFLICT, NOT_FOUND` (+ operational `INVALID_GTIN, PENDING_PROVIDER_QUOTA, PENDING_PROVIDERS_OFFLINE, FEATURE_DISABLED`).
- Field decisions: `AUTO_FILL_SAFE | SUGGESTION_ONLY | CONFLICT | INSUFFICIENT_EVIDENCE | ALREADY_SET | NOT_APPLICABLE | HUMAN_OVERRIDDEN`. A field that is not resolved never blocks the others.

## Knowledge base (same PostgreSQL: `kb_products`, `kb_knowledge` with per-field `status`)
`UNVERIFIED / EVIDENCE_ONLY / VERIFIED_KB / CONFLICT / STALE`. NCM becomes `VERIFIED_KB` only when it is **valid and vigente in the official NCM dataset**, has **≥ 2 independent evidence families** (each provider = 1 family; client observations = 1 family and only when ≥ 2 distinct clients)
and **no conflicting value**. Client consensus alone is never authority (measured on the lab databases: 241 GTINs shared by ≥ 2 clients, **93 with different NCMs**). Company-specific decisions (CSOSN, CFOP, …) never enter the KB.
**Human edits never train the KB.** `NCM` auto-fill: only `VERIFIED_KB`; a single non-official source is `SUGGESTION_ONLY`; conflicts write nothing.
CEST and IPI are deterministic functions of an official NCM through the existing engine (`Classifier`): they are filled when the NCM is being written (AUTO) **or is already present and official**, only for a unique official CEST mapping / a numeric TIPI rate.
Name-only: only an exact fingerprint (sorted normalized tokens, quantity/unit normalized) of a **specific** description (≥ 2 informative tokens + quantity, or ≥ 4) that maps to a unique `VERIFIED_KB` product may fill; generic names never do.
ERP observations (`scripts/enrichment_lab_kb.py`, read-only) enter as `CLIENT_OBSERVATION` evidence only.

## Human override protection
After writing, the Agent watches the value. Any value different from what it wrote — **including a deleted value** — marks the field `HUMAN_OVERRIDDEN` (local SQLite + reported to the Hub). The Hub then answers `HUMAN_OVERRIDDEN` for that field, the enrichment never rewrites it,
and `enrichment.is_human_overridden(state_path, code, field)` is the hook every automatic writer must call. No write loops: second run = 0 writes.

## Providers (Hub only; `central/app/enrichment/providers.py`, single interface `lookup_gtin / search_description / health`)
| Provider | Access | Limit used | Gives |
|---|---|---|---|
| Cosmos (Bluesoft) | `COSMOS_TOKEN` in the Hub env (optional) | 25/day | description, brand, **NCM, CEST** — `NOT_CONFIGURED` without a token (not an error) |
| UPCitemdb (trial) | no key | 100/day, burst 15 / 30 s (headers `X-RateLimit-*`) | title, brand, category |
| OSCBR / RSC Sistemas GTIN | free plan account made by the operator; `OSCBR_USER` / `OSCBR_PASSWORD` in the Hub env; 1 h bearer renewed automatically | 20 req/min | name, brand, category, **NCM** (CEST when present) — `NOT_CONFIGURED` without credentials |
| Open Food Facts / Open Products Facts / Open Beauty Facts | open (ODbL), identifiable `User-Agent` | 15 req/min | title, brand, quantity |
Quota is central (`ProviderState`, `ProviderUsageDaily`, `quota_kind = REMOTE_QUOTA | LOCAL_ESTIMATE | DOCUMENTED_RATE`), with positive (30 d) and negative (24 h) `GtinLookupCache`, single-flight per (provider, GTIN), circuit breaker
(3 failures → 60 s doubling up to 15 min; 429 → until reset; 401/403 → 1 h). Quota exhausted → `PENDING_PROVIDER_QUOTA` + retry after reset.
**Benchmark on real lab GTINs** (`docs/PRODUCT_PROVIDER_BENCHMARK_V1.json`): identity found rate is low for Brazilian goods (Open Food Facts 4/30, UPCitemdb rate-limited after 6 calls) and **only Cosmos returns NCM**. Consequently
NCM auto-fill depends on the KB (Cosmos + client observations) and the product delivers CEST/IPI immediately whenever the operator already typed a valid NCM. Cosmos without a token is not a blocker.

## Official data
`NCM` (Siscomex JSON) is refreshed daily by `central/app/datasets/updater.py` (staging in memory → schema validation → plausibility ≥ 90 % of the active count → atomic promotion by `DatasetService.import_payload`; on any failure the last valid dataset stays).
Every attempt is a `source_refresh_runs` row (url, time, sha256, version, count, status, error). TIPI/CEST/CFOP/CST/CSOSN/PIS-COFINS/RTC datasets are reused as they are. The dashboard shows *Fontes oficiais* (records, version, sha, last update, status).

## Panel
Dashboard: *Fontes de produtos* (status, used/limit/remaining/reset, last success/error, latency, cache-hit rate, `configurado: sim/não` — never a token), *Fontes oficiais*, small *Enriquecimento* coverage.
Agent detail: *Funcionalidades* (admin) and *Enriquecimento* (counters + product lookup with per-field provenance). Problems appear through the existing blockers (`field=ENRICHMENT`, PT-BR reasons).

## Coverage metrics (`observability.coverage_metrics`)
`GTIN_VALID_RATE, KB_HIT_RATE, EXTERNAL_LOOKUP_RATE, PRODUCT_FOUND_RATE, NCM_AUTO_FILL_SAFE_RATE, NAME_ONLY_RESOLUTION_RATE, AMBIGUOUS_RATE, CONFLICT_RATE, PROVIDER_QUOTA_BLOCK_RATE` (+ cache stats).

## Evidence
- Offline tests: `central/tests/test_product_enrichment_resolver.py` (12 resolution scenarios, quota/circuit/cache, provider contract fixtures), `test_product_enrichment_api.py` (feature gates, audit, override, blockers, multi-installation), `test_official_refresh.py`, `agent/tests/test_enrichment.py`.
- Real Firebird **copy-only** E2E `scripts/enrichment_copy_e2e.py` → `docs/runs/enrichment_copy_e2e.json`: detect→written P50 2.8 s / P95 3.9 s / max 4.1 s; only the 3 allowed columns; other tables changed only by the SGV's own triggers for the test rows;
  second run 0 writes; DRY_RUN 0 writes; WAITING/DISABLED 0 Firebird activity and no baseline; human override incl. deleted value; Hub offline → pending → recovers.
- Migration 0029: fresh-vs-upgrade `pg_dump -s` identical; upgrade → downgrade → upgrade clean.
- Live smoke `scripts/enrichment_live_smoke.py` (1 lookup/provider, real internet, separate from the suite).


## Provider evaluation (2026-09-26)
- **DotCompany — NOT integrated.** The free query (25/day per IP, no signup) is the *website* demo for human use; the API (`/api/v1/catalogo/gtin/{ean}`) is prepaid credits (4 credits = R$ 0,04 per query, packages from R$ 120); the only free API access is a `dc_test_` sandbox with test data. Automated free use is not offered, so no adapter (would violate zero cost / permitted automation).
- **OSCBR / RSC Sistemas — adapter implemented, optional.** Free initial plan without card, 20 req/min, token via account registration (Basic auth on `/oauth/token`, 1 h Bearer), `GET /api/gtin/infor/{gtin}` (fields `ncm`, `nome`, `marca`, `categoria`, `pais`). Terms are shown only inside the authenticated panel, so the operator must create the account and read them; the Hub never creates accounts. Without credentials it is `NOT_CONFIGURED` and makes no request.
- Cosmos stays optional (`COSMOS_TOKEN`). Nothing else qualified (GS1 CNP is a paid membership; Verified by GS1 is not an automation API).
- `PAID_PROVIDER_CALLS=0`; `ALLOW_PAID_PROVIDERS=false`.

## Field audit (`docs/PRODUCT_ENRICHMENT_FIELD_AUDIT_V1.json`)
24 fields checked against `field_contract`; only `CLASSIFICACAO_FISCAL`, `CEST`, `IPI` have a deterministic, company-independent rule. All others need company/operator context (product role, ICMS treatment, origin, PIS/COFINS profile, entry scope, rounding) or description evidence (EX_TIPI) and are excluded on purpose; `CST_IPI` is NOT_APPLICABLE for non-IPI companies; `CCLASSTRIB` stays with the RTC gates.

## Re-baseline after inactivity (1.3.1)
While the function is DISABLED / WAITING_FEATURE_CONFIGURATION the Agent marks a re-baseline; on reactivation the current catalogue becomes history, so a long-disabled installation never sends a burst of products to the providers.

## Production KB seed strategy
Production KB starts empty (0 products, head 0028 before deploy). The lab databases are **not** imported into production: their observations are `CLIENT_OBSERVATION` evidence only and cannot verify anything alone (93/241 shared GTINs disagree on NCM). The KB grows from provider evidence on real new products; `scripts/enrichment_lab_kb.py --seed-url` exists for a dev/staging seed only.


## Coverage V2 (1.3.2, Hub-only, migration 0030) - historical evidence + KB model
- **Miner** (`scripts/enrichment_history_miner.py`, READ-ONLY, server-side `GROUP BY`): C000025 master, `NFCE_ITENS` (NCM/CEST per document line, dated), `C000062` (NF-e entry items: `CLASSIFICACAO_FISCAL`, `CODBARRA`, `DESCRICAO2`), `C000032`/`C000057` (EAN seen in movements).
  Five databases in ~17 s, peak 28 MB Python, 7 MB of output (`docs/runs/enrichment_history_miner_report.json`). No XML stores exist; there is no NF-e header/item table with dates beyond the ones above.
- **Evidence model** (`central/app/enrichment/history.py`, `kb.py`): per (GTIN, value) one *lineage* per client (master flag, document count, documents per year). Master data and the fiscal documents of the same ERP are ONE family (`CLIENT_DATA`); each NCM-bearing provider is another family.
  `CLIENT_DATA` counts only when strong (>= 20 documents over >= 2 years, or >= 2 distinct clients) and even then reaches at most `CORROBORATED`.
- **KB states** per NCM: `OBSERVED, CORROBORATED, EXTERNAL_CORROBORATED, VERIFIED_KB, CONFLICT, STALE`. Temporal state per NCM: `HISTORICAL_STABLE_CURRENT, HISTORICAL_CHANGED, HISTORICAL_EXPIRED, HISTORICAL_INVALID` (+ `HISTORICAL_CONFLICT`), judged against the ACTIVE official dataset today.
- **Promotion to VERIFIED_KB** (rule `kb-promotion-v2`): exact valid GTIN + NCM vigente in the official dataset + >= 2 independent families (strong `CLIENT_DATA` + >= 1 NCM-bearing provider, or >= 2 providers) + zero conflicting current value. Every promotion writes `kb_promotion_audits`
  (families, clients/documents/years/providers, official dataset version+sha, rule version). Three clients agreeing NEVER verify by themselves. Two different current NCMs = `CONFLICT` (no majority), an expired code that is as recent as the current one blocks "stable".
- **Order per product**: VERIFIED_KB (zero external requests) -> local CONFLICT (zero external, no write) -> locally corroborated (only NCM-bearing providers may still be asked) -> provider cache -> provider (single-flight per GTIN) -> fiscal validation.
  Backfill calls can never use the operational reserve (`PROVIDER_OPERATIONAL_RESERVE_PERCENT`, default 20). No mass provider use for the historical seed: it is local-only.
- **Revalidation**: after a new official NCM dataset, `kb.revalidate` never deletes; codes that left the dataset become `STALE` (`STALE_REVIEW_REQUIRED`, audited when they were VERIFIED_KB).
- **Aliases**: descriptions seen over the years are stored as non-fiscal aliases + fingerprints (marketing words such as ORIGINAL/TRADICIONAL ignored) to improve name-only matching; name-only still fills only from a unique VERIFIED_KB product.
- **Import**: `python -m central.scripts.import_kb_history --staging DIR [--apply]` (dry-run by default, idempotent, replaces a client's lineage instead of accumulating). Tenant labels are kept; CRT/CFOP/CSOSN or any client tax attribute is never stored.
- **Panel** (same dashboard): provider columns `FOUND_TODAY, NCM_FOUND_TODAY, CACHE_HITS, RATE_LIMITED, NEXT_RESET` and a compact Knowledge Base block (known products, verified, corroborated, conflicts, KB hits today, latest conflicts with clients/documents/providers).

### Providers added / evaluated in V2
- **Kodebar** (adapter, optional, identity only): `KODEBAR_API_KEY`, free "Teste" plan 50/day (+10% tolerance not used), no card, `GET /gtin/lookup?gtin=` with `X-API-Key`; terms allow integration in our own system and internal caching, forbid redistribution; 429 above quota, no overage. Returns name/brand/thumbnail, **no NCM**.
- **OSCBR**: adapter limits set to the current free plan (50/day, 5/min). `NOT_CONFIGURED` without `OSCBR_USER/OSCBR_PASSWORD`.
- **UPCitemdb**: after a 429 the circuit stays open >= 1 h (never insist); the observed trial limit is stricter than the documented burst.
- **Tabelas Fiscais**: NOT integrated at runtime. Our NCM/TIPI/CEST datasets already come from the official sources and live locally; the site is a third-party mirror, so it would add no authority. If ever needed, use its downloadable CSV/JSON (local dataset mode) instead of per-product requests.

### Measured (local data only; `docs/runs/enrichment_history_analysis.json`)
See the V2 report in the mission summary; key findings: cross-client NCM agreement is poor (a candidate taken from the other clients matches the held-out client's own NCM only 62 % of the time, and only 42 % even when >= 2 clients corroborate it), which is exactly why client data can never verify an NCM on its own.


## TECHNICAL_PLACEHOLDER (1.3.4)
The SGV refuses to save a product without NCM, so operators type a filler. An empty-only rule therefore never fires for that product. Contract:
- Per-field allowlist `PRODUCT_ENRICHMENT_PLACEHOLDERS = {CLASSIFICACAO_FISCAL: {"00000000"}}` (dots ignored: `0000.00.00` is the same filler). `0`, `99999999`, `SEM NCM`... are NOT placeholders (no evidence of SGV behaviour); nothing else is ever overwritten.
- A placeholder is "not classified yet", never `HUMAN_OVERRIDDEN`. A real NCM (even a wrong one) is preserved, whatever the KB says. cClassTrib typed by the operator is never touched (RTC gates own it).
- Where it is decided: the Hub treats a placeholder like an empty field when it builds the per-field decisions (`resolver._empty`, audit note `technical_placeholder`); the Agent must ALSO accept it in the typed write (`write_field`: compare-and-swap `UPDATE ... WHERE TRIM(f)=<exact value just read>`), so this needs Agent >= 1.3.4.
  Both tables are identical (parity test). The audit records `old = 00000000`.
- Scope reminders: products created while the function was DISABLED / WAITING stay history (re-baseline); REPROCESS_ALL/SYNC do not push products through the enrichment (only NEW products after the baseline do);
  a product WITHOUT EAN is resolved by name only from a unique VERIFIED_KB match.


## Product-visible pipeline, re-evaluation and operator decision (1.3.5, migration 0031)
- **No silent product**: every tracked product has `ui_status` + `reason_code` (Agent detail, section *Funcionalidades*): ENRIQUECIDO, PRONTO_PARA_PREENCHER, SIMULADO, AGUARDANDO_EVIDENCIA, EAN_AUSENTE, EAN_INVALIDO, NOME_AMBIGUO, SEM_MATCH_DE_PRODUTO,
  NCM_NAO_VERIFICADO, CONFLITO_DE_NCM, PROVIDER_SEM_QUOTA, AGUARDANDO_RETRY, HUMAN_OVERRIDE, TECHNICAL_PLACEHOLDER (note), WRITE_FAILED, VERIFICACAO_POS_WRITE_FALHOU, NADA_A_PREENCHER, with EAN, detection time, KB status, candidates, last attempt and next retry.
- **Re-evaluation** (`product_enrichments.reevaluate`): set by the panel (*Reavaliar* / *Reavaliar todos os pendentes*), by **Reprocessar / Sincronizar** (`create_job`, even when an equivalent job is already queued) and by a KB decision; the Agent polls
  `GET /v1/products/enrichment/pending` every 10 s and re-runs the normal resolve for those tracked, not-yet-enriched products (KB/cache first; the Hub decides about providers; enriched / human-overridden products are never flagged).
  Price edits are NOT a new product; *Reavaliar* is the explicit new evaluation. REPROCESS_ALL does not push the whole catalogue through the enrichment: only the tracked unresolved set.
- **Stale-write guard** (Agent): a write is refused (`STALE_CONFLICT`) when the field no longer holds the value the Hub decided on (a person typed something meanwhile).
- **Name-only** (`kb.alias_match`): the query token set equals an alias token set, or one contains the other and the difference is only neutral category nouns (REFRIGERANTE, BEBIDA...). ZERO / DIET / a flavour / another size are never neutral, so
  "COCA COLA LATA 350 ML" = "REFRIGERANTE COCA COLA LATA 350 ML" but != "... ZERO ...". Writing by name still requires ONE matching KB product whose NCM is VERIFIED_KB.
- **Operator decision (ADMIN)**: `POST /kb/attest` (dashboard, *Base de conhecimento*: search by EAN or name, official NCM text next to each candidate, button *Confirmar este NCM*). Only an existing, vigente candidate can be chosen; the others become `STALE (REJECTED_BY_OPERATOR)`;
  the family `OPERATOR_ATTESTATION` + the local evidence of that value = VERIFIED_KB with a promotion audit naming the operator; every affected tracked product is flagged for re-evaluation, so the write happens automatically right after the click.
- **RTC != Enrichment**: distinct labels on the Agent detail and a test that each switch never changes the other.
- **Supplier NF-e**: investigated, no independent supplier evidence exists in the five databases (entry-item tables C000022/C000075 are empty; C000061/C000062/NFCE are documents ISSUED by the client from its own master); no `SUPPLIER_NFE` family was created.
- **Free NCM sources re-checked (2026-09-26)**: Kodebar returns NCM only for medicines (CMED) and needs a key; DotCompany API is prepaid; Pesquisa GTIN has a free plan of 5 queries/MONTH (not integrated); OSCBR V3 NCM is unproven. The official GTIN->NCM source is the SEFAZ CCG web service, which needs the client's ICP-Brasil certificate (not used).