feat: OSCBR optional provider, re-baseline after inactivity, field audit, provider UI/feature matrix tests, benchmark chain; 1.3.1

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
gti
2026-09-26 00:48:18 -03:00
co-authored by Claude Sonnet 5
parent c3c9d1b4e0
commit e6bcf12d53
27 changed files with 1015 additions and 41 deletions
+183
View File
@@ -0,0 +1,183 @@
{
"artifact": "PRODUCT_ENRICHMENT_FIELD_AUDIT_V1",
"source": "central/app/field_contract.py CONTRACT (all confirmed C000025 write fields)",
"fields_evaluated": 24,
"auto_fill_safe": [
"CLASSIFICACAO_FISCAL",
"IPI",
"CEST"
],
"rows": [
{
"FIELD": "CLASSIFICACAO_FISCAL",
"APPLICABILITY_SOURCE": "dataset NCM (Siscomex) ACTIVE + legacy NCM table",
"DETERMINISTIC_RULE_AVAILABLE": true,
"AUTO_FILL_SAFE": true,
"WHY_NOT": null,
"CONDITION": "only when KB-verified; single source = suggestion"
},
{
"FIELD": "IPI",
"APPLICABILITY_SOURCE": "dataset TIPI_IPI ACTIVE",
"DETERMINISTIC_RULE_AVAILABLE": true,
"AUTO_FILL_SAFE": true,
"WHY_NOT": null,
"CONDITION": "NT/blank rate or several EX rows => not written"
},
{
"FIELD": "EX_TIPI",
"APPLICABILITY_SOURCE": "dataset TIPI_IPI + product description evidence",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "needs the description to evidence the official EX text; ambiguous from name/GTIN -> manual"
},
{
"FIELD": "CEST",
"APPLICABILITY_SOURCE": "dataset CEST (CONFAZ 142/18) ACTIVE + BA_ST_MVA",
"DETERMINISTIC_RULE_AVAILABLE": true,
"AUTO_FILL_SAFE": true,
"WHY_NOT": null,
"CONDITION": "several candidates / description mismatch => not written"
},
{
"FIELD": "SITUACAO_TRIBUTARIA",
"APPLICABILITY_SOURCE": "ST engine (BA Anexo 1 + CEST corroboration) + purchase evidence / operator st_withheld",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on ST withheld / operator icms_treatment (company decision)"
},
{
"FIELD": "CFOP",
"APPLICABILITY_SOURCE": "dataset CFOP oficial + product role + ST",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on product_role, ST, service CFOP (company/operator decision)"
},
{
"FIELD": "CFOP_ENTRADA",
"APPLICABILITY_SOURCE": "dataset CFOP + product role + ST + company entry scope",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on company entry_scope and product role"
},
{
"FIELD": "CST",
"APPLICABILITY_SOURCE": "MOC CST_ICMS + ORIGEM tables + icms_treatment",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on icms_treatment and origin (operator)"
},
{
"FIELD": "CST_ENTRADA",
"APPLICABILITY_SOURCE": "MOC tables + icms_treatment",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on icms_treatment and origin (operator)"
},
{
"FIELD": "CSOSN",
"APPLICABILITY_SOURCE": "MOC CSOSN table + icms_treatment (operator or proven ST)",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on icms_treatment (operator or proven ST)"
},
{
"FIELD": "ALIQUOTA",
"APPLICABILITY_SOURCE": "regime + CSOSN/CST family + operator icms_rate",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on company/group icms_rate"
},
{
"FIELD": "COD_ORIGEM_MERCADORIA",
"APPLICABILITY_SOURCE": "operator origin only (no ERP field carries origin evidence)",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "no evidence source exists; operator only (explicitly never written)"
},
{
"FIELD": "CST_IPI",
"APPLICABILITY_SOURCE": "company C000004.CONTRIBUINTE_IPI",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "company flag decides: non-IPI-taxpayer => NOT_APPLICABLE (nothing to write); taxpayer => operator value"
},
{
"FIELD": "COD_ENTRADA_PIS",
"APPLICABILITY_SOURCE": "MOC CST_PIS + operator pis_cofins_profile",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on operator pis_cofins_profile"
},
{
"FIELD": "COD_SAIDA_PIS",
"APPLICABILITY_SOURCE": "MOC CST_PIS + operator pis_cofins_profile",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on operator pis_cofins_profile"
},
{
"FIELD": "ALIQ_PIS",
"APPLICABILITY_SOURCE": "operator pis_cofins_profile",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on operator pis_cofins_profile"
},
{
"FIELD": "COD_ENTRADA_COFINS",
"APPLICABILITY_SOURCE": "MOC CST_COFINS + operator pis_cofins_profile",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on operator pis_cofins_profile"
},
{
"FIELD": "COD_SAIDA_COFINS",
"APPLICABILITY_SOURCE": "MOC CST_COFINS + operator pis_cofins_profile",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on operator pis_cofins_profile"
},
{
"FIELD": "ALIQ_COFINS",
"APPLICABILITY_SOURCE": "operator pis_cofins_profile",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on operator pis_cofins_profile"
},
{
"FIELD": "INCIDENCIA_PISCOFINS",
"APPLICABILITY_SOURCE": "ERP declaration (NENHUMA / MONOFÁSICA / SUBST.TRIBUTÁRIA) + operator incidence",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "ERP declaration / operator incidence"
},
{
"FIELD": "PISCOFINS",
"APPLICABILITY_SOURCE": "ERP declaration + operator piscofins",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "ERP declaration / operator"
},
{
"FIELD": "IAT",
"APPLICABILITY_SOURCE": "company rounding policy (operator) or ERP declaration",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "company rounding policy (operator)"
},
{
"FIELD": "IPPT",
"APPLICABILITY_SOURCE": "product_role (own production evidence / purchase evidence / operator)",
"DETERMINISTIC_RULE_AVAILABLE": false,
"AUTO_FILL_SAFE": false,
"WHY_NOT": "depends on product_role (own production vs resale) - company decision"
},
{
"FIELD": "CCLASSTRIB",
"APPLICABILITY_SOURCE": "official RTC catalogue + CRT/date gates (existing RTC_CCLASSTRIB_WRITE_V1)",
"DETERMINISTIC_RULE_AVAILABLE": true,
"AUTO_FILL_SAFE": "only through the existing RTC gates/modes",
"WHY_NOT": "not part of PRODUCT_ENRICHMENT_WRITE_V1; the Agent only requests REPROCESS_PRODUCTS and the RTC central mode decides"
}
]
}
+618 -10
View File
@@ -1,5 +1,5 @@
{
"measured_at": "2026-09-25T21:53:01Z",
"measured_at": "2026-09-26T03:30:10Z",
"sample_source": "lab ERP databases (read-only), valid GTINs only",
"limit_per_provider": 30,
"providers": [
@@ -16,6 +16,23 @@
"errors": 0,
"error_kinds": {},
"latencies_ms": [],
"per_gtin": {},
"status": "NOT_CONFIGURED"
},
{
"provider": "oscbr",
"configured": false,
"sample_size": 0,
"found": 0,
"not_found": 0,
"with_description": 0,
"with_brand": 0,
"with_ncm": 0,
"with_cest": 0,
"errors": 0,
"error_kinds": {},
"latencies_ms": [],
"per_gtin": {},
"status": "NOT_CONFIGURED"
},
{
@@ -32,9 +49,47 @@
"error_kinds": {
"RATE_LIMITED": 1
},
"per_gtin": {
"0000000000017": {
"found": true,
"description": true,
"ncm": false,
"cest": false
},
"0000078947269": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"3165140434904": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7509546671574": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891000062722": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891010804626": {
"found": false,
"description": false,
"ncm": false,
"cest": false
}
},
"error_rate": 0.143,
"latency_p50_ms": 139.5,
"latency_p95_ms": 140,
"latency_p50_ms": 155.5,
"latency_p95_ms": 156,
"found_rate": 0.167,
"status": "MEASURED"
},
@@ -50,9 +105,191 @@
"with_cest": 0,
"errors": 0,
"error_kinds": {},
"per_gtin": {
"0000000000017": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"0000078947269": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"3165140434904": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7509546671574": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891000062722": {
"found": true,
"description": true,
"ncm": false,
"cest": false
},
"7891010804626": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891060773071": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891117029595": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891150066038": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891182019842": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891515901066": {
"found": true,
"description": true,
"ncm": false,
"cest": false
},
"7892261330643": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7893946483739": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7895316729362": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896007200368": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896014183111": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896058257663": {
"found": true,
"description": true,
"ncm": false,
"cest": false
},
"7896110005805": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896229906383": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896306617669": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896451880482": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896798602020": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897051900570": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897417402847": {
"found": true,
"description": true,
"ncm": false,
"cest": false
},
"7897751901631": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898016380031": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898128473119": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898219120700": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898322642458": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898402390316": {
"found": false,
"description": false,
"ncm": false,
"cest": false
}
},
"error_rate": 0.0,
"latency_p50_ms": 655.0,
"latency_p95_ms": 827,
"latency_p50_ms": 718.0,
"latency_p95_ms": 796,
"found_rate": 0.133,
"status": "MEASURED"
},
@@ -68,9 +305,191 @@
"with_cest": 0,
"errors": 0,
"error_kinds": {},
"per_gtin": {
"0000000000017": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"0000078947269": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"3165140434904": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7509546671574": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891000062722": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891010804626": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891060773071": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891117029595": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891150066038": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891182019842": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891515901066": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7892261330643": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7893946483739": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7895316729362": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896007200368": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896014183111": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896058257663": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896110005805": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896229906383": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896306617669": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896451880482": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896798602020": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897051900570": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897417402847": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897751901631": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898016380031": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898128473119": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898219120700": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898322642458": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898402390316": {
"found": false,
"description": false,
"ncm": false,
"cest": false
}
},
"error_rate": 0.0,
"latency_p50_ms": 640.0,
"latency_p95_ms": 969,
"latency_p50_ms": 703.0,
"latency_p95_ms": 953,
"found_rate": 0.0,
"status": "MEASURED"
},
@@ -86,11 +505,200 @@
"with_cest": 0,
"errors": 0,
"error_kinds": {},
"per_gtin": {
"0000000000017": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"0000078947269": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"3165140434904": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7509546671574": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891000062722": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891010804626": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891060773071": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891117029595": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891150066038": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891182019842": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7891515901066": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7892261330643": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7893946483739": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7895316729362": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896007200368": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896014183111": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896058257663": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896110005805": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896229906383": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896306617669": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896451880482": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7896798602020": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897051900570": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897417402847": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7897751901631": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898016380031": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898128473119": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898219120700": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898322642458": {
"found": false,
"description": false,
"ncm": false,
"cest": false
},
"7898402390316": {
"found": false,
"description": false,
"ncm": false,
"cest": false
}
},
"error_rate": 0.0,
"latency_p50_ms": 655.5,
"latency_p95_ms": 1311,
"latency_p50_ms": 703.0,
"latency_p95_ms": 938,
"found_rate": 0.0,
"status": "MEASURED"
}
]
],
"chain": {
"gtins_tested": 30,
"ANY_PROVIDER_FOUND_RATE": 0.167,
"ANY_PROVIDER_NCM_RATE": 0.0,
"ANY_PROVIDER_CEST_RATE": 0.0,
"ANY_PROVIDER_DESCRIPTION_RATE": 0.167
}
}
+17
View File
@@ -42,6 +42,7 @@ and `enrichment.is_human_overridden(state_path, code, field)` is the hook every
|---|---|---|---|
| Cosmos (Bluesoft) | `COSMOS_TOKEN` in the Hub env (optional) | 25/day | description, brand, **NCM, CEST** — `NOT_CONFIGURED` without a token (not an error) |
| UPCitemdb (trial) | no key | 100/day, burst 15 / 30 s (headers `X-RateLimit-*`) | title, brand, category |
| OSCBR / RSC Sistemas GTIN | free plan account made by the operator; `OSCBR_USER` / `OSCBR_PASSWORD` in the Hub env; 1 h bearer renewed automatically | 20 req/min | name, brand, category, **NCM** (CEST when present) — `NOT_CONFIGURED` without credentials |
| Open Food Facts / Open Products Facts / Open Beauty Facts | open (ODbL), identifiable `User-Agent` | 15 req/min | title, brand, quantity |
Quota is central (`ProviderState`, `ProviderUsageDaily`, `quota_kind = REMOTE_QUOTA | LOCAL_ESTIMATE | DOCUMENTED_RATE`), with positive (30 d) and negative (24 h) `GtinLookupCache`, single-flight per (provider, GTIN), circuit breaker
(3 failures → 60 s doubling up to 15 min; 429 → until reset; 401/403 → 1 h). Quota exhausted → `PENDING_PROVIDER_QUOTA` + retry after reset.
@@ -65,3 +66,19 @@ Agent detail: *Funcionalidades* (admin) and *Enriquecimento* (counters + product
second run 0 writes; DRY_RUN 0 writes; WAITING/DISABLED 0 Firebird activity and no baseline; human override incl. deleted value; Hub offline → pending → recovers.
- Migration 0029: fresh-vs-upgrade `pg_dump -s` identical; upgrade → downgrade → upgrade clean.
- Live smoke `scripts/enrichment_live_smoke.py` (1 lookup/provider, real internet, separate from the suite).
## Provider evaluation (2026-09-26)
- **DotCompany — NOT integrated.** The free query (25/day per IP, no signup) is the *website* demo for human use; the API (`/api/v1/catalogo/gtin/{ean}`) is prepaid credits (4 credits = R$ 0,04 per query, packages from R$ 120); the only free API access is a `dc_test_` sandbox with test data. Automated free use is not offered, so no adapter (would violate zero cost / permitted automation).
- **OSCBR / RSC Sistemas — adapter implemented, optional.** Free initial plan without card, 20 req/min, token via account registration (Basic auth on `/oauth/token`, 1 h Bearer), `GET /api/gtin/infor/{gtin}` (fields `ncm`, `nome`, `marca`, `categoria`, `pais`). Terms are shown only inside the authenticated panel, so the operator must create the account and read them; the Hub never creates accounts. Without credentials it is `NOT_CONFIGURED` and makes no request.
- Cosmos stays optional (`COSMOS_TOKEN`). Nothing else qualified (GS1 CNP is a paid membership; Verified by GS1 is not an automation API).
- `PAID_PROVIDER_CALLS=0`; `ALLOW_PAID_PROVIDERS=false`.
## Field audit (`docs/PRODUCT_ENRICHMENT_FIELD_AUDIT_V1.json`)
24 fields checked against `field_contract`; only `CLASSIFICACAO_FISCAL`, `CEST`, `IPI` have a deterministic, company-independent rule. All others need company/operator context (product role, ICMS treatment, origin, PIS/COFINS profile, entry scope, rounding) or description evidence (EX_TIPI) and are excluded on purpose; `CST_IPI` is NOT_APPLICABLE for non-IPI companies; `CCLASSTRIB` stays with the RTC gates.
## Re-baseline after inactivity (1.3.1)
While the function is DISABLED / WAITING_FEATURE_CONFIGURATION the Agent marks a re-baseline; on reactivation the current catalogue becomes history, so a long-disabled installation never sends a burst of products to the providers.
## Production KB seed strategy
Production KB starts empty (0 products, head 0028 before deploy). The lab databases are **not** imported into production: their observations are `CLIENT_OBSERVATION` evidence only and cannot verify anything alone (93/241 shared GTINs disagree on NCM). The KB grows from provider evidence on real new products; `scripts/enrichment_lab_kb.py --seed-url` exists for a dev/staging seed only.