feat: coverage V2 - historical evidence miner, KB evidence model + promotion audit + revalidation, Kodebar/OSCBR limits, KB/provider panel, seed import; 1.3.2 (Hub only)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-26 08:14:57 -03:00
parent 19e145ea47
commit ac1ecccc50
41 changed files with 1418 additions and 74 deletions
@@ -82,3 +82,28 @@ While the function is DISABLED / WAITING_FEATURE_CONFIGURATION the Agent marks a

 ## Production KB seed strategy
 Production KB starts empty (0 products, head 0028 before deploy). The lab databases are **not** imported into production: their observations are `CLIENT_OBSERVATION` evidence only and cannot verify anything alone (93/241 shared GTINs disagree on NCM). The KB grows from provider evidence on real new products; `scripts/enrichment_lab_kb.py --seed-url` exists for a dev/staging seed only.
+
+
+## Coverage V2 (1.3.2, Hub-only, migration 0030) - historical evidence + KB model
+- **Miner** (`scripts/enrichment_history_miner.py`, READ-ONLY, server-side `GROUP BY`): C000025 master, `NFCE_ITENS` (NCM/CEST per document line, dated), `C000062` (NF-e entry items: `CLASSIFICACAO_FISCAL`, `CODBARRA`, `DESCRICAO2`), `C000032`/`C000057` (EAN seen in movements).
+  Five databases in ~17 s, peak 28 MB Python, 7 MB of output (`docs/runs/enrichment_history_miner_report.json`). No XML stores exist; there is no NF-e header/item table with dates beyond the ones above.
+- **Evidence model** (`central/app/enrichment/history.py`, `kb.py`): per (GTIN, value) one *lineage* per client (master flag, document count, documents per year). Master data and the fiscal documents of the same ERP are ONE family (`CLIENT_DATA`); each NCM-bearing provider is another family.
+  `CLIENT_DATA` counts only when strong (>= 20 documents over >= 2 years, or >= 2 distinct clients) and even then reaches at most `CORROBORATED`.
+- **KB states** per NCM: `OBSERVED, CORROBORATED, EXTERNAL_CORROBORATED, VERIFIED_KB, CONFLICT, STALE`. Temporal state per NCM: `HISTORICAL_STABLE_CURRENT, HISTORICAL_CHANGED, HISTORICAL_EXPIRED, HISTORICAL_INVALID` (+ `HISTORICAL_CONFLICT`), judged against the ACTIVE official dataset today.
+- **Promotion to VERIFIED_KB** (rule `kb-promotion-v2`): exact valid GTIN + NCM vigente in the official dataset + >= 2 independent families (strong `CLIENT_DATA` + >= 1 NCM-bearing provider, or >= 2 providers) + zero conflicting current value. Every promotion writes `kb_promotion_audits`
+  (families, clients/documents/years/providers, official dataset version+sha, rule version). Three clients agreeing NEVER verify by themselves. Two different current NCMs = `CONFLICT` (no majority), an expired code that is as recent as the current one blocks "stable".
+- **Order per product**: VERIFIED_KB (zero external requests) -> local CONFLICT (zero external, no write) -> locally corroborated (only NCM-bearing providers may still be asked) -> provider cache -> provider (single-flight per GTIN) -> fiscal validation.
+  Backfill calls can never use the operational reserve (`PROVIDER_OPERATIONAL_RESERVE_PERCENT`, default 20). No mass provider use for the historical seed: it is local-only.
+- **Revalidation**: after a new official NCM dataset, `kb.revalidate` never deletes; codes that left the dataset become `STALE` (`STALE_REVIEW_REQUIRED`, audited when they were VERIFIED_KB).
+- **Aliases**: descriptions seen over the years are stored as non-fiscal aliases + fingerprints (marketing words such as ORIGINAL/TRADICIONAL ignored) to improve name-only matching; name-only still fills only from a unique VERIFIED_KB product.
+- **Import**: `python -m central.scripts.import_kb_history --staging DIR [--apply]` (dry-run by default, idempotent, replaces a client's lineage instead of accumulating). Tenant labels are kept; CRT/CFOP/CSOSN or any client tax attribute is never stored.
+- **Panel** (same dashboard): provider columns `FOUND_TODAY, NCM_FOUND_TODAY, CACHE_HITS, RATE_LIMITED, NEXT_RESET` and a compact Knowledge Base block (known products, verified, corroborated, conflicts, KB hits today, latest conflicts with clients/documents/providers).
+
+### Providers added / evaluated in V2
+- **Kodebar** (adapter, optional, identity only): `KODEBAR_API_KEY`, free "Teste" plan 50/day (+10% tolerance not used), no card, `GET /gtin/lookup?gtin=` with `X-API-Key`; terms allow integration in our own system and internal caching, forbid redistribution; 429 above quota, no overage. Returns name/brand/thumbnail, **no NCM**.
+- **OSCBR**: adapter limits set to the current free plan (50/day, 5/min). `NOT_CONFIGURED` without `OSCBR_USER/OSCBR_PASSWORD`.
+- **UPCitemdb**: after a 429 the circuit stays open >= 1 h (never insist); the observed trial limit is stricter than the documented burst.
+- **Tabelas Fiscais**: NOT integrated at runtime. Our NCM/TIPI/CEST datasets already come from the official sources and live locally; the site is a third-party mirror, so it would add no authority. If ever needed, use its downloadable CSV/JSON (local dataset mode) instead of per-product requests.
+
+### Measured (local data only; `docs/runs/enrichment_history_analysis.json`)
+See the V2 report in the mission summary; key findings: cross-client NCM agreement is poor (a candidate taken from the other clients matches the held-out client's own NCM only 62 % of the time, and only 42 % even when >= 2 clients corroborate it), which is exactly why client data can never verify an NCM on its own.
@@ -0,0 +1,147 @@
+{
+ "DATABASES_ANALYZED": [
+  "CONSTRUCAO",
+  "MMMODAS",
+  "RINCAO",
+  "TESTESOFICIAIS",
+  "UNIAO"
+ ],
+ "YEARS_COVERED": [
+  2022,
+  2026
+ ],
+ "CURRENT_PRODUCT_ROWS": 35066,
+ "HISTORICAL_DOCUMENT_ROWS_ANALYZED": 438592,
+ "MOVEMENT_ROWS_SCANNED_FOR_BARCODES": 3512884,
+ "DISTINCT_GTINS": 8615,
+ "GTINS_WITH_HISTORICAL_NCM": 8615,
+ "GTINS_WITH_CURRENT_VALID_NCM": 8304,
+ "GTINS_STABLE_ACROSS_YEARS": 2970,
+ "GTINS_CHANGED_OVER_TIME": 2,
+ "GTIN_NCM_CONFLICTS": 264,
+ "GTINS_IN_2PLUS_CLIENTS": 253,
+ "KB_STATUS": {
+  "CORROBORATED": 1107,
+  "OBSERVED": 7244,
+  "CONFLICT": 264
+ },
+ "WOULD_VERIFY_WITHOUT_EXTERNAL_PROVIDER": 0,
+ "build_stats": {
+  "master_rows": 35066,
+  "links_from_documents": 62,
+  "nfce_groups": 25300,
+  "nfce_documents": 130790,
+  "nfce_groups_without_gtin": 15411,
+  "nfe_documents": 1394,
+  "ambiguous_product_links_skipped": 4,
+  "gtins": 8615
+ },
+ "miner_cost": {
+  "rows_scanned_and_seconds": [
+   [
+    "CONSTRUCAO",
+    2.0
+   ],
+   [
+    "MMMODAS",
+    6.7
+   ],
+   [
+    "RINCAO",
+    2.7
+   ],
+   [
+    "UNIAO",
+    5.2
+   ],
+   [
+    "TESTESOFICIAIS",
+    0.4
+   ]
+  ],
+  "peak_python_memory_mb": 28.1,
+  "output_mb": 7.15
+ },
+ "LOCAL_HISTORY_IDENTITY_RATE": 100.0,
+ "LOCAL_HISTORY_NCM_EVIDENCE_RATE": 100.0,
+ "LOCAL_HISTORY_NCM_CORROBORATED_RATE": 12.8,
+ "LOCAL_MULTI_CLIENT_RATE": 2.9,
+ "LOCAL_CONFLICT_RATE": 3.1,
+ "LEAVE_ONE_CLIENT_OUT": {
+  "tested": 506,
+  "identity_known": 506,
+  "ncm_candidate": 481,
+  "corroborated": 79,
+  "conflict": 20,
+  "agrees_with_holdout_ncm": 299,
+  "disagrees_with_holdout_ncm": 182,
+  "corroborated_agree": 33,
+  "corroborated_disagree": 46,
+  "per_client": {
+   "CONSTRUCAO": {
+    "tested": 219,
+    "ncm_candidate": 202,
+    "agree": 142,
+    "disagree": 60
+   },
+   "MMMODAS": {
+    "tested": 0,
+    "ncm_candidate": 0,
+    "agree": 0,
+    "disagree": 0
+   },
+   "RINCAO": {
+    "tested": 34,
+    "ncm_candidate": 34,
+    "agree": 4,
+    "disagree": 30
+   },
+   "TESTESOFICIAIS": {
+    "tested": 0,
+    "ncm_candidate": 0,
+    "agree": 0,
+    "disagree": 0
+   },
+   "UNIAO": {
+    "tested": 253,
+    "ncm_candidate": 245,
+    "agree": 153,
+    "disagree": 92
+   }
+  },
+  "IDENTITY_RATE": 100.0,
+  "NCM_CANDIDATE_RATE": 95.1,
+  "CORROBORATED_RATE": 15.6,
+  "CONFLICT_RATE": 4.0,
+  "corroborated_agrees_with_holdout_pct": 41.8,
+  "candidate_agrees_with_holdout_pct": 62.2
+ },
+ "BENCHMARK_SEGMENTS": {
+  "CONSTRUCAO": {
+   "segment": "constru\u00e7\u00e3o",
+   "gtins": 2111,
+   "with_ncm_evidence": 2111
+  },
+  "UNIAO": {
+   "segment": "mercado/varejo",
+   "gtins": 6225,
+   "with_ncm_evidence": 6225
+  },
+  "MMMODAS": {
+   "segment": "moda",
+   "gtins": 70,
+   "with_ncm_evidence": 70
+  },
+  "RINCAO": {
+   "segment": "outros (ra\u00e7\u00e3o/agro)",
+   "gtins": 408,
+   "with_ncm_evidence": 408
+  },
+  "TESTESOFICIAIS": {
+   "segment": "outros/teste",
+   "gtins": 52,
+   "with_ncm_evidence": 52
+  }
+ },
+ "BENCHMARK_SAMPLE_SIZE": 8615
+}
@@ -0,0 +1,86 @@
+{
+ "databases": [
+  {
+   "database": "CONSTRUCAO",
+   "nfce_rows_scanned": 28934,
+   "nfe_rows_scanned": 547,
+   "c000032_rows_scanned": 91886,
+   "c000057_rows_scanned": 28428,
+   "master_rows": 3712,
+   "nfce_groups": 5602,
+   "nfe_groups": 301,
+   "barcode_groups": 0,
+   "years": [
+    2024,
+    2026
+   ],
+   "seconds": 2.0
+  },
+  {
+   "database": "MMMODAS",
+   "nfce_rows_scanned": 26067,
+   "nfe_rows_scanned": 66,
+   "c000032_rows_scanned": 192763,
+   "c000057_rows_scanned": 5241,
+   "master_rows": 20997,
+   "nfce_groups": 9717,
+   "nfe_groups": 43,
+   "barcode_groups": 9982,
+   "years": [
+    2024,
+    2026
+   ],
+   "seconds": 6.7
+  },
+  {
+   "database": "RINCAO",
+   "nfce_rows_scanned": 224355,
+   "nfe_rows_scanned": 548,
+   "c000032_rows_scanned": 1586249,
+   "c000057_rows_scanned": 0,
+   "master_rows": 894,
+   "nfce_groups": 1288,
+   "nfe_groups": 118,
+   "barcode_groups": 0,
+   "years": [
+    2024,
+    2026
+   ],
+   "seconds": 2.7
+  },
+  {
+   "database": "UNIAO",
+   "nfce_rows_scanned": 121266,
+   "nfe_rows_scanned": 1170,
+   "c000032_rows_scanned": 1582133,
+   "c000057_rows_scanned": 21171,
+   "master_rows": 8942,
+   "nfce_groups": 8363,
+   "nfe_groups": 460,
+   "barcode_groups": 13,
+   "years": [
+    2024,
+    2026
+   ],
+   "seconds": 5.2
+  },
+  {
+   "database": "TESTESOFICIAIS",
+   "nfce_rows_scanned": 34946,
+   "nfe_rows_scanned": 693,
+   "c000032_rows_scanned": 4905,
+   "c000057_rows_scanned": 108,
+   "master_rows": 521,
+   "nfce_groups": 330,
+   "nfe_groups": 304,
+   "barcode_groups": 13,
+   "years": [
+    2022,
+    2026
+   ],
+   "seconds": 0.4
+  }
+ ],
+ "peak_python_memory_mb": 28.1,
+ "output_mb": 7.15
+}
@@ -0,0 +1,32 @@
+{
+ "copy_only": true,
+ "hub": "http://127.0.0.1:8000",
+ "info": {
+  "promotion": "PROMO CORROBORATED VERIFIED_KB ['PROVIDER:oscbr', 'CLIENT_DATA'] 1 kb-promotion-v2",
+  "verified_kb_new_product_row": [
+   "96034090",
+   null,
+   null
+  ],
+  "latency_s": 3.38,
+  "second_client": {
+   "state": "EXACT_GTIN_VERIFIED",
+   "ncm": "96034090"
+  },
+  "conflict_candidates": "21039021:[\"UNIAO\"] x 39174090:[\"UNIAO\"]"
+ },
+ "checks": {
+  "HISTORICAL_GTIN_PROMOTED_WITH_AUDIT": true,
+  "DRY_RUN_ZERO_WRITES": true,
+  "VERIFIED_KB_NEW_PRODUCT_NCM_WRITTEN": true,
+  "ZERO_EXTERNAL_REQUESTS_ON_KB_HIT": true,
+  "KB_HIT_COUNTED": true,
+  "SECOND_CLIENT_ENABLED": true,
+  "SECOND_CLIENT_KB_REUSE_ZERO_PROVIDER_REQUESTS": true,
+  "CONFLICT_NO_NCM_WRITE": true,
+  "CONFLICT_ZERO_PROVIDER_REQUESTS": true,
+  "CONFLICT_EXPLAINED_IN_PANEL_DATA": true,
+  "REAL_CLIENT_WRITES_0": true
+ },
+ "ALL_PASS": true
+}