4.1 KiB
NCM RETRIEVAL ARCHITECTURE V2
Deterministic semantic candidate retrieval (zero LLM) implemented in
central/app/ai/retrieval.py. Goal: surface the correct official code among candidates
BEFORE spending on LLM selection.
Sources merged into one ranking
- A. OFFICIAL hierarchical — every 8-digit NCM document is enriched with its ancestors' official descriptions (chapter 2-digit, position 4, subposition 6) from the ACTIVE dataset, so short legal texts ("Outros") carry their path vocabulary. Search is two-stage: rank positions, then expand their 8-digit descendants (top-15 positions).
- B. KB neighbor bridge — the Knowledge Base maps COMMERCIAL descriptions to VALIDATED NCMs (conflicting products excluded). Query = commercial-token similarity; evidence per candidate = similarity, observation_count, union of distinct tenants (agreement reported, never a majority decision).
- C. HYBRID — single ranking, weights OFFICIAL 0.3 / KB 0.7 (tuned on the V2 benchmark; the KB bridge carries the signal, the official component breaks ties and covers products absent from the KB).
Commercial normalization (section 7)
commercial_tokens(): NFKD/ascii/upper stems with Portuguese plural stemming, minus
units (UN CX FD PCT KG G ML L LT PET LATA GARRAFA ...), sizes/codes (1KG, 500ML, CX 12,
pure numbers), and TAM/COR/REF/COD/MOD + their value token. Impact measured on the
benchmark, not assumed.
Abbreviation lexicon (section 8)
derive_abbreviation_lexicon(): statistical, KB-derived — a 3-5 char token that is a
strict prefix of a longer token, both occurring across distinct products sharing the
same validated NCM (support ≥ 3). 644 abbreviations derived (e.g. REFRI→REFRIGERANTE,
CAM→CAMISA). No LLM involved. Query stems are expanded before official retrieval.
Leave-one-out discipline
Every KB/eval measurement excludes the queried product itself from the neighbor index
(exclude_product_id); the first run without it showed fake 99.5% recall — self-retrieval
leakage — and was corrected before any reporting.
Benchmark (docs/NCM_RETRIEVAL_EVALUATION_V2.json, 200 samples, seed 42)
Quality audit: GOLD 65 (GTIN or cross-tenant agreement), SILVER 118, NOISY 17 (<2 informative tokens; not pursued).
| Mode | R@1 | R@5 | R@10 | R@20 | R@50 | MRR |
|---|---|---|---|---|---|---|
| OFFICIAL_ONLY (hierarchical+enriched) | .015 | .08 | .11 | .14 | .195 | .046 |
| KB_ONLY (neighbors, leave-one-out) | .665 | .85 | .895 | .925 | .935 | .750 |
| HYBRID | .635 | .855 | .90 | .92 | .935 | .733 |
GOLD × HYBRID: R@1 .785, R@10 .954, R@20 .954, R@50 .954, MRR .839.
Liberation gate (section 14): PASS — GOLD_RECALL@20 = 0.954 ≥ 0.80 and GOLD_RECALL@50 = 0.954 ≥ 0.90. Real-LLM selection may now be evaluated on the same benchmark; AI write remains forbidden.
GTIN audit (docs/GTIN_KB_AUDIT_V1.json)
VALID 8,559 (0 invalid — checksum-validated at ingest) · duplicated-same-product 256 · cross-tenant matches 251 (strong knowledge-sharing evidence) · conflicting NCMs on the same GTIN 90 (surfaced as data-quality conflicts, examples in the artifact).
Why KB_EXACT_HITS was 0 (section 11)
Root cause: the V1 work-item pass resolved identity by fingerprint only — the GTIN lookup was never attempted — and INVALID_NCM products have no validated NCM of their own, so hits only arise via a shared product with a valid NCM. V2 tries GTIN first (KB_EXACT_HIT=4) then fingerprint (KB_FINGERPRINT_HIT=16). No identity loosening.
Work-item reprocessing (docs/AI_RESIDUAL_PLAN_V2.json)
750 items → 4 KB_EXACT_HIT + 16 KB_FINGERPRINT_HIT + 1 HIGH_CONFIDENCE_RETRIEVAL (hybrid ≥ 0.75, official-valid, cross-tenant ≥ 2 — a candidate, NOT an auto fiscal resolution) + 143 SINGLE_OFFICIAL_CANDIDATE (CEST/EX with exactly one official candidate) + 511 LLM_SELECTION_REQUIRED (4,181 cells) + 75 UNRESOLVED.
Projected LLM cost drops $0.54 → $0.44 (730 → 511 calls).
CEST / EX TIPI (section 15)
Reuse the same official-candidate lists (cest_candidates_for_ncm,
ex_tipi_candidates_for_ncm); 143 items resolved to a single official candidate with no
AI at all. Multi-candidate items wait for the LLM evaluation pass.