GTI_GERENCIADOR_FISCAL/fiscal/docs/NCM_RETRIEVAL_ARCHITECTURE_V2.md

NCM RETRIEVAL ARCHITECTURE V2

Deterministic semantic candidate retrieval (zero LLM) implemented in central/app/ai/retrieval.py. Goal: surface the correct official code among candidates BEFORE spending on LLM selection.

Sources merged into one ranking

• A. OFFICIAL hierarchical — every 8-digit NCM document is enriched with its ancestors' official descriptions (chapter 2-digit, position 4, subposition 6) from the ACTIVE dataset, so short legal texts ("Outros") carry their path vocabulary. Search is two-stage: rank positions, then expand their 8-digit descendants (top-15 positions).
• B. KB neighbor bridge — the Knowledge Base maps COMMERCIAL descriptions to VALIDATED NCMs (conflicting products excluded). Query = commercial-token similarity; evidence per candidate = similarity, observation_count, union of distinct tenants (agreement reported, never a majority decision).
• C. HYBRID — single ranking, weights OFFICIAL 0.3 / KB 0.7 (tuned on the V2 benchmark; the KB bridge carries the signal, the official component breaks ties and covers products absent from the KB).

Commercial normalization (section 7)

commercial_tokens(): NFKD/ascii/upper stems with Portuguese plural stemming, minus units (UN CX FD PCT KG G ML L LT PET LATA GARRAFA ...), sizes/codes (1KG, 500ML, CX 12, pure numbers), and TAM/COR/REF/COD/MOD + their value token. Impact measured on the benchmark, not assumed.

Abbreviation lexicon (section 8)

derive_abbreviation_lexicon(): statistical, KB-derived — a 3-5 char token that is a strict prefix of a longer token, both occurring across distinct products sharing the same validated NCM (support ≥ 3). 644 abbreviations derived (e.g. REFRI→REFRIGERANTE, CAM→CAMISA). No LLM involved. Query stems are expanded before official retrieval.

Leave-one-out discipline

Every KB/eval measurement excludes the queried product itself from the neighbor index (exclude_product_id); the first run without it showed fake 99.5% recall — self-retrieval leakage — and was corrected before any reporting.

Benchmark (docs/NCM_RETRIEVAL_EVALUATION_V2.json, 200 samples, seed 42)

Quality audit: GOLD 65 (GTIN or cross-tenant agreement), SILVER 118, NOISY 17 (<2 informative tokens; not pursued).

Mode	R@1	R@5	R@10	R@20	R@50	MRR
OFFICIAL_ONLY (hierarchical+enriched)	.015	.08	.11	.14	.195	.046
KB_ONLY (neighbors, leave-one-out)	.665	.85	.895	.925	.935	.750
HYBRID	.635	.855	.90	.92	.935	.733

GOLD × HYBRID: R@1 .785, R@10 .954, R@20 .954, R@50 .954, MRR .839.

Liberation gate (section 14): PASS — GOLD_RECALL@20 = 0.954 ≥ 0.80 and GOLD_RECALL@50 = 0.954 ≥ 0.90. Real-LLM selection may now be evaluated on the same benchmark; AI write remains forbidden.

GTIN audit (docs/GTIN_KB_AUDIT_V1.json)

VALID 8,559 (0 invalid — checksum-validated at ingest) · duplicated-same-product 256 · cross-tenant matches 251 (strong knowledge-sharing evidence) · conflicting NCMs on the same GTIN 90 (surfaced as data-quality conflicts, examples in the artifact).

Why KB_EXACT_HITS was 0 (section 11)

Root cause: the V1 work-item pass resolved identity by fingerprint only — the GTIN lookup was never attempted — and INVALID_NCM products have no validated NCM of their own, so hits only arise via a shared product with a valid NCM. V2 tries GTIN first (KB_EXACT_HIT=4) then fingerprint (KB_FINGERPRINT_HIT=16). No identity loosening.

Work-item reprocessing (docs/AI_RESIDUAL_PLAN_V2.json)

750 items → 4 KB_EXACT_HIT + 16 KB_FINGERPRINT_HIT + 1 HIGH_CONFIDENCE_RETRIEVAL (hybrid ≥ 0.75, official-valid, cross-tenant ≥ 2 — a candidate, NOT an auto fiscal resolution) + 143 SINGLE_OFFICIAL_CANDIDATE (CEST/EX with exactly one official candidate) + 511 LLM_SELECTION_REQUIRED (4,181 cells) + 75 UNRESOLVED.

Projected LLM cost drops $0.54 → $0.44 (730 → 511 calls).

CEST / EX TIPI (section 15)

Reuse the same official-candidate lists (cest_candidates_for_ncm, ex_tipi_candidates_for_ncm); 143 items resolved to a single official candidate with no AI at all. Multi-candidate items wait for the LLM evaluation pass.