85 lines
4.1 KiB
Markdown
85 lines
4.1 KiB
Markdown
# NCM RETRIEVAL ARCHITECTURE V2
|
||
|
||
Deterministic semantic candidate retrieval (zero LLM) implemented in
|
||
`central/app/ai/retrieval.py`. Goal: surface the correct official code among candidates
|
||
BEFORE spending on LLM selection.
|
||
|
||
## Sources merged into one ranking
|
||
|
||
- **A. OFFICIAL hierarchical** — every 8-digit NCM document is enriched with its
|
||
ancestors' official descriptions (chapter 2-digit, position 4, subposition 6) from the
|
||
ACTIVE dataset, so short legal texts ("Outros") carry their path vocabulary. Search is
|
||
two-stage: rank positions, then expand their 8-digit descendants (top-15 positions).
|
||
- **B. KB neighbor bridge** — the Knowledge Base maps COMMERCIAL descriptions to
|
||
VALIDATED NCMs (conflicting products excluded). Query = commercial-token similarity;
|
||
evidence per candidate = similarity, observation_count, union of distinct tenants
|
||
(agreement reported, never a majority decision).
|
||
- **C. HYBRID** — single ranking, weights OFFICIAL 0.3 / KB 0.7 (tuned on the V2
|
||
benchmark; the KB bridge carries the signal, the official component breaks ties and
|
||
covers products absent from the KB).
|
||
|
||
## Commercial normalization (section 7)
|
||
|
||
`commercial_tokens()`: NFKD/ascii/upper stems with Portuguese plural stemming, minus
|
||
units (UN CX FD PCT KG G ML L LT PET LATA GARRAFA ...), sizes/codes (1KG, 500ML, CX 12,
|
||
pure numbers), and TAM/COR/REF/COD/MOD + their value token. Impact measured on the
|
||
benchmark, not assumed.
|
||
|
||
## Abbreviation lexicon (section 8)
|
||
|
||
`derive_abbreviation_lexicon()`: statistical, KB-derived — a 3-5 char token that is a
|
||
strict prefix of a longer token, both occurring across distinct products sharing the
|
||
same validated NCM (support ≥ 3). 644 abbreviations derived (e.g. REFRI→REFRIGERANTE,
|
||
CAM→CAMISA). No LLM involved. Query stems are expanded before official retrieval.
|
||
|
||
## Leave-one-out discipline
|
||
|
||
Every KB/eval measurement excludes the queried product itself from the neighbor index
|
||
(`exclude_product_id`); the first run without it showed fake 99.5% recall — self-retrieval
|
||
leakage — and was corrected before any reporting.
|
||
|
||
## Benchmark (docs/NCM_RETRIEVAL_EVALUATION_V2.json, 200 samples, seed 42)
|
||
|
||
Quality audit: GOLD 65 (GTIN or cross-tenant agreement), SILVER 118, NOISY 17 (<2
|
||
informative tokens; not pursued).
|
||
|
||
| Mode | R@1 | R@5 | R@10 | R@20 | R@50 | MRR |
|
||
|---|---|---|---|---|---|---|
|
||
| OFFICIAL_ONLY (hierarchical+enriched) | .015 | .08 | .11 | .14 | .195 | .046 |
|
||
| KB_ONLY (neighbors, leave-one-out) | .665 | .85 | .895 | .925 | .935 | .750 |
|
||
| HYBRID | .635 | .855 | .90 | .92 | .935 | .733 |
|
||
|
||
GOLD × HYBRID: R@1 .785, R@10 .954, **R@20 .954**, **R@50 .954**, MRR .839.
|
||
|
||
**Liberation gate (section 14): PASS** — GOLD_RECALL@20 = 0.954 ≥ 0.80 and
|
||
GOLD_RECALL@50 = 0.954 ≥ 0.90. Real-LLM selection may now be evaluated on the same
|
||
benchmark; AI write remains forbidden.
|
||
|
||
## GTIN audit (docs/GTIN_KB_AUDIT_V1.json)
|
||
|
||
VALID 8,559 (0 invalid — checksum-validated at ingest) · duplicated-same-product 256 ·
|
||
cross-tenant matches 251 (strong knowledge-sharing evidence) · conflicting NCMs on the
|
||
same GTIN 90 (surfaced as data-quality conflicts, examples in the artifact).
|
||
|
||
## Why KB_EXACT_HITS was 0 (section 11)
|
||
|
||
Root cause: the V1 work-item pass resolved identity by fingerprint only — the GTIN
|
||
lookup was never attempted — and INVALID_NCM products have no validated NCM of their own,
|
||
so hits only arise via a shared product with a valid NCM. V2 tries GTIN first
|
||
(KB_EXACT_HIT=4) then fingerprint (KB_FINGERPRINT_HIT=16). No identity loosening.
|
||
|
||
## Work-item reprocessing (docs/AI_RESIDUAL_PLAN_V2.json)
|
||
|
||
750 items → 4 KB_EXACT_HIT + 16 KB_FINGERPRINT_HIT + 1 HIGH_CONFIDENCE_RETRIEVAL
|
||
(hybrid ≥ 0.75, official-valid, cross-tenant ≥ 2 — a candidate, NOT an auto fiscal
|
||
resolution) + 143 SINGLE_OFFICIAL_CANDIDATE (CEST/EX with exactly one official
|
||
candidate) + 511 LLM_SELECTION_REQUIRED (4,181 cells) + 75 UNRESOLVED.
|
||
|
||
Projected LLM cost drops $0.54 → $0.44 (730 → 511 calls).
|
||
|
||
## CEST / EX TIPI (section 15)
|
||
|
||
Reuse the same official-candidate lists (`cest_candidates_for_ncm`,
|
||
`ex_tipi_candidates_for_ncm`); 143 items resolved to a single official candidate with no
|
||
AI at all. Multi-candidate items wait for the LLM evaluation pass.
|