chore: capture semantic pipeline baseline

2026-09-21 15:52:17 -03:00
parent 9dcc263217
commit 6731eeca08
47 changed files with 3724 additions and 15 deletions
@@ -1,6 +1,55 @@
 # AI Project State

-## Current status
+## Current status (real LLM evaluation V1 — 2026-09-21)
+
+`BLOCKED_EXTERNAL_AI_CREDENTIAL` (implementation complete). The entire real-LLM
+evaluation pipeline is built and tested offline: OpenAI-compatible provider
+(`central/app/ai/llm_provider.py`, key only from Hub env, injectable transport),
+strict per-item validator (candidate-list confinement, abstain semantics, 1:1
+work_item_id, calibrated confidence in [0,1]), progressive batch probing (20/10/5),
+post-AI validator (active-dataset membership), metrics + A/B/C error attribution +
+confidence-bucket calibration (`central/app/ai/eval_metrics.py`), permanent result
+cache (`ai_eval_cache`, migration 0014) with zero-call re-execution proof, and the
+runner `central/scripts/run_real_llm_ncm_eval.py` (gold-first; silver/noisy only
+after the gate: selection accuracy >= 0.95 AND high-confidence wrong rate <= 0.02;
+artifacts REAL_LLM_NCM_EVALUATION_V1/_ERRORS_/_COST_). The runner refuses to call
+anything without `AI_API_KEY` and exits BLOCKED_EXTERNAL_AI_CREDENTIAL with the exact
+operator action. Zero LLM calls, zero Firebird writes, AI_WRITE_ALLOWED=false.
+Operator action: set AI_API_KEY (optionally AI_BASE_URL/AI_MODEL) in the Hub .env.
+
+## Semantic candidate retrieval V2 (2026-09-21)
+
+`NCM_RETRIEVAL_V2_GATE_PASSED`. Deterministic hybrid retrieval (hierarchical official
+corpus enriched with ancestor descriptions + KB commercial->NCM neighbor bridge +
+KB-derived abbreviation lexicon, 644 entries; weights 0.3/0.7; leave-one-out discipline
+after catching self-retrieval leakage) lifted candidate recall from 7.5%@50 (V1) to
+HYBRID 92%@20 / 93.5%@50, GOLD 95.4%@20/@50, MRR 0.84 — meeting the LLM liberation gate
+(GOLD@20 ≥ 0.80, @50 ≥ 0.90). Benchmark quality audit: GOLD 65 / SILVER 118 / NOISY 17.
+GTIN audit: 8,559 valid, 251 cross-tenant matches, 90 GTIN-conflicting NCMs surfaced.
+Reprocessing the 750 work items: 4 KB_EXACT + 16 KB_FINGERPRINT + 1 high-confidence
+retrieval + 143 single-official-candidate (CEST/EX) resolved deterministically; 511
+items (4,181 cells) still need LLM selection; projected cost $0.54 → $0.44. Zero LLM
+calls, zero Firebird writes, AI_WRITE_ALLOWED remains false. Artifacts:
+docs/NCM_RETRIEVAL_EVALUATION_V2.json, docs/NCM_RETRIEVAL_ARCHITECTURE_V2.md,
+docs/GTIN_KB_AUDIT_V1.json, docs/AI_RESIDUAL_PLAN_V2.json.
+
+## AI semantic layer V1 (2026-09-21)
+
+`AI_SEMANTIC_LAYER_V1_IMPLEMENTED`. The deterministic gate was closed
+(OFFICIAL_RULE_RESOLVABLE_REMAINING=0, ERP_DATA_RESOLVABLE_REMAINING=0) and the Hub now
+owns a Product Knowledge Base plus a deduplicated AI work-item queue. entry_scope now
+admits MIXED_OR_DYNAMIC (CFOP_ENTRADA becomes operation-dependent; no CFOP invented).
+KB seeded read-only from all 5 databases: 30,801 global products (8,559 GTIN), 35,065
+observations, 163 surfaced knowledge conflicts, idempotent load. 850 residual questions
+deduped into 750 work items (20 answered by KB cache alone). Budget-guarded cost plan:
+$0.54 estimated for 730 items (authorized, $0 spent — no AI credential exists here).
+The 200-sample NCM benchmark proved the honest floor: candidate recall@10 = 5.5%
+(ERP commercial wording vs official legal wording), so AI selection over closed
+candidates is capped there — READY_FOR_AI_ASSISTED_WRITE_V2=false with candidate
+generation as the named prerequisite. AI_WRITE_ALLOWED=false in V1: AI results stay
+AI_CANDIDATE_VALIDATED, never Firebird writes. Artifacts:
+docs/PRODUCT_KNOWLEDGE_BASE_V1.md, docs/AI_SEMANTIC_LAYER_V1.md,
+docs/AI_EVALUATION_V1.json, docs/AI_COST_PLAN_V1.json, docs/AI_RESIDUAL_PLAN_V1.json.

 `FISCAL_DATA_FOUNDATION_MOSTLY_COMPLETE`. Deploy/pilot work remains paused
 per the current mission's scope. The operator manually placed the 6 official