64 lines
3.5 KiB
Markdown
64 lines
3.5 KiB
Markdown
# AI SEMANTIC LAYER V1
|
|
|
|
Central Hub module `central/app/ai/` + tables `ai_work_items` / `ai_runs`.
|
|
The AI is NOT a fiscal authority, never writes Firebird, never replaces official
|
|
datasets, and never answers company-level questions.
|
|
|
|
## Contract
|
|
|
|
- **Cache first**: KB exact/fingerprint hits never trigger AI (20 of 750 work items
|
|
were answered by the KB alone in the build pass).
|
|
- **Dedup**: pendings are grouped by knowledge needed, not per cell — 850 residual
|
|
questions collapsed into 750 work items (EX_TIPI 112, CEST 353, PRODUCT_ROLE 41,
|
|
INVALID_NCM 244).
|
|
- **Closed candidates only**: for every task the Hub first builds the official
|
|
candidate list (NCM top-K, CEST links for the NCM, TIPI EX rows). A provider may
|
|
only answer with a candidate code or `NONE`; `result_from_dict` rejects anything
|
|
else (`SemanticOutputError`). Providers never see an open "what is the NCM?" question.
|
|
- **Structured output** (strict JSON schema): task_type, input_fingerprint,
|
|
selected_candidate, confidence, reason, candidate_ids_seen, model, prompt_version,
|
|
created_at, dataset_versions, engine_version.
|
|
- **Provenance**: every persisted AI result is `source=AI` (or
|
|
`AI_PLUS_OFFICIAL_VALIDATION` after passing the official validator), with model,
|
|
prompt_version, dataset_versions — versioned history, never overwritten.
|
|
- **Forbidden for AI**: ST retention (ST_WITHHELD), origin (AI_ORIGIN_AUTO_APPLY=false,
|
|
at most NEEDS_EVIDENCE), company profile questions (PIS/COFINS profile, entry_scope,
|
|
ICMS profile, store_ipi_profile, CRT, UF), CFOP invention, majority conflict resolution.
|
|
- **Budget guard**: `BudgetGuard` refuses execution above `AI_MAX_RUN_COST_USD`;
|
|
no fallback to a more expensive model. Credentials live only in the Hub
|
|
(`AI_PROVIDER`, `AI_MODEL`, `AI_MAX_BATCH_SIZE`, `AI_MAX_RUN_COST_USD`,
|
|
`AI_MIN_CONFIDENCE`); the Windows Agent never sees them.
|
|
- **AI_WRITE_ALLOWED=false in V1**: validated AI results stay
|
|
`AI_CANDIDATE_VALIDATED`; no Firebird write is generated from AI output.
|
|
|
|
## Provider abstraction
|
|
|
|
`AISemanticProvider` (select / estimate). Shipped: `BaselineProvider`
|
|
(deterministic token-overlap top-1, zero cost) used by the evaluation harness and as
|
|
the mandatory no-credential fallback. Real providers plug in without touching the
|
|
fiscal engine.
|
|
|
|
## Evaluation result (docs/AI_EVALUATION_V1.json, 200 samples, 5 segments)
|
|
|
|
Baseline floor: EXACT_MATCH 2/200 (1.0%), NONE_WHEN_KNOWN 34, WRONG_SELECTION 164,
|
|
INVALID_OUTPUT 0, cost $0. RECALL@10=5.5%, RECALL@50=7.5%
|
|
(docs/runs/ai_evaluation_k50.json).
|
|
|
|
**Root-cause finding**: ERP commercial descriptions ("CREME DE TRATAMENTO NOVEX
|
|
QUERATINA 1KG") share almost no vocabulary with official legal NCM descriptions
|
|
("Outros preparações para uso dos cabelos..."), so the deterministic candidate
|
|
generator cannot surface the true NCM in top-K — the bottleneck is candidate
|
|
recall, not provider selection. Since AI selection is confined to the candidate
|
|
list, any provider would be capped near this floor.
|
|
|
|
**Consequence**: READY_FOR_AI_ASSISTED_WRITE_V2=false. V2 prerequisite: semantic
|
|
candidate generation (e.g. AI-proposed chapter/heading validated against the official
|
|
NCM tree by prefix — still closed-world), then re-run this benchmark until accuracy
|
|
and cost are jointly healthy.
|
|
|
|
## Cost plan (docs/AI_COST_PLAN_V1.json)
|
|
|
|
730 unique AI work items (20 KB hits excluded), 405,700 input / 66,540 output tokens,
|
|
estimated $0.54 against a $5.00 budget — authorized. Actual spend $0 (baseline only;
|
|
no AI credential exists in this environment).
|