chore: capture semantic pipeline baseline
This commit is contained in:
OpenCode
@@ -0,0 +1,187 @@
|
||||
# PRE-AI Local Completion V3 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Complete and verify the local semantic pipeline, knowledge preparation, and operator tooling without external AI or uncertain Firebird writes.
|
||||
|
||||
**Architecture:** Extend the existing `central.app.ai` contracts with a deterministic scenario provider and an ID-correlated batch harness. Add conservative, versioned KB/retrieval preparation artifacts and CLI-first operational commands that compose existing read-only mechanisms.
|
||||
|
||||
**Tech Stack:** Python 3.11+, SQLAlchemy, PostgreSQL/SQLite test engines, pytest, existing Alembic models, JSON/CSV artifacts, Docker only for existing local test execution.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Do not call paid/external AI APIs.
|
||||
- Keep `DETERMINISTIC_GATE_CLOSED=true`.
|
||||
- Keep `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0`.
|
||||
- Never use fake answers as semantic decisions.
|
||||
- Never write uncertain semantic values to Firebird.
|
||||
- Preserve tenant-specific fiscal context outside global KB identity.
|
||||
- Preserve all conflicts and historical evidence.
|
||||
- Do not add arbitrary fiscal rules.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Fake Provider Contract
|
||||
|
||||
**Files:**
|
||||
- Create: `central/app/ai/fake_provider.py`
|
||||
- Modify: `central/app/ai/provider.py`
|
||||
- Test: `central/tests/test_fake_provider.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `DeterministicFakeSemanticProvider.select_batch(items, scenario=None) -> list[dict]`
|
||||
- Scenarios: `CORRECT`, `WRONG`, `ABSTAIN`, `INVALID_JSON`, `MISSING_ITEM`, `DUPLICATE_ITEM`, `OUT_OF_CANDIDATE`, `TIMEOUT`, `429`, `500`.
|
||||
- The provider returns only test responses and never persists them.
|
||||
|
||||
- [ ] Write tests asserting every scenario and deterministic call counters.
|
||||
- [ ] Run `python -m pytest central/tests/test_fake_provider.py -q` and observe failure.
|
||||
- [ ] Implement the provider using the existing `LlmEvalItem`/validator contract.
|
||||
- [ ] Ensure failures use typed transport/contract errors and do not mutate input items.
|
||||
- [ ] Run the focused tests and confirm pass.
|
||||
|
||||
### Task 2: ID-Correlated Batch Harness
|
||||
|
||||
**Files:**
|
||||
- Create: `central/app/ai/batch_runner.py`
|
||||
- Test: `central/tests/test_batch_runner.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `run_batches(items, provider, batch_size, max_retries=2) -> BatchRunResult`.
|
||||
- `BatchRunResult` exposes `results_by_work_item_id`, `errors`, `provider_calls`, `retries`, and `usage`.
|
||||
|
||||
- [ ] Add failing tests for batch sizes 5, 10, and 20, shuffled responses, missing IDs, duplicates, invalid JSON, timeout, 429, 500, and partial failure.
|
||||
- [ ] Implement correlation by `work_item_id`, never array position.
|
||||
- [ ] Retry only transport failures and preserve failed item IDs.
|
||||
- [ ] Shrink/retry only according to explicit policy; never silently drop items.
|
||||
- [ ] Run focused tests.
|
||||
|
||||
### Task 3: Budget, Cache, and Dataset Drift Tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/app/ai/provider.py`
|
||||
- Modify: `central/app/ai/llm_provider.py`
|
||||
- Create: `central/app/ai/cache_policy.py`
|
||||
- Test: `central/tests/test_ai_controls_v3.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `cache_key(input_fingerprint, candidate_set_hash, model, prompt_version, dataset_versions) -> str`.
|
||||
- `classify_knowledge_freshness(record_versions, active_versions) -> VALID | NEEDS_REVALIDATION | CONFLICT`.
|
||||
|
||||
- [ ] Test sufficient and exceeded budgets, including zero provider calls on pre-call rejection.
|
||||
- [ ] Test input/output/cached token accounting and unknown prices.
|
||||
- [ ] Test cache hit/miss for each key component with `provider_calls=0` on hit.
|
||||
- [ ] Test dataset drift classification without deleting history.
|
||||
- [ ] Implement minimal helpers and connect them to existing cache/provider paths.
|
||||
- [ ] Run focused tests.
|
||||
|
||||
### Task 4: Offline E2E Simulation
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/scripts/run_real_llm_ncm_eval.py`
|
||||
- Create: `central/scripts/run_offline_ai_simulation.py`
|
||||
- Test: `central/tests/test_offline_ai_simulation.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `run_offline_ai_simulation(output_dir, scenario, batch_sizes=(5,10,20)) -> dict`.
|
||||
- Output explicitly marks fake results as plumbing-only and `FIREBIRD_AI_WRITES=0`.
|
||||
|
||||
- [ ] Test GOLD -> gate -> SILVER/NOISY -> residual plan sequencing.
|
||||
- [ ] Test that fake selections never enter Firebird or KB semantic truth tables.
|
||||
- [ ] Implement the command using final existing work-item/candidate contracts.
|
||||
- [ ] Emit sanitized metrics and zero-write evidence.
|
||||
- [ ] Run the command and focused tests.
|
||||
|
||||
### Task 5: Canonical KB and Conflict Audit
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/app/models.py`
|
||||
- Create: `central/alembic/versions/0015_kb_canonical_identity.py`
|
||||
- Create: `central/app/ai/kb_consolidation.py`
|
||||
- Create: `central/scripts/audit_kb_conflicts.py`
|
||||
- Create: `central/tests/test_kb_consolidation.py`
|
||||
- Generate: `docs/KB_CONFLICT_AUDIT_V1.json`
|
||||
|
||||
**Interfaces:**
|
||||
- `canonical_identity(observation) -> str` using GTIN first, then conservative fingerprint fields.
|
||||
- `classify_conflict(observations) -> str` using the six requested classifications.
|
||||
- `audit_conflicts(observations) -> dict` with counts, decisions, and evidence.
|
||||
|
||||
- [ ] Test GTIN-equivalent abbreviation merges and incompatible same-GTIN conflicts.
|
||||
- [ ] Test packaging variations remain distinct or explicitly classified.
|
||||
- [ ] Add additive canonical identity without promoting tenant fiscal fields.
|
||||
- [ ] Audit all available KB conflicts and generate the JSON artifact.
|
||||
- [ ] Run focused tests and validate artifact counts.
|
||||
|
||||
### Task 6: Retrieval V3 Evaluation and Candidate Preparation
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/app/ai/retrieval.py`
|
||||
- Create: `central/app/ai/retrieval_v3.py`
|
||||
- Create: `central/scripts/run_retrieval_eval_v3.py`
|
||||
- Create: `central/scripts/prepare_ai_candidates.py`
|
||||
- Test: `central/tests/test_retrieval_v3.py`
|
||||
- Generate: `docs/NCM_RETRIEVAL_EVALUATION_V3.json`
|
||||
|
||||
**Interfaces:**
|
||||
- `retrieval_v3_candidates(corpus, index, description, metadata, k=20) -> list`.
|
||||
- `prepare_candidate_set(work_item, active_datasets) -> dict` with normalized product, top-20, scores, official descriptions, hierarchy, neighbors, GTIN evidence, and input fingerprint.
|
||||
|
||||
- [ ] Add holdout tests for BM25/TF-IDF/character/rarity/hierarchy/KB/GTIN/group evidence.
|
||||
- [ ] Compare V3 to V2 and activate only if GOLD recall@20 is at least 0.954 and no required metric regresses.
|
||||
- [ ] Prepare NCM invalid, CEST, EX_TIPI, and PRODUCT_ROLE payloads without making semantic decisions.
|
||||
- [ ] Persist versioned candidate artifacts idempotently.
|
||||
- [ ] Run retrieval evaluation and focused tests.
|
||||
|
||||
### Task 7: Payload Optimization and Final Phase-1 Rerun
|
||||
|
||||
**Files:**
|
||||
- Create: `central/app/ai/payloads.py`
|
||||
- Modify: `central/scripts/run_offline_ai_simulation.py`
|
||||
- Test: `central/tests/test_payloads.py`
|
||||
- Generate: `docs/AI_PAYLOAD_OPTIMIZATION_V1.json`
|
||||
|
||||
- [ ] Test compaction removes only redundant repeated evidence and preserves candidate recall.
|
||||
- [ ] Measure before/after token estimates for all 511 selection items.
|
||||
- [ ] Rerun Phase 1 simulation against final work items and candidate sets.
|
||||
- [ ] Confirm fake outputs remain non-authoritative and zero-write.
|
||||
|
||||
### Task 8: Guided Company and Group Questionnaires
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/answer_company_context.py`
|
||||
- Create: `scripts/answer_group_questions.py`
|
||||
- Create: `central/app/ai/questionnaire.py`
|
||||
- Test: `central/tests/test_questionnaire.py`
|
||||
|
||||
- [ ] Test validation, review-before-import, cancel-without-write, JSON output, and CSV output.
|
||||
- [ ] Implement one-question-at-a-time Portuguese prompts with affected products/cells/options.
|
||||
- [ ] Group questions by company, group, subgroup, NCM, question, and consequence; show examples.
|
||||
- [ ] Enforce `ENTRY_SCOPE` values and document `MIXED_OR_DYNAMIC` behavior.
|
||||
- [ ] Implement `STORE_IPI_PROFILE` as explicit choice plus preview/rollback plan only.
|
||||
|
||||
### Task 9: Rollback Tooling and Full Local Validation
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/rollback_field.py`
|
||||
- Create: `scripts/rollback_run.py`
|
||||
- Create: `scripts/run_local_validation.py`
|
||||
- Test: `tests/test_rollback_run.py`
|
||||
- Test: `tests/test_local_validation.py`
|
||||
|
||||
- [ ] Test preview fields: run_id, database, field, product, before, current, target, affected rows.
|
||||
- [ ] Enforce `current == expected_after`; emit `CONFLICT` and never overwrite drift.
|
||||
- [ ] Compose preflight, sources, company context, KB, deterministic engine, residual classification, AI-ready preparation, and reports.
|
||||
- [ ] Make repeated runs idempotent and assert no semantic Firebird writes.
|
||||
|
||||
### Task 10: Health Report and Final Verification
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/generate_local_health.py`
|
||||
- Generate: `docs/LOCAL_PRODUCT_HEALTH_V1.json`
|
||||
- Modify: `docs/AI_PROJECT_STATE.md`
|
||||
|
||||
- [ ] Generate all requested health fields with measured values, not placeholders.
|
||||
- [ ] Run Phase-1 E2E again using final Phase-2 work items/candidates.
|
||||
- [ ] Run `python -m pytest tests -q` and `python -m pytest central/tests -q`.
|
||||
- [ ] Run `git diff --check` and verify no API key or token appears in tracked artifacts.
|
||||
- [ ] Report remaining blockers precisely: external Z.ai balance/credential, missing source files, unresolved human decisions, and any measurable failures.
|
||||
Reference in New Issue
Block a user