chore: capture semantic pipeline baseline
This commit is contained in:
OpenCode
@@ -0,0 +1,187 @@
|
||||
# PRE-AI Local Completion V3 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Complete and verify the local semantic pipeline, knowledge preparation, and operator tooling without external AI or uncertain Firebird writes.
|
||||
|
||||
**Architecture:** Extend the existing `central.app.ai` contracts with a deterministic scenario provider and an ID-correlated batch harness. Add conservative, versioned KB/retrieval preparation artifacts and CLI-first operational commands that compose existing read-only mechanisms.
|
||||
|
||||
**Tech Stack:** Python 3.11+, SQLAlchemy, PostgreSQL/SQLite test engines, pytest, existing Alembic models, JSON/CSV artifacts, Docker only for existing local test execution.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Do not call paid/external AI APIs.
|
||||
- Keep `DETERMINISTIC_GATE_CLOSED=true`.
|
||||
- Keep `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0`.
|
||||
- Never use fake answers as semantic decisions.
|
||||
- Never write uncertain semantic values to Firebird.
|
||||
- Preserve tenant-specific fiscal context outside global KB identity.
|
||||
- Preserve all conflicts and historical evidence.
|
||||
- Do not add arbitrary fiscal rules.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Fake Provider Contract
|
||||
|
||||
**Files:**
|
||||
- Create: `central/app/ai/fake_provider.py`
|
||||
- Modify: `central/app/ai/provider.py`
|
||||
- Test: `central/tests/test_fake_provider.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `DeterministicFakeSemanticProvider.select_batch(items, scenario=None) -> list[dict]`
|
||||
- Scenarios: `CORRECT`, `WRONG`, `ABSTAIN`, `INVALID_JSON`, `MISSING_ITEM`, `DUPLICATE_ITEM`, `OUT_OF_CANDIDATE`, `TIMEOUT`, `429`, `500`.
|
||||
- The provider returns only test responses and never persists them.
|
||||
|
||||
- [ ] Write tests asserting every scenario and deterministic call counters.
|
||||
- [ ] Run `python -m pytest central/tests/test_fake_provider.py -q` and observe failure.
|
||||
- [ ] Implement the provider using the existing `LlmEvalItem`/validator contract.
|
||||
- [ ] Ensure failures use typed transport/contract errors and do not mutate input items.
|
||||
- [ ] Run the focused tests and confirm pass.
|
||||
|
||||
### Task 2: ID-Correlated Batch Harness
|
||||
|
||||
**Files:**
|
||||
- Create: `central/app/ai/batch_runner.py`
|
||||
- Test: `central/tests/test_batch_runner.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `run_batches(items, provider, batch_size, max_retries=2) -> BatchRunResult`.
|
||||
- `BatchRunResult` exposes `results_by_work_item_id`, `errors`, `provider_calls`, `retries`, and `usage`.
|
||||
|
||||
- [ ] Add failing tests for batch sizes 5, 10, and 20, shuffled responses, missing IDs, duplicates, invalid JSON, timeout, 429, 500, and partial failure.
|
||||
- [ ] Implement correlation by `work_item_id`, never array position.
|
||||
- [ ] Retry only transport failures and preserve failed item IDs.
|
||||
- [ ] Shrink/retry only according to explicit policy; never silently drop items.
|
||||
- [ ] Run focused tests.
|
||||
|
||||
### Task 3: Budget, Cache, and Dataset Drift Tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/app/ai/provider.py`
|
||||
- Modify: `central/app/ai/llm_provider.py`
|
||||
- Create: `central/app/ai/cache_policy.py`
|
||||
- Test: `central/tests/test_ai_controls_v3.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `cache_key(input_fingerprint, candidate_set_hash, model, prompt_version, dataset_versions) -> str`.
|
||||
- `classify_knowledge_freshness(record_versions, active_versions) -> VALID | NEEDS_REVALIDATION | CONFLICT`.
|
||||
|
||||
- [ ] Test sufficient and exceeded budgets, including zero provider calls on pre-call rejection.
|
||||
- [ ] Test input/output/cached token accounting and unknown prices.
|
||||
- [ ] Test cache hit/miss for each key component with `provider_calls=0` on hit.
|
||||
- [ ] Test dataset drift classification without deleting history.
|
||||
- [ ] Implement minimal helpers and connect them to existing cache/provider paths.
|
||||
- [ ] Run focused tests.
|
||||
|
||||
### Task 4: Offline E2E Simulation
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/scripts/run_real_llm_ncm_eval.py`
|
||||
- Create: `central/scripts/run_offline_ai_simulation.py`
|
||||
- Test: `central/tests/test_offline_ai_simulation.py`
|
||||
|
||||
**Interfaces:**
|
||||
- `run_offline_ai_simulation(output_dir, scenario, batch_sizes=(5,10,20)) -> dict`.
|
||||
- Output explicitly marks fake results as plumbing-only and `FIREBIRD_AI_WRITES=0`.
|
||||
|
||||
- [ ] Test GOLD -> gate -> SILVER/NOISY -> residual plan sequencing.
|
||||
- [ ] Test that fake selections never enter Firebird or KB semantic truth tables.
|
||||
- [ ] Implement the command using final existing work-item/candidate contracts.
|
||||
- [ ] Emit sanitized metrics and zero-write evidence.
|
||||
- [ ] Run the command and focused tests.
|
||||
|
||||
### Task 5: Canonical KB and Conflict Audit
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/app/models.py`
|
||||
- Create: `central/alembic/versions/0015_kb_canonical_identity.py`
|
||||
- Create: `central/app/ai/kb_consolidation.py`
|
||||
- Create: `central/scripts/audit_kb_conflicts.py`
|
||||
- Create: `central/tests/test_kb_consolidation.py`
|
||||
- Generate: `docs/KB_CONFLICT_AUDIT_V1.json`
|
||||
|
||||
**Interfaces:**
|
||||
- `canonical_identity(observation) -> str` using GTIN first, then conservative fingerprint fields.
|
||||
- `classify_conflict(observations) -> str` using the six requested classifications.
|
||||
- `audit_conflicts(observations) -> dict` with counts, decisions, and evidence.
|
||||
|
||||
- [ ] Test GTIN-equivalent abbreviation merges and incompatible same-GTIN conflicts.
|
||||
- [ ] Test packaging variations remain distinct or explicitly classified.
|
||||
- [ ] Add additive canonical identity without promoting tenant fiscal fields.
|
||||
- [ ] Audit all available KB conflicts and generate the JSON artifact.
|
||||
- [ ] Run focused tests and validate artifact counts.
|
||||
|
||||
### Task 6: Retrieval V3 Evaluation and Candidate Preparation
|
||||
|
||||
**Files:**
|
||||
- Modify: `central/app/ai/retrieval.py`
|
||||
- Create: `central/app/ai/retrieval_v3.py`
|
||||
- Create: `central/scripts/run_retrieval_eval_v3.py`
|
||||
- Create: `central/scripts/prepare_ai_candidates.py`
|
||||
- Test: `central/tests/test_retrieval_v3.py`
|
||||
- Generate: `docs/NCM_RETRIEVAL_EVALUATION_V3.json`
|
||||
|
||||
**Interfaces:**
|
||||
- `retrieval_v3_candidates(corpus, index, description, metadata, k=20) -> list`.
|
||||
- `prepare_candidate_set(work_item, active_datasets) -> dict` with normalized product, top-20, scores, official descriptions, hierarchy, neighbors, GTIN evidence, and input fingerprint.
|
||||
|
||||
- [ ] Add holdout tests for BM25/TF-IDF/character/rarity/hierarchy/KB/GTIN/group evidence.
|
||||
- [ ] Compare V3 to V2 and activate only if GOLD recall@20 is at least 0.954 and no required metric regresses.
|
||||
- [ ] Prepare NCM invalid, CEST, EX_TIPI, and PRODUCT_ROLE payloads without making semantic decisions.
|
||||
- [ ] Persist versioned candidate artifacts idempotently.
|
||||
- [ ] Run retrieval evaluation and focused tests.
|
||||
|
||||
### Task 7: Payload Optimization and Final Phase-1 Rerun
|
||||
|
||||
**Files:**
|
||||
- Create: `central/app/ai/payloads.py`
|
||||
- Modify: `central/scripts/run_offline_ai_simulation.py`
|
||||
- Test: `central/tests/test_payloads.py`
|
||||
- Generate: `docs/AI_PAYLOAD_OPTIMIZATION_V1.json`
|
||||
|
||||
- [ ] Test compaction removes only redundant repeated evidence and preserves candidate recall.
|
||||
- [ ] Measure before/after token estimates for all 511 selection items.
|
||||
- [ ] Rerun Phase 1 simulation against final work items and candidate sets.
|
||||
- [ ] Confirm fake outputs remain non-authoritative and zero-write.
|
||||
|
||||
### Task 8: Guided Company and Group Questionnaires
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/answer_company_context.py`
|
||||
- Create: `scripts/answer_group_questions.py`
|
||||
- Create: `central/app/ai/questionnaire.py`
|
||||
- Test: `central/tests/test_questionnaire.py`
|
||||
|
||||
- [ ] Test validation, review-before-import, cancel-without-write, JSON output, and CSV output.
|
||||
- [ ] Implement one-question-at-a-time Portuguese prompts with affected products/cells/options.
|
||||
- [ ] Group questions by company, group, subgroup, NCM, question, and consequence; show examples.
|
||||
- [ ] Enforce `ENTRY_SCOPE` values and document `MIXED_OR_DYNAMIC` behavior.
|
||||
- [ ] Implement `STORE_IPI_PROFILE` as explicit choice plus preview/rollback plan only.
|
||||
|
||||
### Task 9: Rollback Tooling and Full Local Validation
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/rollback_field.py`
|
||||
- Create: `scripts/rollback_run.py`
|
||||
- Create: `scripts/run_local_validation.py`
|
||||
- Test: `tests/test_rollback_run.py`
|
||||
- Test: `tests/test_local_validation.py`
|
||||
|
||||
- [ ] Test preview fields: run_id, database, field, product, before, current, target, affected rows.
|
||||
- [ ] Enforce `current == expected_after`; emit `CONFLICT` and never overwrite drift.
|
||||
- [ ] Compose preflight, sources, company context, KB, deterministic engine, residual classification, AI-ready preparation, and reports.
|
||||
- [ ] Make repeated runs idempotent and assert no semantic Firebird writes.
|
||||
|
||||
### Task 10: Health Report and Final Verification
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/generate_local_health.py`
|
||||
- Generate: `docs/LOCAL_PRODUCT_HEALTH_V1.json`
|
||||
- Modify: `docs/AI_PROJECT_STATE.md`
|
||||
|
||||
- [ ] Generate all requested health fields with measured values, not placeholders.
|
||||
- [ ] Run Phase-1 E2E again using final Phase-2 work items/candidates.
|
||||
- [ ] Run `python -m pytest tests -q` and `python -m pytest central/tests -q`.
|
||||
- [ ] Run `git diff --check` and verify no API key or token appears in tracked artifacts.
|
||||
- [ ] Report remaining blockers precisely: external Z.ai balance/credential, missing source files, unresolved human decisions, and any measurable failures.
|
||||
@@ -0,0 +1,74 @@
|
||||
# PRE-AI Local Completion V3 Design
|
||||
|
||||
## Goal
|
||||
|
||||
Finish and verify every local, deterministic part of the semantic completion
|
||||
product without external LLM calls, Firebird semantic writes, deployment, or
|
||||
new fiscal rules. When credentials become available, only the real benchmark
|
||||
execution should remain.
|
||||
|
||||
## Scope and Order
|
||||
|
||||
1. Offline AI plumbing: deterministic fake provider, batch correlation,
|
||||
retries, budget enforcement, cache invalidation, and end-to-end dry-run.
|
||||
2. Knowledge base: canonical identity and conflict audit, conservative
|
||||
deduplication, retrieval V3 evaluation, final candidate preparation, and
|
||||
payload compaction.
|
||||
3. Operations: guided company/group questionnaires, rollback preview/guard,
|
||||
idempotent local validation, and a concise health artifact.
|
||||
|
||||
The phases are sequential. Phase 1 is rerun after Phase 2 with the final work
|
||||
items and candidate sets. No fake answer is persisted as semantic truth.
|
||||
|
||||
## Safety Invariants
|
||||
|
||||
- `DETERMINISTIC_GATE_CLOSED=true` remains unchanged.
|
||||
- `AI_WRITE_ALLOWED=false` and `FIREBIRD_AI_WRITES=0` are asserted by tests and
|
||||
reports.
|
||||
- External HTTP/API calls are forbidden in offline commands and tests.
|
||||
- Firebird access remains read-only for discovery; no uncertain semantic value
|
||||
is written to Firebird.
|
||||
- Tenant fiscal context is never promoted into global product identity.
|
||||
- Conflicts are retained and classified; automatic consolidation requires
|
||||
exact structural evidence and equivalent normalized descriptions.
|
||||
- `MIXED_OR_DYNAMIC` never produces a single invented `CFOP_ENTRADA`.
|
||||
- Rollback changes only rows whose current value equals the recorded expected
|
||||
post-write value.
|
||||
|
||||
## Architecture
|
||||
|
||||
The fake provider implements the existing semantic selection contract and is
|
||||
driven by named scenarios. A batch harness maps responses by `work_item_id`,
|
||||
validates candidate confinement, retries only transport failures, and records
|
||||
provider calls, token usage, and budget decisions. Cache keys include input,
|
||||
candidate set, model, prompt version, and all relevant dataset versions.
|
||||
|
||||
KB consolidation remains additive: `canonical_product_id` is a stable global
|
||||
identity for equivalent observations, while tenant-specific values remain in
|
||||
tenant context. Conflict audit and candidate preparation are persisted as
|
||||
versioned JSON artifacts or Hub records; no fiscal decision is inferred.
|
||||
|
||||
Operational tools are CLI-first and preview-first. They reuse the existing
|
||||
question and context persistence mechanisms, produce reviewable JSON/CSV, and
|
||||
are idempotent. The full validation command composes existing read-only
|
||||
scripts plus the new preparation/report steps.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Fake scenarios cover correct, wrong, abstain, invalid JSON, missing,
|
||||
duplicate, out-of-candidate, timeout, 429, and 500 responses.
|
||||
- Batch sizes 5, 10, and 20 are tested with out-of-order responses and
|
||||
partial/retry failures, always correlated by ID.
|
||||
- Budget overrun blocks before provider invocation; unknown prices remain
|
||||
explicitly unknown.
|
||||
- Cache hit performs zero provider calls; each key component independently
|
||||
causes a miss; dataset drift yields `NEEDS_REVALIDATION` or `CONFLICT`.
|
||||
- KB conflict audit contains all 163 baseline conflicts or an explicit
|
||||
measured current count, with no silent merge.
|
||||
- Retrieval V3 never lowers GOLD recall@20 below 0.954 on the holdout and only
|
||||
becomes active when the comparison artifact proves improvement or equal
|
||||
recall with better recall@1/MRR.
|
||||
- Final E2E simulation completes GOLD -> gate -> SILVER/NOISY -> residual plan
|
||||
with zero Firebird writes.
|
||||
- Full local validation is idempotent and emits `docs/LOCAL_PRODUCT_HEALTH_V1.json`.
|
||||
- Full test suite passes.
|
||||
Reference in New Issue
Block a user