feat: Reference Data Registry (Fase 1) + P1 NCM validated + feasibility studies (Fase 5)

Missão 5:

Fase 1 (Reference Data Registry): migration 0034 adds licence_terms, valid_from/
valid_to, file_size_bytes, raw_file_path, parse_errors, submitted_by/approved_by/
approved_at to the existing `datasets` table (additive only - the 9 pre-existing
dataset kinds keep their current auto-activate behavior in
central/app/datasets/service.py, completely untouched, so this changes nothing
about the currently-running engine). New central/app/reference_registry.py is a
separate, NEW ingestion workflow: register_draft() always lands as DRAFT (never
auto-activates), diff_against_vigente() summarizes a draft against the current
VIGENTE of the same kind for admin review, approve_draft()/reject_draft() are
the only way to promote a DRAFT (refuses non-DRAFT rows and refuses a draft
with recorded parse_errors). New admin screen "Tabelas oficiais"
(/admin/reference-datasets, ADMIN only, CSRF, every decision audited as
REFERENCE_DATASET_APPROVE/REJECT) lists the registry with a diff view and
approve/reject buttons. New CLI (scripts/ingest_reference_dataset.py) downloads
an official source (currently: NCM/Siscomex, no login/captcha), validates
(minimum row count, a known-sample spot check), and registers a DRAFT - --dry-run
never writes anything.

P1 (NCM) validated for real: downloaded the live official Siscomex NCM JSON
(2026-09-28, sha256 8a3bf89a..., 15,156 entries) with an identifiable User-Agent,
one request. Its hash is IDENTICAL to the NCM dataset already ACTIVE in the Hub -
confirms the currently loaded table is already current against the official
source today. Raw file kept out of git (.gitignore:
sources/raw/reference_registry/); manifest (URL, date, hash, size) recorded in
docs/REFERENCE_DATASET_MANIFEST_V1.md, including P2-P6 sources still pending URL
confirmation.

Fase 5 (feasibility studies, no code): docs/VIABILIDADE_CCG_DFE_V1.md - CCG/SEFAZ
GTIN lookup and Distribuição DF-e both require the client's own ICP-Brasil
certificate; recommendation is NOT to implement now (would require either the
Hub holding a client certificate - breaks the project's core security model - or
the Agent making new outbound internet calls, against this mission's fixed
rules). Reading the already-authorized purchase-NFe XML from disk via the Agent
(path already stored in XML_MES) is recommended as the next concrete step
instead: no certificate, no new internet call, opt-in per installation, result
always tagged DECLARADO (never auto-promoted - consensus across purchase
invoices is not fiscal truth). Open Food Facts stays identity-only, unchanged.

21 new tests, all passing. Full suite: 1147 passed, 1 skipped.

No write to any Firebird database anywhere. No Agent code touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

central/tests/test_reference_datasets_page.py

@@ -0,0 +1,80 @@
+"""Missão 5 / Fase 1.2: 'Tabelas oficiais' admin screen - list, diff, approve/reject (CSRF, audited)."""
+from sqlalchemy import select
+
+from central.app.models import AuditEvent, Dataset
+from central.app.reference_registry import register_draft
+from central.tests.test_control_panel_pages import _panel_client
+from central.tests.test_rtc_security import _become, _post
+
+
+def _seed_draft(client, **kw):
+    with client._session_factory() as s:
+        d = register_draft(s, kind=kw.get("kind", "NCM"), provider=kw.get("provider", "Siscomex"), source_url="https://x",
+                           sha256=kw.get("sha256", "a" * 64), version_label=kw.get("version_label", "v1"), record_count=kw.get("record_count", 100),
+                           parse_errors=kw.get("parse_errors") or {})
+        return d.id
+
+
+def test_page_requires_admin():
+    client, agent_id, _ = _panel_client()
+    _become(client, "PADRAO")
+    resp = client.get("/admin/reference-datasets", follow_redirects=False)
+    assert resp.status_code == 403
+
+
+def test_admin_sees_the_draft_and_its_diff():
+    client, agent_id, _ = _panel_client()
+    _seed_draft(client)
+    _become(client, "ADMIN")
+    page = client.get("/admin/reference-datasets").text
+    assert "Tabelas oficiais" in page and "NCM" in page and "DRAFT" in page
+    assert "Primeira vers" in page  # first version of this kind, no vigente to diff against
+
+
+def test_approve_promotes_and_is_audited():
+    client, agent_id, _ = _panel_client()
+    dataset_id = _seed_draft(client)
+    _become(client, "ADMIN")
+    resp = _post(client, f"/admin/reference-datasets/{dataset_id}/decide", action="approve")
+    assert resp.status_code == 303
+    with client._session_factory() as s:
+        assert s.get(Dataset, dataset_id).status == "VIGENTE"
+        acts = [e.action for e in s.scalars(select(AuditEvent))]
+        assert "REFERENCE_DATASET_APPROVE" in acts
+
+
+def test_reject_never_promotes():
+    client, agent_id, _ = _panel_client()
+    dataset_id = _seed_draft(client)
+    _become(client, "ADMIN")
+    assert _post(client, f"/admin/reference-datasets/{dataset_id}/decide", action="reject").status_code == 303
+    with client._session_factory() as s:
+        assert s.get(Dataset, dataset_id).status == "REJECTED"
+
+
+def test_approve_with_parse_errors_is_refused_with_422():
+    client, agent_id, _ = _panel_client()
+    dataset_id = _seed_draft(client, parse_errors={"MINIMUM_ENTRIES": "too few"})
+    _become(client, "ADMIN")
+    resp = _post(client, f"/admin/reference-datasets/{dataset_id}/decide", action="approve")
+    assert resp.status_code == 422
+    with client._session_factory() as s:
+        assert s.get(Dataset, dataset_id).status == "DRAFT"
+
+
+def test_non_admin_cannot_decide_and_it_is_audited_as_a_denial():
+    client, agent_id, _ = _panel_client()
+    dataset_id = _seed_draft(client)
+    _become(client, "PADRAO")
+    resp = _post(client, f"/admin/reference-datasets/{dataset_id}/decide", action="approve")
+    assert resp.status_code == 403
+    with client._session_factory() as s:
+        assert s.get(Dataset, dataset_id).status == "DRAFT"
+
+
+def test_wrong_csrf_is_refused():
+    client, agent_id, _ = _panel_client()
+    dataset_id = _seed_draft(client)
+    _become(client, "ADMIN")
+    resp = client.post(f"/admin/reference-datasets/{dataset_id}/decide", data={"csrf_token": "bad", "action": "approve"}, follow_redirects=False)
+    assert resp.status_code in (400, 403)