feat: Reference Data Registry (Fase 1) + P1 NCM validated + feasibility studies (Fase 5)

Missão 5:

Fase 1 (Reference Data Registry): migration 0034 adds licence_terms, valid_from/
valid_to, file_size_bytes, raw_file_path, parse_errors, submitted_by/approved_by/
approved_at to the existing `datasets` table (additive only - the 9 pre-existing
dataset kinds keep their current auto-activate behavior in
central/app/datasets/service.py, completely untouched, so this changes nothing
about the currently-running engine). New central/app/reference_registry.py is a
separate, NEW ingestion workflow: register_draft() always lands as DRAFT (never
auto-activates), diff_against_vigente() summarizes a draft against the current
VIGENTE of the same kind for admin review, approve_draft()/reject_draft() are
the only way to promote a DRAFT (refuses non-DRAFT rows and refuses a draft
with recorded parse_errors). New admin screen "Tabelas oficiais"
(/admin/reference-datasets, ADMIN only, CSRF, every decision audited as
REFERENCE_DATASET_APPROVE/REJECT) lists the registry with a diff view and
approve/reject buttons. New CLI (scripts/ingest_reference_dataset.py) downloads
an official source (currently: NCM/Siscomex, no login/captcha), validates
(minimum row count, a known-sample spot check), and registers a DRAFT - --dry-run
never writes anything.

P1 (NCM) validated for real: downloaded the live official Siscomex NCM JSON
(2026-09-28, sha256 8a3bf89a..., 15,156 entries) with an identifiable User-Agent,
one request. Its hash is IDENTICAL to the NCM dataset already ACTIVE in the Hub -
confirms the currently loaded table is already current against the official
source today. Raw file kept out of git (.gitignore:
sources/raw/reference_registry/); manifest (URL, date, hash, size) recorded in
docs/REFERENCE_DATASET_MANIFEST_V1.md, including P2-P6 sources still pending URL
confirmation.

Fase 5 (feasibility studies, no code): docs/VIABILIDADE_CCG_DFE_V1.md - CCG/SEFAZ
GTIN lookup and Distribuição DF-e both require the client's own ICP-Brasil
certificate; recommendation is NOT to implement now (would require either the
Hub holding a client certificate - breaks the project's core security model - or
the Agent making new outbound internet calls, against this mission's fixed
rules). Reading the already-authorized purchase-NFe XML from disk via the Agent
(path already stored in XML_MES) is recommended as the next concrete step
instead: no certificate, no new internet call, opt-in per installation, result
always tagged DECLARADO (never auto-promoted - consensus across purchase
invoices is not fiscal truth). Open Food Facts stays identity-only, unchanged.

21 new tests, all passing. Full suite: 1147 passed, 1 skipped.

No write to any Firebird database anywhere. No Agent code touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

central/app/web.py

@@ -25,6 +25,7 @@ from .audit import record_audit
 from .enrichment import kb as kb_mod, status as enrichment_status
 from .fiscal_context_ui import build_fiscal_context_view, set_fiscal_context_decision
 from .enrichment.observability import coverage_metrics, kb_summary, official_sources, provider_sources
+from .reference_registry import RegistryError, approve_draft, diff_against_vigente, list_registry, reject_draft
 
 
 def router(session_factory):
@@ -166,6 +167,53 @@ def router(session_factory):
         try: return render(request, "dashboard.html", {"summary": dashboard_summary(session), "auth": auth_data, "kb_q": kb_q or "", "is_admin": _is_privileged(auth_data[1]), "csrf_token": request.cookies.get("gti_csrf", ""), "sources": provider_sources(session), "official": official_sources(session), "enrich_metrics": coverage_metrics(session), "kb": kb_summary(session, kb_q, kb_page)})
         finally: session.close()
 
+    @routes.get("/admin/reference-datasets", response_class=HTMLResponse)
+    def reference_datasets_page(request: Request):
+        """Missão 5 / Fase 1.2: 'Tabelas oficiais' - lista o Reference Data Registry (DRAFT/VIGENTE/
+        OBSOLETO/REJECTED), com diff contra o VIGENTE atual para cada DRAFT. ADMIN only."""
+        session, auth_data, redirect = page_auth(request)
+        if redirect: return redirect
+        try:
+            if not _is_privileged(auth_data[1]):
+                return HTMLResponse("Acesso restrito a administradores.", status_code=403)
+            rows = list_registry(session)
+            view = []
+            for r in rows:
+                item = {"id": r.id, "kind": r.kind, "provider": r.provider, "version_label": r.version_label, "status": r.status,
+                       "record_count": r.record_count, "downloaded_at": format_ui_datetime(r.downloaded_at), "source_url": r.source_url,
+                       "licence_terms": r.licence_terms, "parse_errors": r.parse_errors, "submitted_by": r.submitted_by,
+                       "approved_by": r.approved_by, "diff": None}
+                if r.status == "DRAFT":
+                    item["diff"] = diff_against_vigente(session, r)
+                view.append(item)
+            return render(request, "reference_datasets.html", {"auth": auth_data, "csrf_token": request.cookies.get("gti_csrf", ""), "is_admin": True, "rows": view})
+        finally: session.close()
+
+    @routes.post("/admin/reference-datasets/{dataset_id}/decide")
+    async def decide_reference_dataset(request: Request, dataset_id: int):
+        session, auth_data, redirect = page_auth(request)
+        if redirect: return redirect
+        try:
+            values = await form_values(request); web_session, admin = auth_data
+            require_csrf(web_session, values.get("csrf_token"))
+            if not _is_privileged(admin):
+                return _deny(session, admin, None, "reference_dataset:decide", str(dataset_id))
+            action = (values.get("action") or "").strip()
+            try:
+                if action == "approve":
+                    dataset = approve_draft(session, dataset_id, admin_name=admin.display_name or admin.username)
+                elif action == "reject":
+                    dataset = reject_draft(session, dataset_id, admin_name=admin.display_name or admin.username)
+                else:
+                    return HTMLResponse("ação inválida", status_code=422)
+            except RegistryError as error:
+                return HTMLResponse(str(error), status_code=422)
+            record_audit(session, admin_user_id=admin.id, tenant_id=None, action=f"REFERENCE_DATASET_{action.upper()}", target_type="dataset", target_id=str(dataset_id),
+                         context={"kind": dataset.kind, "version_label": dataset.version_label, "operator": admin.display_name or admin.username}, severity="WARNING")
+            session.commit()
+            return RedirectResponse("/admin/reference-datasets", status_code=303)
+        finally: session.close()
+
     @routes.get("/agents", response_class=HTMLResponse)
     def agents(request: Request, page: int = 1, search: str | None = None, health: str | None = None, sort: str = "display_name", scope: str = "active", version: str | None = None):
         session, auth_data, redirect = page_auth(request)