feat: Reference Data Registry (Fase 1) + P1 NCM validated + feasibility studies (Fase 5)

Missão 5:

Fase 1 (Reference Data Registry): migration 0034 adds licence_terms, valid_from/
valid_to, file_size_bytes, raw_file_path, parse_errors, submitted_by/approved_by/
approved_at to the existing `datasets` table (additive only - the 9 pre-existing
dataset kinds keep their current auto-activate behavior in
central/app/datasets/service.py, completely untouched, so this changes nothing
about the currently-running engine). New central/app/reference_registry.py is a
separate, NEW ingestion workflow: register_draft() always lands as DRAFT (never
auto-activates), diff_against_vigente() summarizes a draft against the current
VIGENTE of the same kind for admin review, approve_draft()/reject_draft() are
the only way to promote a DRAFT (refuses non-DRAFT rows and refuses a draft
with recorded parse_errors). New admin screen "Tabelas oficiais"
(/admin/reference-datasets, ADMIN only, CSRF, every decision audited as
REFERENCE_DATASET_APPROVE/REJECT) lists the registry with a diff view and
approve/reject buttons. New CLI (scripts/ingest_reference_dataset.py) downloads
an official source (currently: NCM/Siscomex, no login/captcha), validates
(minimum row count, a known-sample spot check), and registers a DRAFT - --dry-run
never writes anything.

P1 (NCM) validated for real: downloaded the live official Siscomex NCM JSON
(2026-09-28, sha256 8a3bf89a..., 15,156 entries) with an identifiable User-Agent,
one request. Its hash is IDENTICAL to the NCM dataset already ACTIVE in the Hub -
confirms the currently loaded table is already current against the official
source today. Raw file kept out of git (.gitignore:
sources/raw/reference_registry/); manifest (URL, date, hash, size) recorded in
docs/REFERENCE_DATASET_MANIFEST_V1.md, including P2-P6 sources still pending URL
confirmation.

Fase 5 (feasibility studies, no code): docs/VIABILIDADE_CCG_DFE_V1.md - CCG/SEFAZ
GTIN lookup and Distribuição DF-e both require the client's own ICP-Brasil
certificate; recommendation is NOT to implement now (would require either the
Hub holding a client certificate - breaks the project's core security model - or
the Agent making new outbound internet calls, against this mission's fixed
rules). Reading the already-authorized purchase-NFe XML from disk via the Agent
(path already stored in XML_MES) is recommended as the next concrete step
instead: no certificate, no new internet call, opt-in per installation, result
always tagged DECLARADO (never auto-promoted - consensus across purchase
invoices is not fiscal truth). Open Food Facts stays identity-only, unchanged.

21 new tests, all passing. Full suite: 1147 passed, 1 skipped.

No write to any Firebird database anywhere. No Agent code touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

central/app/models.py

@@ -10,6 +10,11 @@ class Base(DeclarativeBase):
 
 
 class Dataset(Base):
+    """The Reference Data Registry (Missão 5): every official reference table the Hub knows about,
+    one row per ingested version. Pre-existing kinds (NCM, TIPI_IPI, CEST, ...) keep their original
+    auto-activate behavior in central/app/datasets/service.py, untouched by this mission - the new
+    DRAFT->admin-approves->ACTIVE workflow applies to datasets registered through the new
+    scripts/ingest_reference_dataset.py CLI / the admin "Tabelas oficiais" screen going forward."""
     __tablename__ = "datasets"
     __table_args__ = (UniqueConstraint("provider", "sha256", name="uq_dataset_provider_hash"),)
     id: Mapped[int] = mapped_column(primary_key=True)
@@ -21,8 +26,17 @@ class Dataset(Base):
     act: Mapped[str] = mapped_column(Text)
     source_url: Mapped[str] = mapped_column(Text)
     downloaded_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
-    status: Mapped[str] = mapped_column(String(32), index=True)
+    status: Mapped[str] = mapped_column(String(32), index=True)  # ACTIVE | SUPERSEDED (legacy kinds) | DRAFT | APPROVED | VIGENTE | OBSOLETO | REJECTED (new registry workflow)
     record_count: Mapped[int] = mapped_column(Integer, default=0)
+    licence_terms: Mapped[str | None] = mapped_column(Text, nullable=True)
+    valid_from: Mapped[str | None] = mapped_column(String(32), nullable=True)
+    valid_to: Mapped[str | None] = mapped_column(String(32), nullable=True)
+    file_size_bytes: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    raw_file_path: Mapped[str | None] = mapped_column(Text, nullable=True)  # local path only (sources/raw/... , gitignored) - never the file content in Postgres
+    parse_errors: Mapped[dict] = mapped_column(JSON, default=dict)
+    submitted_by: Mapped[str | None] = mapped_column(String(128), nullable=True)
+    approved_by: Mapped[str | None] = mapped_column(String(128), nullable=True)
+    approved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
 
 
 class NcmEntry(Base):