gtiandClaude Sonnet 5
21e8282224
Missão 5: Fase 1 (Reference Data Registry): migration 0034 adds licence_terms, valid_from/ valid_to, file_size_bytes, raw_file_path, parse_errors, submitted_by/approved_by/ approved_at to the existing `datasets` table (additive only - the 9 pre-existing dataset kinds keep their current auto-activate behavior in central/app/datasets/service.py, completely untouched, so this changes nothing about the currently-running engine). New central/app/reference_registry.py is a separate, NEW ingestion workflow: register_draft() always lands as DRAFT (never auto-activates), diff_against_vigente() summarizes a draft against the current VIGENTE of the same kind for admin review, approve_draft()/reject_draft() are the only way to promote a DRAFT (refuses non-DRAFT rows and refuses a draft with recorded parse_errors). New admin screen "Tabelas oficiais" (/admin/reference-datasets, ADMIN only, CSRF, every decision audited as REFERENCE_DATASET_APPROVE/REJECT) lists the registry with a diff view and approve/reject buttons. New CLI (scripts/ingest_reference_dataset.py) downloads an official source (currently: NCM/Siscomex, no login/captcha), validates (minimum row count, a known-sample spot check), and registers a DRAFT - --dry-run never writes anything. P1 (NCM) validated for real: downloaded the live official Siscomex NCM JSON (2026-09-28, sha256 8a3bf89a..., 15,156 entries) with an identifiable User-Agent, one request. Its hash is IDENTICAL to the NCM dataset already ACTIVE in the Hub - confirms the currently loaded table is already current against the official source today. Raw file kept out of git (.gitignore: sources/raw/reference_registry/); manifest (URL, date, hash, size) recorded in docs/REFERENCE_DATASET_MANIFEST_V1.md, including P2-P6 sources still pending URL confirmation. Fase 5 (feasibility studies, no code): docs/VIABILIDADE_CCG_DFE_V1.md - CCG/SEFAZ GTIN lookup and Distribuição DF-e both require the client's own ICP-Brasil certificate; recommendation is NOT to implement now (would require either the Hub holding a client certificate - breaks the project's core security model - or the Agent making new outbound internet calls, against this mission's fixed rules). Reading the already-authorized purchase-NFe XML from disk via the Agent (path already stored in XML_MES) is recommended as the next concrete step instead: no certificate, no new internet call, opt-in per installation, result always tagged DECLARADO (never auto-promoted - consensus across purchase invoices is not fiscal truth). Open Food Facts stays identity-only, unchanged. 21 new tests, all passing. Full suite: 1147 passed, 1 skipped. No write to any Firebird database anywhere. No Agent code touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
36 lines
1.8 KiB
Python
36 lines
1.8 KiB
Python
"""Reference Data Registry (Missão 5, Fase 1): additive columns on datasets for licence terms,
|
|
vigência, file provenance, parse errors and admin approval tracking. Existing rows/kinds are
|
|
untouched - new nullable columns, no data migration, no behavior change for the 9 pre-existing
|
|
dataset kinds (their own service.py ingestion methods still auto-activate exactly as before).
|
|
|
|
Revision ID: 0034_reference_data_registry
|
|
Revises: 0032_contribuinte_ipi
|
|
|
|
Note: migration 0033_cfop_ippt_write_mode exists ONLY on feature/cfop-ippt-write, never merged to
|
|
master - this migration chains from 0032 (master's real head), not from 0033.
|
|
"""
|
|
from alembic import op
|
|
import sqlalchemy as sa
|
|
|
|
revision = "0034_reference_data_registry"
|
|
down_revision = "0032_contribuinte_ipi"
|
|
branch_labels = None
|
|
depends_on = None
|
|
|
|
|
|
def upgrade():
|
|
op.add_column("datasets", sa.Column("licence_terms", sa.Text(), nullable=True))
|
|
op.add_column("datasets", sa.Column("valid_from", sa.String(32), nullable=True))
|
|
op.add_column("datasets", sa.Column("valid_to", sa.String(32), nullable=True))
|
|
op.add_column("datasets", sa.Column("file_size_bytes", sa.Integer(), nullable=True))
|
|
op.add_column("datasets", sa.Column("raw_file_path", sa.Text(), nullable=True))
|
|
op.add_column("datasets", sa.Column("parse_errors", sa.JSON(), nullable=False, server_default="{}"))
|
|
op.add_column("datasets", sa.Column("submitted_by", sa.String(128), nullable=True))
|
|
op.add_column("datasets", sa.Column("approved_by", sa.String(128), nullable=True))
|
|
op.add_column("datasets", sa.Column("approved_at", sa.DateTime(timezone=True), nullable=True))
|
|
|
|
|
|
def downgrade():
|
|
for name in ("approved_at", "approved_by", "submitted_by", "parse_errors", "raw_file_path", "file_size_bytes", "valid_to", "valid_from", "licence_terms"):
|
|
op.drop_column("datasets", name)
|