123 lines
11 KiB
Markdown
123 lines
11 KiB
Markdown
# SOURCE_REGISTRY_V2
|
||
|
||
Supersedes `SOURCE_REGISTRY_V1.md` (kept for history). Every dataset here
|
||
follows the same lifecycle: acquire → hash → parse → normalize → validate →
|
||
persist → smoke → `ACTIVE`. A failed reimport never silently replaces a
|
||
valid `ACTIVE` version (see `central/scripts/import_ba_st_mva.py`'s >10%
|
||
record-loss gate for a concrete example of that guard).
|
||
|
||
**Update**: the 9 domains this file originally listed as `FAILED` (external
|
||
acquisition blocker) were re-attempted after the operator manually placed
|
||
the official files that were blocked by the Portal NF-e/SPED session gate
|
||
into `sources/raw/bootstrap_v1/`. 6 of those 9 are now `ACTIVE`. The
|
||
remaining 3 (ICMS Monofásico, RTC cClassTrib, RTC Alíquotas CBS) plus all 8
|
||
PIS/COFINS Tabela 4.3.10–4.3.17 files are `MISSING_BOOTSTRAP_FILE` — no
|
||
file for them was provided, which is a different, more precise status than
|
||
"FAILED": it means acquisition is still blocked, not that something broke.
|
||
|
||
Every new file's domain was confirmed by its actual sheet names and header
|
||
row, never by filename alone (see "Identification evidence" below).
|
||
|
||
## Domains ACTIVE (preserved from before this session)
|
||
|
||
| DOMAIN | AUTHORITY | SOURCE | RAW_SHA256 (prefix) | PARSER_VERSION | RECORD_COUNT | STATUS |
|
||
|---|---|---|---|---|---|---|
|
||
| NCM | Receita Federal / Siscomex | Previously-fetched Siscomex JSON | `3abc0d2eb2501...` | ncm-v1 | 15156 | ACTIVE (preserved, no internet to re-verify) |
|
||
| TIPI_IPI | Receita Federal | `1 TIPI 2022 - Atualizada ADE 001-2026.xlsx` | `d155f1baafb40...` | tipi-xlsx-v1 | 11103 | ACTIVE (hash-verified unchanged) |
|
||
| CEST | CONFAZ Convênio ICMS 142/18 | Previously-fetched CONFAZ HTML | `d7fb7e8da0ace...` | cest-html-v1 | 1051 | ACTIVE (preserved) |
|
||
| BA_ST_MVA | SEFAZ-BA / RICMS-BA Anexo 1 | `decreto_2012_13780_ricms_anexo_1_vigente_2026.pdf` | `e07e908595...` | ba-st-mva-pdf-v1 | 303 | ACTIVE (re-imported, content identical) |
|
||
| ORIGEM_MERCADORIA / CST_ICMS / CSOSN / CST_PIS / CST_COFINS | Receita Federal / ENCAT (MOC 7.0 Anexo I) | `moc7-anexo-i-leiaute-e-rv.pdf` | `5eb4cf2010b1...` | moc7-tables-v1 | 9 / 11 / 10 / 33 / 33 | ACTIVE |
|
||
|
||
## Domains ACTIVE this session (bootstrap_v1 operator-supplied files)
|
||
|
||
| DOMAIN | AUTHORITY | FILE | RAW_SHA256 (prefix) | PARSER_VERSION | RECORD_COUNT | IDENTIFICATION EVIDENCE |
|
||
|---|---|---|---|---|---|---|
|
||
| CFOP | Portal Nacional NF-e, Informe Técnico 2023.002 v2.10 | `IT_2023_002_v2_10_Tabela_CFOP_Descrição.xlsx` (an identical PDF of the same IT is also present, kept as a cross-check copy, not separately imported) | `c95c91ed1644...` | cfop-xlsx-v1 | 619 | Sheet literally named `CFOP`; header row is `CFOP, Início de vigência, Fim de vigência, indNFe, indComunica, indTransp, indDevol, indRetor, indAnula, indRemes, indComb, indExcIBSCBS, Título do CFOP, Descrição` — the exact official field set, including the newer `indExcIBSCBS` (Reforma Tributária) column |
|
||
| FCP | Portal Nacional NF-e | `Tabela_Alíquotas_FCP_por_UF_2025-02_07.xlsx` | `3079d08816...` | fcp-xlsx-v1 | 27 (26 states + DF) | Sheet `FCP`; header `cUF, UF, Nome UF, Aliq_1, Aliq_2, Aliq_3, ., Observação`; every Brazilian UF present exactly once |
|
||
| ANP | Portal Nacional NF-e | `Tabela_Codigo_Produto_ANP-20241118.xlsx` | `8271a80007...` | anp-xlsx-v1 | 1031 | Sheet `cProdANP`; single-column header `cProdANP` — confirmed this really is the NF-e portal's validation code list (no description column exists in the official source; not invented) |
|
||
| NCM_UTRIB | Portal Nacional NF-e, Informe Técnico 2024.001 v2.40 | `IT2024.001v2.40 - Tabela NCM 2022 com Utrib_Comércio Exterior_vigência 01.10.26.xlsx` | `afc8bfc94c...` | ncm-utrib-xlsx-v1 | 10516 | Sheet `NCM X uTrib_Vig 1-10-2026`; header `NCM, Início de vigência da NCM, Fim de vigência da NCM, uTrib..., Descrição da uTrib...`; effective from 2026-10-01 (only version present — no history to preserve) |
|
||
| RTC_BIOFUEL_INDEX | Portal Nacional NF-e, Informe Técnico 2025.004 v1.20 | `IT 2025.004 v.1.20 - Tabela de Índice.xlsx` | `c49a01150...` | rtc-biofuel-index-xlsx-v1 | 23 (14 current + 9 historical) | Two sheets, `pBioObrigatorio` and `pBioObrigatorio_Historico`; header `cProdANP, Descrição, pBio (% Anidro Obrigatorio), Início de vigência, Fim de vigência`; both current and historical rows preserved (`is_historical` flag), forward-filling only the repeated `cProdANP`/`Descrição` a historical block shares with its preceding row — never inventing new values |
|
||
| RTC_CCREDPRES | Portal Nacional NF-e / LC 214/2025 | `cCredPres_2026-06-22.xlsx` | `65f246139...` | rtc-cred-pres-xlsx-v1 | 13 | Sheet `cCredPres`; header has 18 real official columns (`LC 214/2025`, `Apropria via NF?`, `ind_gCBSCredPres`, `Alíquota CBS`, `pAliqCredPresCBS`, `dIniVigCBS`, etc.) — all 18 preserved as distinct typed columns, not reduced to code+description |
|
||
|
||
Bug found and fixed mid-import: `rtc_cred_pres_entries.cclass_nota_referenciada`
|
||
was originally `VARCHAR(16)` (assumed always a numeric class code); several
|
||
official rows carry free text there instead (e.g. `"cClass de alíquota
|
||
zero"`), which Postgres rejected with a truncation error on insert.
|
||
Migration `0009` widens it to `Text`; no data was lost or silently truncated
|
||
— the import failed closed and was retried after the fix (see `Dataset`
|
||
history: the first failed attempt left no orphan row, confirmed via
|
||
`ROLLBACK`).
|
||
|
||
New read-only API endpoints for these domains (used by the coverage script,
|
||
reusable for future classifier work): `GET /v1/reference/cfop/{code}`,
|
||
`/v1/reference/ncm_utrib/{ncm}`, `/v1/reference/anp/{cprod_anp}`,
|
||
`/v1/reference/fcp/{uf}`.
|
||
|
||
## Domains MISSING_BOOTSTRAP_FILE (still blocked — no file provided)
|
||
|
||
Not `FAILED` (which implied a broken acquisition attempt) — these simply
|
||
have no corresponding file in `sources/raw/bootstrap_v1/` yet. The Portal
|
||
Nacional NF-e (`nfe.fazenda.gov.br`) remains the authority for all of them;
|
||
this session's Bash tool has zero DNS/internet resolution and the built-in
|
||
browser was denied navigation into the portal's document sections on every
|
||
attempt after one initial homepage load (see prior evidence, unchanged this
|
||
session — this mission's instructions explicitly said not to attempt
|
||
internet access again).
|
||
|
||
| DOMAIN | AUTHORITY | STATUS |
|
||
|---|---|---|
|
||
| ICMS Monofásico — Combustíveis | Portal Nacional NF-e | MISSING_BOOTSTRAP_FILE |
|
||
| RTC — cClassTrib (IBS/CBS) | Portal Nacional NF-e | MISSING_BOOTSTRAP_FILE |
|
||
| RTC — Alíquotas CBS | Portal Nacional NF-e | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.10 (alíquotas diferenciadas) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.11 (alíquota por unidade) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.12 (substituição tributária) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.13 (alíquota zero) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.14 (isenção) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.15 (sem incidência) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.16 (suspensão) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
| PIS/COFINS Tabela 4.3.17 (demais alíquotas diferenciadas) | Receita Federal / SPED | MISSING_BOOTSTRAP_FILE |
|
||
|
||
```
|
||
BLOCKER_EXACT=No file for ICMS Monofásico, RTC cClassTrib, RTC Alíquotas CBS, or any of the 8 PIS/COFINS Tabela 4.3.10-17 documents exists in sources/raw/bootstrap_v1/; the Portal Nacional NF-e/SPED remains unreachable from this environment (unchanged from the prior mission's evidence; not re-tested per this mission's explicit "não tente acessar internet" instruction).
|
||
EVIDENCE=find sources/raw -type f lists 11 files; none match these domains by content (only CFOP/FCP/ANP/NCM_UTRIB/RTC_BIOFUEL_INDEX/RTC_CCREDPRES files were provided).
|
||
OPERATOR_ACTION_REQUIRED=Place the corresponding official files into sources/raw/bootstrap_v1/ (or a new bootstrap_v3/) for a future mission to identify, hash, and import the same way this session did for the other 6.
|
||
```
|
||
|
||
## Scope prioritization (business scope, 2026-09-19)
|
||
|
||
GTI clients: restaurants, markets, clothing/retail, workshops, general commerce.
|
||
No fuel stations; no EFD-Contribuições/SPED in product scope. "Every Brazilian
|
||
fiscal table" is NOT an MVP requirement. Nothing below is a technical error and
|
||
no existing support was removed.
|
||
|
||
**CORE_MVP (required):** NCM, TIPI_IPI, CEST, BA_ST_MVA, CFOP, ORIGIN, CST_ICMS,
|
||
CSOSN, CST_PIS, CST_COFINS, FCP, RTC_CCLASSTRIB, RTC_CCREDPRES, RTC_CBS_RATES.
|
||
ACTIVE: all except RTC_CCLASSTRIB and RTC_CBS_RATES (`MISSING_BOOTSTRAP_FILE`).
|
||
|
||
| DOMAIN | SCOPE STATUS |
|
||
|---|---|
|
||
| ICMS_MONOPHASIC | OUT_OF_SCOPE_MVP (reason: fuel/station segment is not the target audience) |
|
||
| ANP | ACTIVE_REFERENCE_ONLY (no further rules for MVP) |
|
||
| RTC_BIOFUEL_INDEX | ACTIVE_REFERENCE_ONLY (no further rules for MVP) |
|
||
| PIS_COFINS_4_3_10..4_3_17 | DEFERRED_BY_BUSINESS_SCOPE (not required for deploy; measure need later from client data) |
|
||
|
||
CST_PIS / CST_COFINS basic code domains stay ACTIVE.
|
||
|
||
### Operator action — the two remaining CORE files
|
||
Place in `sources/raw/bootstrap_v1/` (Portal Nacional NF-e, Reforma Tributária / RTC area):
|
||
1. **RTC_CCLASSTRIB** — official cClassTrib table (IBS/CBS classificação tributária, xlsx) with CST/cClassTrib codes and their descriptions/indicators.
|
||
2. **RTC_CBS_RATES** — official CBS rates table (alíquotas CBS, xlsx) with rate and validity dates.
|
||
Status until then: `MISSING_BOOTSTRAP_FILE` (verified: no matching file locally).
|
||
|
||
## Update 2026-09-19 — CORE MVP closed (RTC_CCLASSTRIB + RTC_CBS_RATES ACTIVE)
|
||
|
||
Both files identified by sheet names/headers, not filename.
|
||
|
||
| DOMAIN | FILE | SHA256 (prefix) | PARSER | RECORDS | EVIDENCE |
|
||
|---|---|---|---|---|---|
|
||
| RTC_CCLASSTRIB | `cClassTrib 2026-06-22.xlsx` | `1448cb63a41b...` | rtc-cclasstrib-xlsx-v1 | 164 cClassTrib + 18 CST-IBS/CBS | Sheets `CST 2026-06-01 Pub` (CST, description, ind_* flags) and `cClass 2026-06-01 Pub` (CST, cClassTrib, name, description, LC/regulamento refs, tipo de alíquota, pRedIBS/pRedCBS, dIniVig/dFimVig, ind_* per document model, ANEXO, Link). All official columns kept (key fields typed, ind_* in JSON). Every cClass CST exists in the CST sheet; cClassTrib unique; dIniVig 2026-01-01; 3 rows have dFimVig |
|
||
| RTC_CBS_RATES | `IT 2026.002 v.1.00 - Aliquotas da CBS.xlsx` | `33f17aa332...` | rtc-cbs-rates-xlsx-v1 | 2 | Title "Aliquota da CBS", header Ano/Alíquota: 2026 = 0.009; 2027 = "aguarda legislação" (kept as text, rate NULL — not invented) |
|
||
|
||
Migration 0010 (3 tables). Reimport idempotent (same dataset ids 49/50, no new rows). Both kinds are in `dataset_versions()` (cache fingerprint changes naturally). New endpoints: `GET /v1/reference/rtc_cclasstrib/{code}`, `/v1/reference/rtc_cbs_rate/{year}`. CORE MVP domains: all 14 ACTIVE. Coverage: `docs/FISCAL_COVERAGE_FULL_V3.json`.
|