chore: establish project baseline
This commit is contained in:
OpenCode
@@ -0,0 +1,85 @@
|
||||
# Mission 00: Firebird Connectivity and Discovery
|
||||
|
||||
## Goal
|
||||
|
||||
Use the Firebird installation associated with the locally installed ERP as
|
||||
read-only evidence to connect only to the authorized copy
|
||||
`D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`, perform complete schema
|
||||
discovery, and document confirmed findings without implementing fiscal writes,
|
||||
image writes, services, or schedulers.
|
||||
|
||||
## Scope and boundaries
|
||||
|
||||
- Files outside `D:\PROJETOS\GTISISTEMAS\Projeto` are read-only.
|
||||
- Only `TESTESOFICIAIS.FDB` may be opened or queried.
|
||||
- Other database files may be listed but must not be connected to, copied, or
|
||||
processed.
|
||||
- No DML, DDL, procedure execution, generator changes, metadata changes,
|
||||
permission changes, or service lifecycle changes are allowed.
|
||||
- Credentials and secrets found in ERP configuration are evidence of location
|
||||
only and must never enter source, reports, logs, or commits.
|
||||
|
||||
## Connection strategy
|
||||
|
||||
Use the installed Firebird server and its compatible Python driver/client as
|
||||
the primary path. First inspect services, processes, executable and DLL
|
||||
versions, tools, and ERP configuration. Select the client only after its
|
||||
version and architecture are evidenced. Use an embedded/local client only as a
|
||||
documented fallback if the server path is unavailable and compatibility is
|
||||
proven. Do not install an arbitrary Firebird version.
|
||||
|
||||
The connection layer will enforce the absolute authorized database path and
|
||||
require database-level or transaction/driver-level read-only enforcement. The
|
||||
first database operation will be a minimal identity query. The sanitized
|
||||
connection description will include driver, client, architecture, host/port
|
||||
form, and target path, but no secrets.
|
||||
|
||||
## Discovery flow
|
||||
|
||||
1. Detect Firebird capabilities and execute only safe catalog SELECT queries.
|
||||
2. Collect engine version, dialect, ODS, charset, page size, database identity,
|
||||
relation counts, fields, keys, indexes, dependencies, generators, triggers,
|
||||
procedures, parameters, and compatible monitoring metadata.
|
||||
3. Identify product candidates using names, fields, keys, indexes, counts,
|
||||
relationships, types, and minimal masked scalar samples.
|
||||
4. Map product, stock, fiscal, company/branch, image/file, audit, and
|
||||
incremental concepts. Mark every mapping `CONFIRMED`, `INFERRED`, or
|
||||
`UNKNOWN` based on direct evidence.
|
||||
5. Inspect trigger and routine source only as metadata text, without invoking
|
||||
routines or collecting source/blob contents that are sensitive or
|
||||
unnecessary.
|
||||
6. Build fingerprint V1 from engine context, product structure, fiscal fields,
|
||||
relationships, image structure, and critical triggers.
|
||||
|
||||
## Safety and sanitization
|
||||
|
||||
The existing read-only SQL boundary remains mandatory. Catalog capability
|
||||
failures are recorded as unavailable or query errors rather than treated as
|
||||
negative findings. Samples are small, scalar, masked, and purpose-limited.
|
||||
Raw BLOBs, credentials, customer data, and unnecessary commercial data are
|
||||
excluded. A path guard prevents accidental access to the ERP's original
|
||||
database even if configuration reveals it.
|
||||
|
||||
## Project changes and verification
|
||||
|
||||
Changes are limited to the authorized project directory: path guard/driver
|
||||
integration, discovery tests and fixtures, fingerprint support, and the three
|
||||
current-state documents plus sanitized mission evidence. The `.FDB` remains
|
||||
ignored by Git. Verification includes focused safety/parser tests, full
|
||||
`python -m pytest tests -q`, `git diff --check`, sensitive-data review, and
|
||||
Git status/identity inspection. A single mission commit is created only when
|
||||
Git identity is already configured; otherwise report
|
||||
`GIT_IDENTITY_NOT_CONFIGURED`.
|
||||
|
||||
## Outcomes
|
||||
|
||||
`MISSION_00_COMPLETE` requires real connection, confirmed engine details,
|
||||
evidence-backed product and fiscal mapping, image/incremental/trigger review,
|
||||
fingerprint V1, green tests, zero application writes, and no external file
|
||||
modification. Any unresolved connectivity or compatibility blocker produces
|
||||
`MISSION_00_BLOCKED` with the evidence and remaining gaps documented.
|
||||
|
||||
## Explicit non-goals
|
||||
|
||||
Do not implement fiscal rules, APIs, image storage, product UPDATEs, Windows
|
||||
services, production schedulers, migrations, triggers, or database changes.
|
||||
@@ -0,0 +1,95 @@
|
||||
# Mission 00D: Structured Firebird Read-Only Boundary
|
||||
|
||||
## Goal
|
||||
|
||||
Remove arbitrary SQL from the public safety boundary, preserve legitimate
|
||||
Firebird metadata expressions internally, then connect read-only only to
|
||||
`D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` and complete evidence-backed
|
||||
discovery when credentials and compatibility permit.
|
||||
|
||||
## Root cause
|
||||
|
||||
The previous `ReadOnlyConnection.execute(sql)` accepted consumer-provided SQL
|
||||
and used a blacklist/allowlist parser as its execution authority. Quoted
|
||||
identifiers could hide callable names from the textual scan, allowing potential
|
||||
procedure/UDF invocation. The same incomplete callable allowlist rejected
|
||||
legitimate Firebird expressions such as `CAST` and aggregate functions. This
|
||||
is a boundary-design failure, not a missing regex case.
|
||||
|
||||
## Architecture
|
||||
|
||||
- Remove `execute(sql)` and equivalent raw execution methods from the public
|
||||
adapter consumed by collectors and mapping code.
|
||||
- Keep driver execution in a private implementation that accepts only internal
|
||||
query identifiers and internally owned static SQL templates.
|
||||
- Expose structured catalog operations for database info, relations, columns,
|
||||
primary/foreign keys, indexes, triggers, procedures, generators,
|
||||
dependencies, and monitoring metadata.
|
||||
- Expose relation sampling only after relation and column identities have been
|
||||
returned by the catalog. Validate those identities structurally, quote them
|
||||
as Firebird identifiers, and cap the integer row limit.
|
||||
- Keep transaction-level Firebird read-only enforcement and fail closed if it
|
||||
cannot be proven. Do not expose commit, rollback, DDL, DML, procedure
|
||||
execution, or generator modification.
|
||||
- `collect_metadata` must require the approved structured reader at runtime,
|
||||
not merely by type annotation.
|
||||
|
||||
## Query ownership
|
||||
|
||||
All metadata SQL belongs to the implementation and is selected by fixed query
|
||||
keys. Internal queries may use `CAST`, `COUNT`, `SUM`, `MIN`, `MAX`,
|
||||
`COALESCE`, joins, grouping, and other required Firebird syntax without going
|
||||
through the old arbitrary-SQL parser. Procedures and UDFs are queried only as
|
||||
metadata and never invoked.
|
||||
|
||||
The old parser may remain as an offline diagnostic/test utility, but it has no
|
||||
authority to decide whether runtime SQL is executable. If retained, its
|
||||
documentation must clearly state that quoted identifiers and Firebird grammar
|
||||
are not security guarantees.
|
||||
|
||||
## Sampling
|
||||
|
||||
Sampling accepts only a relation identity previously returned by catalog
|
||||
metadata, a column set previously returned for that relation, and a validated
|
||||
small integer limit. It rejects expressions, clauses, procedures, UDFs,
|
||||
fragments, arbitrary `WHERE`/`ORDER BY`, unknown relations/columns, malformed
|
||||
quoted identifiers, and alternate database paths. It returns sanitized scalar
|
||||
evidence only; BLOB content, credentials, customer data, and unnecessary
|
||||
commercial values are excluded.
|
||||
|
||||
## Verification gates
|
||||
|
||||
Before any database access, tests must prove that arbitrary SQL cannot reach
|
||||
the driver, quoted identifiers cannot reintroduce callable execution,
|
||||
procedures/UDFs cannot be invoked, legitimate internal Firebird expressions are
|
||||
supported, and arbitrary adapters are rejected. Only after all focused tests
|
||||
and the full suite pass may the installed Firebird 2.5.9 client be used.
|
||||
|
||||
The connection then targets only the authorized `.FDB`, performs a minimal
|
||||
identity query first, and uses a read-only transaction. If credentials,
|
||||
compatibility, or identity verification fail, the result remains
|
||||
`MISSION_00_BLOCKED` and no discovery query continues.
|
||||
|
||||
## Discovery and documentation
|
||||
|
||||
After a successful connection, collect capability-aware catalog metadata,
|
||||
identify the product relation using structural and masked sample evidence,
|
||||
correlate the ERP UI fields to database fields with `CONFIRMED`, `INFERRED`,
|
||||
or `UNKNOWN` classifications, inspect fiscal/image/incremental structures,
|
||||
and analyze trigger/procedure metadata without execution. Generate a real
|
||||
schema fingerprint only from sanitized evidence. Update the current-state
|
||||
documents and final report to describe the actual adapter, parser role,
|
||||
runtime protections, transaction behavior, connection facts, and remaining
|
||||
gaps.
|
||||
|
||||
## Safety invariants
|
||||
|
||||
- `AUTHORIZED_DATABASE` is the sole file opened or queried.
|
||||
- `OTHER_DATABASES_ACCESSED = 0`.
|
||||
- `APPLICATION_DDL_DML_WRITES = 0`.
|
||||
- `ERP_FILES_MODIFIED = 0`.
|
||||
- Files outside `Projeto` are read-only.
|
||||
- No credentials, raw BLOBs, full rows, or original ERP database paths enter
|
||||
project documentation.
|
||||
- No API fiscal, image API, service, scheduler, product UPDATE, or migration
|
||||
is implemented.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Mission 00E: Threat Model, FDB Provenance, and Real Discovery
|
||||
|
||||
## Goal
|
||||
|
||||
Formalize the trusted-process threat model, validate the actual `fdb` module
|
||||
provenance and Firebird client compatibility, then run read-only discovery only
|
||||
against `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB`.
|
||||
|
||||
## Threat model
|
||||
|
||||
`TRUSTED_PROCESS = TRUE`: only versioned application code runs in the process;
|
||||
there are no untrusted plugins, user-provided scripts, external `eval`/`exec`,
|
||||
or externally selected dynamic imports. Deliberate introspection or monkey
|
||||
patching by code that already has arbitrary Python execution is equivalent to
|
||||
application compromise and is outside this component's boundary.
|
||||
|
||||
The boundary protects against implementation mistakes, accidental unauthorized
|
||||
operations, arbitrary SQL exposed to normal callers, parameter-to-SQL injection,
|
||||
routine execution, wrong database selection, unintended writes, and dependency
|
||||
shadowing/import mistakes.
|
||||
|
||||
## API boundary
|
||||
|
||||
The public API exposes only structured catalog operations and catalog-issued
|
||||
bounded sampling. It exposes no `execute(sql)`, `query(sql)`, `raw_execute`,
|
||||
operation-registration mutator, procedure/UDF invocation, arbitrary clauses, or
|
||||
alternate database path. The registry remains module-private, immutable, and
|
||||
centrally constructed. Python internals are not treated as a cryptographic
|
||||
sandbox.
|
||||
|
||||
The old SQL parser is offline-only diagnostic/test code. It has no authority to
|
||||
approve runtime SQL. Internal Firebird query templates may use legitimate
|
||||
`CAST`, aggregates, joins, and grouping because they are selected by fixed
|
||||
operation keys owned by the project.
|
||||
|
||||
## Provenance gate
|
||||
|
||||
Before any database connection, inspect the actual `import fdb` module:
|
||||
|
||||
- `fdb.__file__` and `fdb.__package__`;
|
||||
- module version and `importlib.metadata` distribution/version/location;
|
||||
- interpreter, working directory, `sys.path`, and `PYTHONPATH`;
|
||||
- local `fdb.py`, `fdb` directory, and unexpected `.pth` shadowing candidates.
|
||||
|
||||
The imported module must correspond to the expected installed distribution and
|
||||
version, without ambiguous duplicate installations. The installed Firebird
|
||||
client/server evidence must match the selected driver compatibility. Any
|
||||
ambiguity blocks before import/use of the driver for database access.
|
||||
|
||||
## Connection and discovery
|
||||
|
||||
After the provenance and safety gates pass, connect only to the authorized FDB,
|
||||
using the installed compatible client and a Firebird read-only transaction. Run
|
||||
the minimal identity operation first and compare the engine-reported identity
|
||||
to the authorized target. Then collect catalog metadata through structured
|
||||
operations, use only catalog-issued sample identities, and inspect product,
|
||||
stock, fiscal, image, incremental, trigger, and procedure metadata without
|
||||
executing routines or reading BLOB contents.
|
||||
|
||||
Correlate the ERP UI fields `NCM`, `CFOP`, `CST ICMS`, ICMS rate, `CEST`,
|
||||
`CSOSN`, PIS/COFINS, IPI, origin, IAT, incidence, and IPPT to table, column,
|
||||
type, domain/FK, dependencies, triggers, evidence, and confidence. Classify
|
||||
each as `CONFIRMED`, `INFERRED`, or `UNKNOWN`. Generate a real fingerprint only
|
||||
from sanitized complete evidence; otherwise use `SCHEMA_UNKNOWN`.
|
||||
|
||||
## Evidence and safety
|
||||
|
||||
- Only `D:\PROJETOS\GTISISTEMAS\Projeto\TESTESOFICIAIS.FDB` may be opened.
|
||||
- External ERP files, services, registry, DLLs, and configuration are read-only.
|
||||
- `APPLICATION_DDL_DML_WRITES = 0`.
|
||||
- `OTHER_DATABASES_ACCESSED = 0`.
|
||||
- `ERP_FILES_MODIFIED = 0`.
|
||||
- No credentials, original ERP database path, raw BLOB/image, full rows, or
|
||||
unnecessary customer/commercial data enters source or reports.
|
||||
- No fiscal API, image API, Windows service, scheduler, product UPDATE, or
|
||||
migration is implemented.
|
||||
|
||||
## Verification and outcome
|
||||
|
||||
Tests cover the public API, registry immutability, SQL non-authority,
|
||||
quoted-identifier and routine cases, legitimate internal expressions, path
|
||||
allowlisting, provenance/shadowing, client compatibility, structured reader,
|
||||
sampling, fingerprint, and discovery. The final report includes provenance,
|
||||
client/engine facts, connection result, schema mappings, image/incremental/
|
||||
trigger evidence, tests, Git state, and real gaps. Missing credentials,
|
||||
ambiguous provenance, compatibility failure, identity mismatch, or incomplete
|
||||
discovery results in `MISSION_00_BLOCKED` without a database connection or
|
||||
fallback to another database.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Fiscal Data Foundation V1
|
||||
|
||||
## Goal
|
||||
|
||||
Provide independently versioned, official fiscal datasets for the Hub without determining taxes for products.
|
||||
|
||||
## Dataset lifecycle
|
||||
|
||||
Each domain runs independently: official download, raw SHA-256, versioned parser, structural validation, `PENDING` import, semantic/volume validation, then `ACTIVE` promotion. A parser failure records `FAILED` and `failure_reason`; the previous `ACTIVE` version remains queryable. No secondary source can promote a canonical dataset.
|
||||
|
||||
Every version records `domain`, `source_url`, `downloaded_at`, `raw_sha256`, `parser_version`, `effective_from`, `effective_to`, `record_count`, `status`, `failure_reason`, and `raw_payload_reference` when retention is allowed.
|
||||
|
||||
## Domains
|
||||
|
||||
Existing NCM/Siscomex and CEST/CONFAZ remain intact. New isolated importers cover BA ST/MVA, CFOP, ICMS CST/CSOSN/origin, PIS/COFINS, TIPI/IPI, RTC, FCP, and monophase ICMS/ANP. Each has a normalized operational table only when queried by Hub consumers.
|
||||
|
||||
## Verification
|
||||
|
||||
Parser tests prove fail-closed behavior and independent lifecycle behavior. The final smoke reads only `ACTIVE` versions and produces the dataset matrix. This mission does not classify products or propose tax fields.
|
||||
@@ -0,0 +1,151 @@
|
||||
# GTI Fiscal Hub V1 - Mission 02 Design
|
||||
|
||||
## Scope
|
||||
|
||||
Deliver one end-to-end vertical slice: a FastAPI fiscal hub backed by
|
||||
PostgreSQL, a versioned official NCM dataset, and a Windows-ready Python agent
|
||||
that reads the authorized Firebird database and submits 20 real products to the
|
||||
hub. The agent never writes to Firebird in this mission.
|
||||
|
||||
The Hub and agent are separate applications. The Hub owns fiscal knowledge;
|
||||
the agent owns ERP communication. The Hub never accesses a customer Firebird
|
||||
database and the agent never accesses central PostgreSQL.
|
||||
|
||||
## Components
|
||||
|
||||
`central/app` is a modular FastAPI monolith with these conceptual modules:
|
||||
|
||||
- API and bearer-token authentication.
|
||||
- Dataset registry and NCM importer.
|
||||
- Product classification: normalized description, GTIN, NCM, and CEST.
|
||||
- Fiscal determination: operation and company context with NCM/CEST, CFOP,
|
||||
CST/CSOSN, PIS/COFINS, IPI, and future IBS/CBS/cClassTrib inputs.
|
||||
- Deterministic classification cache.
|
||||
- Audit and provenance.
|
||||
- Provider interfaces, including `RtcCalculatorProvider`.
|
||||
|
||||
Product classification and fiscal determination remain separate modules inside
|
||||
the same process. This makes an existing valid NCM distinct from a fully proven
|
||||
tax treatment.
|
||||
|
||||
`agent/src` contains a Firebird reader, schema fingerprint gate, product
|
||||
fingerprinting, SQLite state/outbox, HTTP client, retry/backoff handling, and
|
||||
comparison/report generation.
|
||||
|
||||
`docker-compose.yml` starts `postgres` and `fiscal-hub` only. The Firebird
|
||||
database remains installed locally on Windows and outside Docker.
|
||||
|
||||
## Hub Persistence
|
||||
|
||||
PostgreSQL stores `datasets`, `ncm_entries`, `tenants`, `agents`,
|
||||
`classification_requests`, `classification_results`, and
|
||||
`classification_cache`. Dataset metadata includes source URL, provider,
|
||||
download time, SHA-256, source validity/version data when present, record
|
||||
count, lifecycle status, and import status.
|
||||
|
||||
Agent tokens are supplied by environment. PostgreSQL stores only token hashes,
|
||||
each associated with one tenant and agent. Tenant identity is resolved from the
|
||||
token and must agree with request metadata.
|
||||
|
||||
The agent uses a separate SQLite database. It stores source and result
|
||||
fingerprints, fiscal dataset versions, timestamps, reason codes, retry data,
|
||||
and an outbox with this lifecycle:
|
||||
|
||||
`DISCOVERED -> PENDING_API -> SENT -> RESULT_RECEIVED -> APPLIED`
|
||||
|
||||
`APPLIED` is reserved for a future write-capable mission and is not reached in
|
||||
Mission 02.
|
||||
|
||||
## NCM Dataset Lifecycle
|
||||
|
||||
The only canonical source in this mission is the Receita Federal / Portal
|
||||
Unico Siscomex JSON endpoint:
|
||||
|
||||
`https://portalunico.siscomex.gov.br/classif/api/publico/nomenclatura/download/json`
|
||||
|
||||
The importer downloads raw content, calculates SHA-256, parses JSON, and
|
||||
inspects its real schema before mapping fields. It must not assume field names
|
||||
before this inspection.
|
||||
|
||||
The lifecycle is:
|
||||
|
||||
`DOWNLOAD -> HASH -> PENDING -> IMPORT -> VALIDATION -> READY -> ATOMIC PROMOTION -> ACTIVE`
|
||||
|
||||
The previous active version becomes `SUPERSEDED` only inside the same short
|
||||
promotion transaction that activates the ready version. NCM entries are scoped
|
||||
to their dataset version, so clients observe either the complete prior dataset
|
||||
or the complete new dataset, never a mixture. Failed imports remain
|
||||
non-active, and prior versions remain queryable for rollback and audit.
|
||||
|
||||
Validation requires a successful HTTP response, parseable JSON, recognized
|
||||
schema, plausible nonzero record volume, persisted hash and provenance, and a
|
||||
fully successful import. A scheduled updater runs this same flow and promotes
|
||||
only a validated version.
|
||||
|
||||
## Classification and Decisions
|
||||
|
||||
`POST /v1/validate/ncm` normalizes a NCM code and validates it against the
|
||||
active official dataset. `/v1/classify` and `/v1/classify/batch` accept a
|
||||
minimal product plus extensible company and operation context.
|
||||
|
||||
If an existing NCM is valid, the classification result is `NCM_VALIDATED`.
|
||||
This is not permission to populate or change CST, CFOP, CSOSN, rates, CEST, or
|
||||
any other fiscal field. Fields without official deterministic support return
|
||||
`null` with explicit reason codes.
|
||||
|
||||
Products without a valid NCM use deterministic normalized-text candidate
|
||||
ranking. Outcomes are `HIGH_CONFIDENCE_CANDIDATE`, `AMBIGUOUS`, or `NOT_FOUND`;
|
||||
description similarity never creates a definitive NCM classification.
|
||||
|
||||
Each response carries `engine_version`, `dataset_versions` (including `ncm`),
|
||||
and `decision_version`. These versions participate in central cache keys and
|
||||
agent incremental decisions. A product with unchanged source fields is
|
||||
reprocessed when relevant dataset or engine versions change.
|
||||
|
||||
`can_auto_apply` is always false in Mission 02 because no Firebird writes are
|
||||
implemented. `NCM_VALIDATED` is a classification status only.
|
||||
|
||||
## Cache, Idempotency, and Audit
|
||||
|
||||
The Hub derives a request fingerprint from tenant, normalized product identity,
|
||||
relevant context, engine version, and active dataset versions. A matching cache
|
||||
entry returns an equivalent decision and does not create uncontrolled duplicate
|
||||
audit records. A changed tenant, context, engine, or dataset version cannot
|
||||
reuse a result.
|
||||
|
||||
The agent derives its product fingerprint from description, GTIN, NCM and
|
||||
relevant current fiscal fields. It avoids an API call only if this fingerprint,
|
||||
the prior decision versions, and result state all remain current. Interrupted
|
||||
or failed sends stay in the outbox for retry. If the API is offline, the agent
|
||||
does not affect the ERP and continues processing other queued items.
|
||||
|
||||
## Provider Status
|
||||
|
||||
NCM is implemented. CEST, CFOP, CST/CSOSN, cClassTrib, IBS/CBS, and the RTC
|
||||
calculator are investigated and recorded with one of `IMPLEMENTED`, `PARTIAL`,
|
||||
`DISCOVERED`, or `NOT_AVAILABLE`. The RTC adapter implements the provider
|
||||
interface but returns `NOT_CONFIGURED` unless a concrete official integration
|
||||
is verified to be simple and safe within this scope. It never fabricates tax
|
||||
results.
|
||||
|
||||
## End-to-End Proof
|
||||
|
||||
The agent reads the 20 sample products from `C000025` using only SELECTs after
|
||||
its schema fingerprint equals the recorded `REFERENCE_SCHEMA_V1` fingerprint.
|
||||
It sends them to the authenticated batch endpoint, persists results locally,
|
||||
and writes `docs/MISSION_02_PRODUCT_RESULTS.json` plus
|
||||
`docs/MISSION_02_REPORT.md`.
|
||||
|
||||
The report includes coverage and timing metrics, API request and cache counts,
|
||||
paid API counters fixed at zero, Firebird safety counters, dataset provenance,
|
||||
integration statuses, and real gaps. No BLOB, image, Firebird schema change,
|
||||
or Firebird DML/DDL is allowed.
|
||||
|
||||
## Verification
|
||||
|
||||
Focused tests cover NCM import and lifecycle, normalization/ranking,
|
||||
authentication and tenant isolation, endpoints, cache/idempotency, agent
|
||||
payloads, read-only Firebird access, fingerprint gating, incremental outbox,
|
||||
and offline API behavior. The complete project test suite runs afterward.
|
||||
Docker health checks cover PostgreSQL and the Hub; Hub health reports failed
|
||||
when PostgreSQL is unavailable.
|
||||
Reference in New Issue
Block a user