Files

T

gtiandClaude Opus 5.5 b1c496fc1b docs: record migration head 0021 (SSO) in production, TLS via NPM, Caddy removed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-09-23 12:30:49 -03:00

21 KiB

Raw Blame History

GTI Fiscal — Pilot Deployment Guide

Living document for operating a real pilot: one Hub, one or more client Agent installations. See docs/OPERATIONS.md for the day-to-day command reference (creating jobs, checking status); this document is about getting a new environment stood up and kept alive.

RC1 (1.0.0-rc1) — supported path

Hub central (Docker: Hub + PostgreSQL, deploy/hub)
        ^   HTTPS (TLS terminated by a reverse proxy)
        |
Agent Windows (service "GTIFiscalAgent", deploy/agent-windows)
        |   local read-only connection
SGV Firebird (the client's C000025 / C000004)

The Hub never accesses Firebird, never stores Firebird credentials and never holds the client's database path. Only the Agent, on the client's Windows host, opens Firebird. RC1 pilots run FISCAL_MODE=DRY_RUN + FIREBIRD_WRITES=0: the Agent is technically unable to write (no write connection is ever opened, the executor is never called).

Hub (GTI infrastructure)

cd deploy/hub && cp .env.example .env — set POSTGRES_PASSWORD (openssl rand -hex 24), BASE_URL (public HTTPS URL). Leave HUB_BIND=127.0.0.1 and publish it through the reverse proxy.
docker compose -f docker-compose.prod.yml --env-file .env up -d --build — PostgreSQL (internal network only, persistent volume, healthcheck), Hub (migrations at start, healthcheck, restart: unless-stopped, json-file logs 10 MB x 5).
Seed the validated datasets once: on the development machine sh export_datasets_seed.sh > gti_datasets_seed.dump; on the Hub host sh seed_datasets.sh gti_datasets_seed.dump (refuses if datasets is not empty; reference tables only, never tenants/agents/jobs).
Register the Agent: docker compose -f docker-compose.prod.yml exec fiscal-hub python central/scripts/register_agent.py --company "<Client>" --name "<installation>" — the token is printed once; store it only in the Agent's local .env.
Health: GET /health (Postgres included), GET /version (release/engine/decision versions). APP_ENV=production refuses to start when AGENT_DEV_TOKEN is set.
Backup/restore: sh backup.sh (pg_dump -Fc into BACKUP_DIR, keeps 14) and sh restore.sh <dump> (asks for the word RESTORE, stops the Hub, restores, restarts; migrations rerun at start).

Agent (client host, PowerShell as Administrator, inside the extracted `GTIFiscalAgent-1.0.0-rc1` folder)

Build the package on the release machine: python scripts/build_agent_package.py -> dist/GTIFiscalAgent-1.0.0-rc1.zip (+ .sha256; secret-scanned, no .env).

step	command
install	`.\gti-agent.ps1 install` (venv + dependencies + service registration; does not start)
configure	`.\gti-agent.ps1 configure` (prompts; secrets never echoed) or `configure -FromTemplate .\pilot-construcao.env.example`
preflight	`.\gti-agent.ps1 preflight` -> `PREFLIGHT=PASS
start / stop / restart	`.\gti-agent.ps1 start` (refuses when preflight fails)
status	`.\gti-agent.ps1 status`
uninstall	`.\gti-agent.ps1 uninstall [-Purge]`

Agent configuration (per installation, `.env` in the install folder)

GTI_HUB_URL, AGENT_ID, AGENT_TOKEN; COMPANY_CODE / COMPANY_CNPJ (company selection), COMPANY_CRT_SOURCE = AUTO | EMISSOR | C000004; COMPANY_UF only as an explicit override (it never replaces the resolver: it must agree with what the database proves); FIREBIRD_DATABASE_PATH, FB_USER (read-only account), FB_PASSWORD; FISCAL_MODE=DRY_RUN, FIREBIRD_WRITES=0; POLL_INTERVAL_SECONDS, AGENT_VERSION. AUTO resolves the CRT only when EMISSOR and C000004 agree; a conflict makes the preflight FAIL until EMISSOR or C000004 is chosen explicitly. The first row of C000004 is never used to pick the company.

Preflight (`agent/scripts/preflight.py`, also enforced at service start)

HUB_REACHABLE, AUTH, HUB_VERSION_COMPATIBLE (major.minor), FIREBIRD_REACHABLE, FIREBIRD_VERSION_ODS, C000004, C000025, CAPABILITIES (required: NCM, IPI, CEST), COMPANY_CONTEXT (UF, CRT), SCHEMA_TYPE_GUARDS, SQLITE, LOG_DIRECTORY, FISCAL_MODE, FIREBIRD_WRITES (DRY_RUN + FIREBIRD_WRITES=1 is a misconfiguration -> FAIL).

Pilot profiles

deploy/agent-windows/pilot-construcao.env.example (first pilot: company 000001, UF BA, CRT 1, COMPANY_CRT_SOURCE=AUTO, DRY_RUN). Planned, not installed: profiles/mmmodas|rincao|uniao.env.example (COMPANY_CRT_SOURCE=EMISSOR required: EMISSOR and C000004 disagree on the CRT). Never reuse the path of a local database copy as the client's real path.

Zero-write proof

python scripts/prove_zero_write.py snapshot <FDB> before.json ... run the pilot ... snapshot <FDB> after.json ... compare before.json after.json (database DML counters, SHA-256 of every row of C000025/C000004, row counts).

The sections below are the previous manual instructions (still accurate for development); the RC1 scripts above are the supported way to deploy.

Architecture

                         ┌────────────────────────────┐
                         │  Hub server (GTI/Proxmox)  │
                         │  Docker Compose:            │
                         │   - postgres (data volume)  │
                         │   - fiscal-hub (FastAPI)     │
                         └─────────────┬────────────────┘
                                       │ HTTPS (reverse proxy terminates TLS)
              ┌────────────────────────┼────────────────────────┐
              │                        │                        │
    ┌─────────▼─────────┐    ┌─────────▼─────────┐    ┌─────────▼─────────┐
    │ Client A — Agent   │    │ Client B — Agent   │    │ Client C — Agent   │
    │ Windows Service    │    │ Windows Service    │    │ Windows Service    │
    │ + local Firebird    │    │ + local Firebird    │    │ + local Firebird    │
    │ + local SQLite audit│    │ + local SQLite audit│    │ + local SQLite audit│
    └────────────────────┘    └────────────────────┘    └────────────────────┘

The Hub never has Firebird credentials and never opens a Firebird connection. Each Agent has its own token and only ever reads/writes the one Firebird database it was configured for.

Hub server requirements

Docker + Docker Compose v2.
Enough disk for the Postgres volume (starts small; grows with classification_cache/jobs history — no large binary data).
One inbound port for the Hub API (default 8000, or whatever your reverse proxy forwards to it). Postgres itself does not need a host port — docker-compose.yml intentionally does not publish 5432; only containers on the compose network reach it.

Deploying the Hub

git clone <repo> && cd Projeto
cp .env.example .env
# Edit .env: set a real POSTGRES_PASSWORD, leave AGENT_DEV_TOKEN empty,
# set APP_ENV=production, BASE_URL to the public HTTPS URL.
docker compose up -d postgres fiscal-hub
curl http://localhost:8000/health   # {"status": "ok"}

fiscal-hub's container command runs alembic upgrade head before starting uvicorn, so schema migrations apply automatically on every deploy/restart. Both services have restart: unless-stopped and Docker healthchecks (pg_isready for Postgres, an HTTP GET to /health for the Hub).

Never put AGENT_DEV_TOKEN in a production .env — it auto-bootstraps an agent from a fixed shared token on every request, fine for local dev, wrong for a pilot with real client secrets. Leave it empty; register real Agents with central/scripts/register_agent.py (see below).

Persistence

Postgres data lives in the named volume postgres_data. Back it up with docker exec <postgres-container> pg_dump -U gti gti_fiscal > backup.sql (or pg_dumpall for the whole cluster) on whatever schedule your operations policy requires.
Always run alembic upgrade head (the container does this automatically) rather than hand-editing the schema. An empty database is initialized by that same command — never with alembic stamp or create_all. 0001 is an explicit, frozen baseline (it used to call Base.metadata.create_all(), which made 0002 collide on datasets.raw_sha256 on any fresh database); central/tests/test_migration_baseline.py keeps that from coming back.
0019 repairs agents.created_at/updated_at and jobs.created_at, whose defaults 0006 had frozen at the instant it ran (rows created before 0019 keep that frozen value — their real creation time is not recoverable).
0020 enforces NOT NULL on 20 timestamp columns that always had DEFAULT now(). It aborts, changing nothing, if any NULL exists — clean those rows explicitly first. 0021 adds the SSO federated identity (see "Human access"). Current head: 0021_sso_federated_identity (also recorded in release/RELEASE_MANIFEST.json).

HTTPS / reverse proxy

The Hub itself speaks plain HTTP. For a pilot reachable outside the Hub's own host, put it behind a reverse proxy (nginx, Caddy, Traefik — whatever the GTI/Proxmox environment already standardizes on) that terminates TLS and forwards to fiscal-hub:8000. Nothing in the app assumes a particular proxy; BASE_URL in .env is informational only (not currently used to generate absolute links, but keep it accurate for anyone reading the config).

GTI production (2026-09-23): TLS terminates in the GTI Nginx Proxy Manager (10.0.10.100), which forwards http://<hub-lan-ip>:8000. On the Hub host:

.env: HUB_BIND=<hub LAN IP> (not 0.0.0.0), FORWARDED_ALLOW_IPS=<NPM IP> (real client IP and https scheme from X-Forwarded-*), BASE_URL=https://…, and WEB_COOKIE_SECURE=1 once HTTPS works. Redirects are relative, so the internal address never leaks.
Docker-published ports bypass UFW: install restrict_hub_port.sh + gti-fiscal-hub-port.service (instructions in the unit file) so only the NPM reaches the port, re-applied on every Docker start.

Human access: SSO from GTI Gerenciador

Collaborators never type a second password. GTI Gerenciador (the panel) is the identity authority; the Hub keeps its own sessions.

"Fiscal" in the panel sidebar -> panel /sso/fiscal -> Hub GET /login, which sets a one-time state cookie (gti_sso_state, HttpOnly, Secure, SameSite=None, 5 min) and redirects to SSO_START_URL?state=...&next=....
The panel requires its own login (normal login page if needed), checks the employee is active and auto-POSTs a JWT to /auth/sso/callback: Ed25519 (EdDSA only), iss=gti-gerenciador, aud=gti-fiscal, sub=funcionario:<ID>, 60 s, unique jti, nonce=state. Never in a URL.
The Hub verifies signature/iss/aud/exp/nbf/lifetime, the nonce against the state cookie (login-CSRF protection), records the jti (sso_used_assertions, replay = rejected), provisions or refreshes the collaborator in admin_users (auth_source=GTI_GERENCIADOR, no password, enforced by ck_admin_users_auth_source) and opens a session of FEDERATED_SESSION_SECONDS (default 1 h; afterwards the same flow signs the user in again silently, which also picks up a deactivation in the panel).

Keys: a dedicated Ed25519 pair, used for nothing else (never reuse SESSION_SECRET, admin, Agent or DB secrets). The private key (raw 32-byte seed, base64url) goes only to the panel's .env (SSO_FISCAL_CHAVE_PRIVADA); the public key only to this Hub's .env (SSO_GERENCIADOR_CHAVE_PUBLICA). Rotating = new pair in both, restart both.

Logout ("Sair do Fiscal") ends only the Hub session; coming back through the panel signs in again without a password. There is no distributed single logout.

Audit (audit_events): SSO_LOGIN_SUCCESS, SSO_LOGIN_REJECTED (with a stable reason: EXPIRED, INVALID_SIGNATURE, WRONG_AUDIENCE, WRONG_ISSUER, REPLAY, STATE_MISMATCH, DISABLED_IN_FISCAL, ...), LOGOUT, LOGIN_LOCAL_SUCCESS/FAILURE — with the real client IP, never the token.

Break-glass: the local admin (create_admin.py) still exists for emergencies at /login/local (not linked from the UI; every use is audited). Disable a collaborator only in the Hub: admin_users.is_active = false.

Registering an Agent (per client installation)

Run once per Agent installation, from anywhere with access to Postgres — easiest is inside the running Hub container:

docker exec <fiscal-hub-container> python central/scripts/register_agent.py \
  --company "Client Company Name" --name "loja-01"

This prints AGENT_ID, a freshly generated AGENT_TOKEN (shown once — copy it immediately), and COMPANY_ID. Only the token's SHA-256 hash is ever stored in Postgres (same token_hash mechanism /v1/classify already used — no new auth system). Each Agent gets its own token; one Agent's token never authenticates another Agent's jobs (agent_id is checked on every Job operation).

Configuring an Agent installation

On the client machine, copy agent/.env.example to agent/.env and fill in the values from registration plus that machine's Firebird details:

GTI_HUB_URL=https://hub.example.com
AGENT_ID=<from registration>
AGENT_TOKEN=<from registration>
COMPANY_ID=<from registration>
COMPANY_UF=BA
FIREBIRD_DATABASE_PATH=C:\ERP\dados\PRODUCAO.FDB
FB_USER=...
FB_PASSWORD=...
FIREBIRD_WRITES=0
POLL_INTERVAL_SECONDS=10
AGENT_VERSION=0.1.0

FIREBIRD_WRITES must default to 0. See "Write safety" below before ever setting it to 1.

Installing the Agent as a Windows Service

Requires pywin32 (pip install pywin32, or pip install -e ".[windows-service]" from agent/).

# From an elevated (Administrator) terminal, in agent/service/:
python gti_agent_windows_service.py --startup=auto install
python gti_agent_windows_service.py start

# One-time, so Windows restarts it automatically after a crash:
sc.exe failure "GTIFiscalAgent" reset= 86400 actions= restart/60000/restart/60000/restart/60000

gti_agent_windows_service.py is a thin wrapper: it does not reimplement polling, heartbeat, or the pipeline — it calls run_agent_service.run() directly, stopping it via the Service Control Manager's own stop signal instead of SIGINT/SIGTERM (which don't apply to services running with no console). See the module's docstring for the exact commands.

Install / configure / start / status / stop / restart / uninstall

# Install (registers with SCM, Startup=Automatic)
python gti_agent_windows_service.py --startup=auto install

# Configure: edit agent/.env, no service restart needed until you `restart`
notepad agent\.env

# Start
python gti_agent_windows_service.py start

# Status
sc.exe query GTIFiscalAgent

# Stop
python gti_agent_windows_service.py stop

# Restart (picks up any agent/.env changes)
python gti_agent_windows_service.py restart

# Uninstall
python gti_agent_windows_service.py stop
python gti_agent_windows_service.py remove

Updating the Agent's version

Stop the service: python gti_agent_windows_service.py stop
Replace the agent/ code with the new version (git pull, or copy files).
Bump AGENT_VERSION in agent/.env if you track it per-release.
Start the service again: python gti_agent_windows_service.py start.
Confirm the new version shows up via GET /v1/agents/me on the Hub.

There is no auto-update mechanism yet (deliberately out of scope for this pilot) — updates are a manual stop/replace/start.

Running without installing as a service (dev/testing)

cd agent
python scripts/run_agent_service.py                 # runs forever, Ctrl+C to stop
python scripts/run_agent_service.py --max-iterations 5   # bounded, for scripting/tests

Creating and checking work

See docs/OPERATIONS.md for the full curl examples (POST /v1/jobs with kind: REPROCESS_ALL or REPROCESS_PRODUCTS, GET /v1/jobs/{id} for status). Every Job is scoped to the Agent whose token created it — no cross-Agent visibility.

Checking Agent status

curl $GTI_HUB_URL/v1/agents/me -H "Authorization: Bearer $AGENT_TOKEN"

Returns agent_id, company_id, enabled, last_seen_at, last_status, schema_fingerprint, agent_version, and a derived status of ONLINE or OFFLINE (offline if no heartbeat arrived within AGENT_ONLINE_THRESHOLD_SECONDS, default 120s — configurable on the Hub). No dashboard yet; this is the query an operator or a future dashboard would poll.

Write safety: enabling FIREBIRD_WRITES for a real client

Do not set FIREBIRD_WRITES=1 against a real client database casually. The authorization model has two independent layers, both required:

Path: AUTHORIZED_WRITE_DATABASE_PATH in that installation's agent/.env must be set to the exact absolute path of FIREBIRD_DATABASE_PATH for that same installation. If it's unset, the guard falls back to the developer-only disposable test database path (TESTESOFICIAIS.FDB), so a real client path is refused automatically — authorization is never inferred from the filename or the .FDB extension.
Schema fingerprint: EXPECTED_SCHEMA_FINGERPRINT must equal the live schema fingerprint that installation reports in its heartbeat (schema_fingerprint, visible via GET /v1/agents/me). If the live schema ever drifts from what was verified, writes are refused until an operator re-verifies and updates this value.

Both checks happen before any write connection is opened (scripts/firebird_write_executor.py::assert_authorized_write_path / assert_schema_fingerprint_authorized), and before any product is even read from Firebird for that job.

Even with both layers satisfied, auto-apply only ever touches C000025.IPI, and only when the TIPI match is rate_kind=NUMERIC and unambiguous. No real client database has been written to during this or any prior mission — all writes so far were against TESTESOFICIAIS.FDB.

Logs and diagnostics

Agent: rotating local log file (AGENT_LOG_PATH in agent/.env, default agent/logs/agent.log, 5MB × 5 backups). Contains startup, heartbeat outcomes, job received/finished, products processed, write counts, failures, and Hub reconnects — never secrets (FB_PASSWORD and AGENT_TOKEN are never logged). Detailed per-product audit is in the Agent's SQLite file (AGENT_STATE_DB, table ipi_write_log).
Hub: structured logging output (visible via docker compose logs fiscal-hub) for startup, health-check failures, heartbeats, and the full Job lifecycle (created/claimed/completed/failed, with sanitized errors). Coarse per-Job audit persists in Postgres (jobs.result_summary).

Troubleshooting

Symptom	Check
Agent shows `OFFLINE`	Is the service running (`sc.exe query GTIFiscalAgent`)? Check `agent/logs/agent.log` for heartbeat failures — usually `GTI_HUB_URL` unreachable or wrong `AGENT_TOKEN`.
Job stuck `PENDING`	Confirm the Agent that owns it is running and polling (its `agent_id` must match the Job's `agent_id` — an Agent never claims another Agent's Job).
Job `FAILED`	`GET /v1/jobs/{id}` for the sanitized `error`. Common causes: Firebird unreachable, `FIREBIRD_WRITES=1` misconfigured (see "Write safety"), Hub unreachable at report time (the Agent keeps the local `ipi_write_log` regardless).
`writes_attempted` always 0 with `FIREBIRD_WRITES=1`	Check both `AUTHORIZED_WRITE_DATABASE_PATH` and `EXPECTED_SCHEMA_FINGERPRINT` are set and correct for that installation.
Hub `/health` returns 503	Postgres is down or unreachable; check `docker compose ps` and `docker compose logs postgres`.

Rollback / uninstalling an Agent

Stop and remove the Windows service (see above).
Delete the agent/ installation directory on that machine (its SQLite audit and logs are local-only; nothing to clean up centrally).
On the Hub, disable the Agent rather than deleting history: set enabled = false for that agents row (no HTTP endpoint for this yet — run it via psql/a short script against DATABASE_URL, the same access register_agent.py uses). Its past Jobs and classification_cache entries remain for audit.

Rolling back the Hub

docker compose down does not remove the postgres_data volume — data survives. To roll back to a previous Hub image/version, redeploy that version's code and run alembic upgrade head again (migrations are additive, so this is safe going forward; there is no tested downgrade path for 0006, so don't alembic downgrade in a pilot without restoring from a backup first).

21 KiB Raw Blame History Unescape Escape