GTI_GERENCIADOR_FISCAL/docs/PILOT_DEPLOYMENT.md

# GTI Fiscal — Pilot Deployment Guide

Living document for operating a real pilot: one Hub, one or more client
Agent installations. See `docs/OPERATIONS.md` for the day-to-day command
reference (creating jobs, checking status); this document is about getting a
new environment stood up and kept alive.

## RC1 (1.0.0-rc1) — supported path

```
Hub central (Docker: Hub + PostgreSQL, deploy/hub)
        ^   HTTPS (TLS terminated by a reverse proxy)
        |
Agent Windows (service "GTIFiscalAgent", deploy/agent-windows)
        |   local read-only connection
SGV Firebird (the client's C000025 / C000004)
```

The Hub **never** accesses Firebird, never stores Firebird credentials and never holds the client's database path. Only the Agent, on the client's Windows host, opens
Firebird. RC1 pilots run `FISCAL_MODE=DRY_RUN` + `FIREBIRD_WRITES=0`: the Agent is technically unable to write (no write connection is ever opened, the executor is never called).

### Hub (GTI infrastructure)
1. `cd deploy/hub && cp .env.example .env` — set `POSTGRES_PASSWORD` (`openssl rand -hex 24`), `BASE_URL` (public HTTPS URL). Leave `HUB_BIND=127.0.0.1` and publish it through the reverse proxy.
2. `docker compose -f docker-compose.prod.yml --env-file .env up -d --build` — PostgreSQL (internal network only, persistent volume, healthcheck), Hub (migrations at start, healthcheck, `restart: unless-stopped`, json-file logs 10 MB x 5).
3. Seed the validated datasets once: on the development machine `sh export_datasets_seed.sh > gti_datasets_seed.dump`; on the Hub host `sh seed_datasets.sh gti_datasets_seed.dump` (refuses if `datasets` is not empty; reference tables only, never tenants/agents/jobs).
4. Register the Agent: `docker compose -f docker-compose.prod.yml exec fiscal-hub python central/scripts/register_agent.py --company "<Client>" --name "<installation>"` — the token is printed once; store it only in the Agent's local `.env`.
5. Health: `GET /health` (Postgres included), `GET /version` (release/engine/decision versions). `APP_ENV=production` refuses to start when `AGENT_DEV_TOKEN` is set.
6. Backup/restore: `sh backup.sh` (pg_dump -Fc into `BACKUP_DIR`, keeps 14) and `sh restore.sh <dump>` (asks for the word RESTORE, stops the Hub, restores, restarts; migrations rerun at start).

### Agent (client host, PowerShell as Administrator, inside the extracted `GTIFiscalAgent-1.0.0-rc1` folder)
Build the package on the release machine: `python scripts/build_agent_package.py` -> `dist/GTIFiscalAgent-1.0.0-rc1.zip` (+ `.sha256`; secret-scanned, no `.env`).

| step | command |
|---|---|
| install | `.\gti-agent.ps1 install` (venv + dependencies + service registration; does not start) |
| configure | `.\gti-agent.ps1 configure` (prompts; secrets never echoed) or `configure -FromTemplate .\pilot-construcao.env.example` |
| preflight | `.\gti-agent.ps1 preflight` -> `PREFLIGHT=PASS|FAIL` + blockers |
| start / stop / restart | `.\gti-agent.ps1 start` (refuses when preflight fails) |
| status | `.\gti-agent.ps1 status` |
| uninstall | `.\gti-agent.ps1 uninstall [-Purge]` |

### Agent configuration (per installation, `.env` in the install folder)
`GTI_HUB_URL`, `AGENT_ID`, `AGENT_TOKEN`; `COMPANY_CODE` / `COMPANY_CNPJ` (company selection), `COMPANY_CRT_SOURCE` = `AUTO` | `EMISSOR` | `C000004`; `COMPANY_UF` only as an explicit
override (it never replaces the resolver: it must agree with what the database proves); `FIREBIRD_DATABASE_PATH`, `FB_USER` (read-only account), `FB_PASSWORD`; `FISCAL_MODE=DRY_RUN`, `FIREBIRD_WRITES=0`;
`POLL_INTERVAL_SECONDS`, `AGENT_VERSION`. `AUTO` resolves the CRT only when EMISSOR and C000004 agree; a conflict makes the preflight FAIL until `EMISSOR` or `C000004` is chosen explicitly.
The first row of C000004 is never used to pick the company.

### Preflight (`agent/scripts/preflight.py`, also enforced at service start)
HUB_REACHABLE, AUTH, HUB_VERSION_COMPATIBLE (major.minor), FIREBIRD_REACHABLE, FIREBIRD_VERSION_ODS, C000004, C000025, CAPABILITIES (required: NCM, IPI, CEST), COMPANY_CONTEXT (UF, CRT),
SCHEMA_TYPE_GUARDS, SQLITE, LOG_DIRECTORY, FISCAL_MODE, FIREBIRD_WRITES (`DRY_RUN` + `FIREBIRD_WRITES=1` is a misconfiguration -> FAIL).

### Pilot profiles
`deploy/agent-windows/pilot-construcao.env.example` (first pilot: company 000001, UF BA, CRT 1, `COMPANY_CRT_SOURCE=AUTO`, DRY_RUN). Planned, **not installed**: `profiles/mmmodas|rincao|uniao.env.example`
(`COMPANY_CRT_SOURCE=EMISSOR` required: EMISSOR and C000004 disagree on the CRT). Never reuse the path of a local database copy as the client's real path.

### Zero-write proof
`python scripts/prove_zero_write.py snapshot <FDB> before.json` ... run the pilot ... `snapshot <FDB> after.json` ... `compare before.json after.json` (database DML counters, SHA-256 of every row of C000025/C000004, row counts).

---

The sections below are the previous manual instructions (still accurate for development); the RC1 scripts above are the supported way to deploy.

## Architecture

```
                         ┌────────────────────────────┐
                         │  Hub server (GTI/Proxmox)  │
                         │  Docker Compose:            │
                         │   - postgres (data volume)  │
                         │   - fiscal-hub (FastAPI)     │
                         └─────────────┬────────────────┘
                                       │ HTTPS (reverse proxy terminates TLS)
              ┌────────────────────────┼────────────────────────┐
              │                        │                        │
    ┌─────────▼─────────┐    ┌─────────▼─────────┐    ┌─────────▼─────────┐
    │ Client A — Agent   │    │ Client B — Agent   │    │ Client C — Agent   │
    │ Windows Service    │    │ Windows Service    │    │ Windows Service    │
    │ + local Firebird    │    │ + local Firebird    │    │ + local Firebird    │
    │ + local SQLite audit│    │ + local SQLite audit│    │ + local SQLite audit│
    └────────────────────┘    └────────────────────┘    └────────────────────┘
```

The Hub never has Firebird credentials and never opens a Firebird
connection. Each Agent has its own token and only ever reads/writes the one
Firebird database it was configured for.

## Hub server requirements

- Docker + Docker Compose v2.
- Enough disk for the Postgres volume (starts small; grows with
  `classification_cache`/`jobs` history — no large binary data).
- One inbound port for the Hub API (default `8000`, or whatever your reverse
  proxy forwards to it). Postgres itself does **not** need a host port —
  `docker-compose.yml` intentionally does not publish `5432`; only containers
  on the compose network reach it.

## Deploying the Hub

```bash
git clone <repo> && cd Projeto
cp .env.example .env
# Edit .env: set a real POSTGRES_PASSWORD, leave AGENT_DEV_TOKEN empty,
# set APP_ENV=production, BASE_URL to the public HTTPS URL.
docker compose up -d postgres fiscal-hub
curl http://localhost:8000/health   # {"status": "ok"}
```

`fiscal-hub`'s container command runs `alembic upgrade head` before starting
`uvicorn`, so schema migrations apply automatically on every deploy/restart.
Both services have `restart: unless-stopped` and Docker healthchecks
(`pg_isready` for Postgres, an HTTP GET to `/health` for the Hub).

**Never** put `AGENT_DEV_TOKEN` in a production `.env` — it auto-bootstraps
an agent from a fixed shared token on every request, fine for local dev,
wrong for a pilot with real client secrets. Leave it empty; register real
Agents with `central/scripts/register_agent.py` (see below).

### Persistence

- Postgres data lives in the named volume `postgres_data`. Back it up with
  `docker exec <postgres-container> pg_dump -U gti gti_fiscal > backup.sql`
  (or `pg_dumpall` for the whole cluster) on whatever schedule your
  operations policy requires.
- Always run `alembic upgrade head` (the container does this automatically)
  rather than hand-editing the schema. An **empty** database is initialized
  by that same command — never with `alembic stamp` or `create_all`. `0001` is
  an explicit, frozen baseline (it used to call `Base.metadata.create_all()`,
  which made `0002` collide on `datasets.raw_sha256` on any fresh database);
  `central/tests/test_migration_baseline.py` keeps that from coming back.
- `0019` repairs `agents.created_at/updated_at` and `jobs.created_at`, whose
  defaults `0006` had frozen at the instant it ran (rows created before `0019`
  keep that frozen value — their real creation time is not recoverable).
- `0020` enforces NOT NULL on 20 timestamp columns that always had
  `DEFAULT now()`. It aborts, changing nothing, if any NULL exists — clean
  those rows explicitly first. Current head: `0020_not_null_timestamps`
  (also recorded in `release/RELEASE_MANIFEST.json`).

### HTTPS / reverse proxy

The Hub itself speaks plain HTTP. For a pilot reachable outside the Hub's
own host, put it behind a reverse proxy (nginx, Caddy, Traefik — whatever
the GTI/Proxmox environment already standardizes on) that terminates TLS and
forwards to `fiscal-hub:8000`. Nothing in the app assumes a particular
proxy; `BASE_URL` in `.env` is informational only (not currently used to
generate absolute links, but keep it accurate for anyone reading the config).

## Registering an Agent (per client installation)

Run once per Agent installation, from anywhere with access to Postgres —
easiest is inside the running Hub container:

```bash
docker exec <fiscal-hub-container> python central/scripts/register_agent.py \
  --company "Client Company Name" --name "loja-01"
```

This prints `AGENT_ID`, a freshly generated `AGENT_TOKEN` (shown once — copy
it immediately), and `COMPANY_ID`. Only the token's SHA-256 hash is ever
stored in Postgres (same `token_hash` mechanism `/v1/classify` already used
— no new auth system). Each Agent gets its own token; one Agent's token
never authenticates another Agent's jobs (`agent_id` is checked on every
Job operation).

## Configuring an Agent installation

On the client machine, copy `agent/.env.example` to `agent/.env` and fill in
the values from registration plus that machine's Firebird details:

```
GTI_HUB_URL=https://hub.example.com
AGENT_ID=<from registration>
AGENT_TOKEN=<from registration>
COMPANY_ID=<from registration>
COMPANY_UF=BA
FIREBIRD_DATABASE_PATH=C:\ERP\dados\PRODUCAO.FDB
FB_USER=...
FB_PASSWORD=...
FIREBIRD_WRITES=0
POLL_INTERVAL_SECONDS=10
AGENT_VERSION=0.1.0
```

**`FIREBIRD_WRITES` must default to `0`.** See "Write safety" below before
ever setting it to `1`.

## Installing the Agent as a Windows Service

Requires `pywin32` (`pip install pywin32`, or `pip install -e ".[windows-service]"`
from `agent/`).

```powershell
# From an elevated (Administrator) terminal, in agent/service/:
python gti_agent_windows_service.py --startup=auto install
python gti_agent_windows_service.py start

# One-time, so Windows restarts it automatically after a crash:
sc.exe failure "GTIFiscalAgent" reset= 86400 actions= restart/60000/restart/60000/restart/60000
```

`gti_agent_windows_service.py` is a thin wrapper: it does not reimplement
polling, heartbeat, or the pipeline — it calls `run_agent_service.run()`
directly, stopping it via the Service Control Manager's own stop signal
instead of `SIGINT`/`SIGTERM` (which don't apply to services running with no
console). See the module's docstring for the exact commands.

### Install / configure / start / status / stop / restart / uninstall

```powershell
# Install (registers with SCM, Startup=Automatic)
python gti_agent_windows_service.py --startup=auto install

# Configure: edit agent/.env, no service restart needed until you `restart`
notepad agent\.env

# Start
python gti_agent_windows_service.py start

# Status
sc.exe query GTIFiscalAgent

# Stop
python gti_agent_windows_service.py stop

# Restart (picks up any agent/.env changes)
python gti_agent_windows_service.py restart

# Uninstall
python gti_agent_windows_service.py stop
python gti_agent_windows_service.py remove
```

### Updating the Agent's version

1. Stop the service: `python gti_agent_windows_service.py stop`
2. Replace the `agent/` code with the new version (git pull, or copy files).
3. Bump `AGENT_VERSION` in `agent/.env` if you track it per-release.
4. Start the service again: `python gti_agent_windows_service.py start`.
5. Confirm the new version shows up via `GET /v1/agents/me` on the Hub.

There is no auto-update mechanism yet (deliberately out of scope for this
pilot) — updates are a manual stop/replace/start.

### Running without installing as a service (dev/testing)

```bash
cd agent
python scripts/run_agent_service.py                 # runs forever, Ctrl+C to stop
python scripts/run_agent_service.py --max-iterations 5   # bounded, for scripting/tests
```

## Creating and checking work

See `docs/OPERATIONS.md` for the full `curl` examples
(`POST /v1/jobs` with `kind: REPROCESS_ALL` or `REPROCESS_PRODUCTS`,
`GET /v1/jobs/{id}` for status). Every Job is scoped to the Agent whose
token created it — no cross-Agent visibility.

## Checking Agent status

```bash
curl $GTI_HUB_URL/v1/agents/me -H "Authorization: Bearer $AGENT_TOKEN"
```

Returns `agent_id`, `company_id`, `enabled`, `last_seen_at`, `last_status`,
`schema_fingerprint`, `agent_version`, and a derived `status` of `ONLINE` or
`OFFLINE` (offline if no heartbeat arrived within
`AGENT_ONLINE_THRESHOLD_SECONDS`, default 120s — configurable on the Hub).
No dashboard yet; this is the query an operator or a future dashboard would
poll.

## Write safety: enabling FIREBIRD_WRITES for a real client

**Do not set `FIREBIRD_WRITES=1` against a real client database casually.**
The authorization model has two independent layers, both required:

1. **Path**: `AUTHORIZED_WRITE_DATABASE_PATH` in that installation's
   `agent/.env` must be set to the *exact* absolute path of
   `FIREBIRD_DATABASE_PATH` for that same installation. If it's unset, the
   guard falls back to the developer-only disposable test database path
   (`TESTESOFICIAIS.FDB`), so a real client path is refused automatically —
   authorization is never inferred from the filename or the `.FDB`
   extension.
2. **Schema fingerprint**: `EXPECTED_SCHEMA_FINGERPRINT` must equal the live
   schema fingerprint that installation reports in its heartbeat
   (`schema_fingerprint`, visible via `GET /v1/agents/me`). If the live
   schema ever drifts from what was verified, writes are refused until an
   operator re-verifies and updates this value.

Both checks happen before any write connection is opened
(`scripts/firebird_write_executor.py::assert_authorized_write_path` /
`assert_schema_fingerprint_authorized`), and before any product is even
read from Firebird for that job.

Even with both layers satisfied, auto-apply only ever touches
`C000025.IPI`, and only when the TIPI match is `rate_kind=NUMERIC` and
unambiguous. **No real client database has been written to during this or
any prior mission** — all writes so far were against `TESTESOFICIAIS.FDB`.

## Logs and diagnostics

- **Agent**: rotating local log file (`AGENT_LOG_PATH` in `agent/.env`,
  default `agent/logs/agent.log`, 5MB × 5 backups). Contains startup,
  heartbeat outcomes, job received/finished, products processed, write
  counts, failures, and Hub reconnects — never secrets (`FB_PASSWORD` and
  `AGENT_TOKEN` are never logged). Detailed per-product audit is in the
  Agent's SQLite file (`AGENT_STATE_DB`, table `ipi_write_log`).
- **Hub**: structured `logging` output (visible via `docker compose logs
  fiscal-hub`) for startup, health-check failures, heartbeats, and the full
  Job lifecycle (created/claimed/completed/failed, with sanitized errors).
  Coarse per-Job audit persists in Postgres (`jobs.result_summary`).

## Troubleshooting

| Symptom | Check |
|---|---|
| Agent shows `OFFLINE` | Is the service running (`sc.exe query GTIFiscalAgent`)? Check `agent/logs/agent.log` for heartbeat failures — usually `GTI_HUB_URL` unreachable or wrong `AGENT_TOKEN`. |
| Job stuck `PENDING` | Confirm the Agent that owns it is running and polling (its `agent_id` must match the Job's `agent_id` — an Agent never claims another Agent's Job). |
| Job `FAILED` | `GET /v1/jobs/{id}` for the sanitized `error`. Common causes: Firebird unreachable, `FIREBIRD_WRITES=1` misconfigured (see "Write safety"), Hub unreachable at report time (the Agent keeps the local `ipi_write_log` regardless). |
| `writes_attempted` always 0 with `FIREBIRD_WRITES=1` | Check both `AUTHORIZED_WRITE_DATABASE_PATH` and `EXPECTED_SCHEMA_FINGERPRINT` are set and correct for that installation. |
| Hub `/health` returns 503 | Postgres is down or unreachable; check `docker compose ps` and `docker compose logs postgres`. |

## Rollback / uninstalling an Agent

1. Stop and remove the Windows service (see above).
2. Delete the `agent/` installation directory on that machine (its SQLite
   audit and logs are local-only; nothing to clean up centrally).
3. On the Hub, disable the Agent rather than deleting history: set
   `enabled = false` for that `agents` row (no HTTP endpoint for this yet —
   run it via `psql`/a short script against `DATABASE_URL`, the same access
   `register_agent.py` uses). Its past Jobs and `classification_cache`
   entries remain for audit.

## Rolling back the Hub

`docker compose down` does not remove the `postgres_data` volume — data
survives. To roll back to a previous Hub image/version, redeploy that
version's code and run `alembic upgrade head` again (migrations are
additive, so this is safe going forward; there is no tested downgrade path
for `0006`, so don't `alembic downgrade` in a pilot without restoring from a
backup first).