sxchnge/docs/local-development.md

# Local development

For the connected Sendico stand, see [local Sendico integration](local-sendico.md).

Task [#42](https://git.sendico.io/tech/sxchnge/issues/42),
[PR #43](https://git.sendico.io/tech/sxchnge/pulls/43), branch `dev-env-42`.

## Build and start

Requires Make, a POSIX shell and a local Docker engine with Compose v2 and
BuildKit. No host Python, Go, Flutter, jq or curl is required. Make runs bootstrap
tooling in Docker with the local Docker socket and workspace mounted. Remote
Docker engines are unsupported.

```sh
make build && make up
```

The build resolves the latest stable releases and pins their digests in ignored
`.local/dev/images.env`. Prereleases are excluded, resolution failures stop the
build, and previous pins are retained on failure. Flutter is pinned to the stable
release commit. Tooling uses a freshly pulled stable Docker CLI digest. Generated
protobuf bindings and Flutter output are built locally/in Docker and never tracked.

Compose is at `infra/local/compose.yml`; Make supplies the repository root as its
project directory. Dockerfiles are in `infra/images/`. Runtime and startup files
are grouped by service under `infra/local/`. The stack consists of Vault, Mongo,
NATS, notifications, BFF and Caddy serving the frontend.

The frontend container uses Compose `init: true` to reap orphaned `ssl_client`
children created by BusyBox's HTTPS `wget` health probe. Keep this enabled:
Caddy alone does not reap those children when it runs as PID 1, and repeated
probes can accumulate thousands of zombie processes. Recreate the frontend
container when applying this setting; restarting an old container does not
change its init configuration.

On first startup, bootstrap initializes/unseals persistent Vault, provisions
AppRoles and secrets, initializes Mongo replica set `rs0`, configures Vault's Mongo
plugin, and starts authenticated NATS JetStream. Product startup fails explicitly
until the SMTP mailbox password exists. Infrastructure remains available for provisioning.

After all services pass their startup health checks and the public Caddy CA is
exported, `make up` prints a Sendico-style completion message with the frontend,
API, API health and Vault URLs, the CA trust reminder, and status/log/stop commands.
Failed startup does not print the success message. Mongo and NATS are private
Compose services, so the summary does not advertise host URLs or credentials for
them. This terminal-only change reuses Compose errors and the command exit status
as startup outcome signals; it adds no analytics or secret output.

Completion-message validation: `make test-dev-tooling` and `git diff --check`
passed. A containerized shell preview with stubbed infrastructure calls confirmed
the success layout and that failed service startup or CA export exits nonzero
without the message. The UX review checked readable plain-text labels, actionable
URLs/commands and confirmed-success timing. This check did not restart the live
stack or verify application readiness.

## Vault layout

Paths below are inside the KV v2 mount `kv`; the API inserts `data/` after `kv/`.

```text
kv/
├── services/
│   ├── bff/jwt                         # jwt
│   └── notifications/
│       └── email/smtp                  # password only
└── infrastructure/
    ├── nats/auth                       # token; read by NATS and notifications
    └── mongo/bootstrap                 # local administrator and replica-set bootstrap material
```

Every secret has an owning service or infrastructure component. Service policies
permit only the exact paths they consume. Applications cannot read bootstrap
credentials. Mongo application users are dynamic secrets under
`database/creds/bff` and `database/creds/notifications`, not KV entries. They have
`readWrite` only on their respective `sxchnge` / `sxchnge_notifications` databases.
Vault uses `mongodb-database-plugin` and its own `vault-manager` account, whose
password is rotated immediately after initial configuration.

There are no frontend certificates, notification certificates or generic `tls`
entries in KV. Browser certificates belong to Caddy; internal gRPC is plaintext
on the trusted private Docker network without caller authentication.

## Local SMTP mail

Open [Vault](http://127.0.0.1:18200) and edit
`kv/services/notifications/email/smtp`:

| Field | Value |
| --- | --- |
| `password` | Password for the configured SMTP mailbox |

The driver and sender settings belong in `infra/local/notifications/runtime.yml`:

```yaml
email:
  driver: smtp
  settings:
    host: mail.sendico.io
    port: 465
    username: no-reply@sendico.io
    password_env: SMTP_PASSWORD
    network_timeout: 10
    sender_address: no-reply@sendico.io
    sender_name: "SXCHNGE"
```

The local stack uses Sendico's existing `no-reply@sendico.io` mailbox with
authenticated implicit TLS on port 465 and certificate verification enabled.
Vault Agent injects only the password; host, username and sender are non-secret
runtime settings. Credentials are provisioned into SXCHNGE's own Vault; the
notification service does not read Sendico configuration or secrets at runtime.

Local administrator login material is in private `.local/dev/admin/init.json`.
Product containers cannot access it. Alternatively use the hidden terminal prompt:

```sh
make email
make build && make up
```

No placeholder password or delivery fallback is installed. Startup validates
configuration and secret presence; the SMTP connection is opened on demand.
Updating the password causes Agent to restart notifications. Driver/sender
changes require an image rebuild. Old Resend secrets remain in Vault history
but are not used or readable by the notifications AppRole. HTTP providers remain
available for other deployments; switching one requires matching Agent/policy
and runtime configuration. SMTP has no provider idempotency guarantee: explicit
4xx rejections retry, 5xx rejections fail, and ambiguous transport outcomes require
operator reconciliation before another send.

## Browser HTTPS and internal connections

Caddy serves Flutter files and proxies `/api/*` to BFF over HTTP. Open
[https://localhost:5126](https://localhost:5126). HTTPS preserves Secure/HttpOnly
cookies and same-origin API access. Caddy uses `tls internal` and manages its local
CA, certificates and renewal in the `caddy-data` volume. The `caddy-config` volume
stores Caddy runtime configuration. Neither volume nor Caddy has Vault access.

After frontend starts, Make exports only the public Caddy CA to `.local/dev/ca.pem`.
Trust it in your browser/OS to avoid certificate errors. This is a new CA when
migrating from Vault-issued browser certificates; the old CA will not trust it.
Trust installation is not automated. A CLI check can use:

```sh
curl --cacert .local/dev/ca.pem https://localhost:5126/api/v1/health
```

BFF calls notifications over plaintext gRPC without caller authentication. Mongo and NATS
also stay within the private Compose network; no internal service publishes a host
port. Only frontend and Vault bind host loopback ports. TLS remains optional in
the reusable notification client/server for deployments that explicitly configure
it; local runtime configuration contains no certificate settings.

Caddy behavior follows its [internal TLS](https://caddyserver.com/docs/caddyfile/directives/tls)
and [automatic HTTPS](https://caddyserver.com/docs/automatic-https) documentation.

## Mongo transactions and credential rotation

Mongo is an authenticated single-node replica set (`rs0`), supporting multi-document
transactions. It has no failover or redundant data copy. A Mongo restart interrupts
access even though application credential replacement does not.

The shared database wrapper authenticates with each service's AppRole and obtains
an independent token/lease for each pool. At half the shorter token/lease lifetime,
it verifies a replacement pool before publishing it. Active operations, cursors
and full transactions retain the old pool until they finish, then it closes and
its token is revoked. Nested stores and commit retries keep the same pool.

Failed refreshes retain valid credentials and retry with bounded backoff. After
expiry, database operations fail closed until automatic recovery. Administrative
revocation, infrastructure outages and operations longer than their lease can
still interrupt work. Repository consumers use `source.Source` operation scopes;
`source.Fixed` adapts externally owned static/test connections.

## Migration and persistence

Bootstrap migrates the old flat `bff`, `email` and `nats` paths into the
hierarchy above, and moves `bootstrap/mongo` to `infrastructure/mongo/bootstrap`.
It copies with CAS protection, compares values, and refuses to
overwrite conflicting destinations. After consumers use the new paths, it removes
old KV metadata and obsolete `tls`, `frontend-tls`, `notifications-tls` entries.
Readable old version history and metadata are archived privately in
`.local/dev/migrations/organized-v1/` before removal. That archive is administrative
recovery material; applications do not mount it. Old deleted/destroyed versions
are not recreated. The old PKI engine is not used or provisioned by this stack;
an existing PKI mount is retained to avoid destroying its non-exportable signing
key and history as part of a KV reorganization.

Vault, Mongo, JetStream and Caddy data use project-scoped named volumes. Private
bootstrap material lives in `.local/dev/`. `make down` preserves both. Vault's local
unseal file allows automatic unsealing on restart. This local convenience and
loopback HTTP Vault endpoint are for development. Keep bootstrap files paired
with their matching volumes; do not discard one side independently.

## Verification

```sh
make ps
make logs
make down
make test-dev-tooling   # shell analysis, failure paths, migration conflict checks
make test-dev-infra     # policy isolation, real Mongo transactions and lease lifecycle
make test-dev-rotation  # temporarily deploy BFF/frontend and exercise short leases
make test-dev           # requires all six services; verified frontend/API HTTPS
```

Rotation validation temporarily shortens the BFF role TTL, recreates BFF/frontend,
and requires three replacements with unchanged BFF PID and zero unexpected
transaction-backed HTTP responses. It restores TTLs and removes services it
started. No verification command sends email. Backend validation remains
`make test` and `make lint-backend`, using the host SDKs documented in module guides.

Bootstrap archives and removes obsolete `caller` and `services/notifications/callers/bff`
KV entries after updating Agent policies/templates. NATS authentication remains enabled.