sxchnge/docs/local-development.md

Local development

For the connected Sendico stand, see local Sendico integration.

Task #42, PR #43, branch dev-env-42.

Build and start

Requires Make, a POSIX shell and a local Docker engine with Compose v2 and BuildKit. No host Python, Go, Flutter, jq or curl is required. Make runs bootstrap tooling in Docker with the local Docker socket and workspace mounted. Remote Docker engines are unsupported.

make build && make up

The build resolves the latest stable releases and pins their digests in ignored .local/dev/images.env. Prereleases are excluded, resolution failures stop the build, and previous pins are retained on failure. Flutter is pinned to the stable release commit. Tooling uses a freshly pulled stable Docker CLI digest. Generated protobuf bindings and Flutter output are built locally/in Docker and never tracked.

Compose is at infra/local/compose.yml; Make supplies the repository root as its project directory. Dockerfiles are in infra/images/. Runtime and startup files are grouped by service under infra/local/. The stack consists of Vault, Mongo, NATS, notifications, BFF and Caddy serving the frontend.

The frontend container uses Compose init: true to reap orphaned ssl_client children created by BusyBox's HTTPS wget health probe. Keep this enabled: Caddy alone does not reap those children when it runs as PID 1, and repeated probes can accumulate thousands of zombie processes. Recreate the frontend container when applying this setting; restarting an old container does not change its init configuration.

On first startup, bootstrap initializes/unseals persistent Vault, provisions AppRoles and secrets, initializes Mongo replica set rs0, configures Vault's Mongo plugin, and starts authenticated NATS JetStream. Product startup fails explicitly until the SMTP mailbox password exists. Infrastructure remains available for provisioning.

After all services pass their startup health checks and the public Caddy CA is exported, make up prints a Sendico-style completion message with the frontend, API, API health and Vault URLs, the CA trust reminder, and status/log/stop commands. Failed startup does not print the success message. Mongo and NATS are private Compose services, so the summary does not advertise host URLs or credentials for them. This terminal-only change reuses Compose errors and the command exit status as startup outcome signals; it adds no analytics or secret output.

Completion-message validation: make test-dev-tooling and git diff --check passed. A containerized shell preview with stubbed infrastructure calls confirmed the success layout and that failed service startup or CA export exits nonzero without the message. The UX review checked readable plain-text labels, actionable URLs/commands and confirmed-success timing. This check did not restart the live stack or verify application readiness.

Vault layout

Paths below are inside the KV v2 mount kv; the API inserts data/ after kv/.

kv/
├── services/
│   ├── bff/jwt                         # jwt
│   └── notifications/
│       └── email/smtp                  # password only
└── infrastructure/
    ├── nats/auth                       # token; read by NATS and notifications
    └── mongo/bootstrap                 # local administrator and replica-set bootstrap material

Every secret has an owning service or infrastructure component. Service policies permit only the exact paths they consume. Applications cannot read bootstrap credentials. Mongo application users are dynamic secrets under database/creds/bff and database/creds/notifications, not KV entries. They have readWrite only on their respective sxchnge / sxchnge_notifications databases. Vault uses mongodb-database-plugin and its own vault-manager account, whose password is rotated immediately after initial configuration.

There are no frontend certificates, notification certificates or generic tls entries in KV. Browser certificates belong to Caddy; internal gRPC is plaintext on the trusted private Docker network without caller authentication.

Local SMTP mail

Open Vault and edit kv/services/notifications/email/smtp:

Field	Value
password	Password for the configured SMTP mailbox

The driver and sender settings belong in infra/local/notifications/runtime.yml:

email:
  driver: smtp
  settings:
    host: mail.sendico.io
    port: 465
    username: no-reply@sendico.io
    password_env: SMTP_PASSWORD
    network_timeout: 10
    sender_address: no-reply@sendico.io
    sender_name: "SXCHNGE"

The local stack uses Sendico's existing no-reply@sendico.io mailbox with authenticated implicit TLS on port 465 and certificate verification enabled. Vault Agent injects only the password; host, username and sender are non-secret runtime settings. Credentials are provisioned into SXCHNGE's own Vault; the notification service does not read Sendico configuration or secrets at runtime.

Local administrator login material is in private .local/dev/admin/init.json. Product containers cannot access it. Alternatively use the hidden terminal prompt:

make email
make build && make up

No placeholder password or delivery fallback is installed. Startup validates configuration and secret presence; the SMTP connection is opened on demand. Updating the password causes Agent to restart notifications. Driver/sender changes require an image rebuild. Old Resend secrets remain in Vault history but are not used or readable by the notifications AppRole. HTTP providers remain available for other deployments; switching one requires matching Agent/policy and runtime configuration. SMTP has no provider idempotency guarantee: explicit 4xx rejections retry, 5xx rejections fail, and ambiguous transport outcomes require operator reconciliation before another send.

Browser HTTPS and internal connections

Caddy serves Flutter files and proxies /api/* to BFF over HTTP. Open https://localhost:5126. HTTPS preserves Secure/HttpOnly cookies and same-origin API access. Caddy uses tls internal and manages its local CA, certificates and renewal in the caddy-data volume. The caddy-config volume stores Caddy runtime configuration. Neither volume nor Caddy has Vault access.

After frontend starts, Make exports only the public Caddy CA to .local/dev/ca.pem. Trust it in your browser/OS to avoid certificate errors. This is a new CA when migrating from Vault-issued browser certificates; the old CA will not trust it. Trust installation is not automated. A CLI check can use:

curl --cacert .local/dev/ca.pem https://localhost:5126/api/v1/health

BFF calls notifications over plaintext gRPC without caller authentication. Mongo and NATS also stay within the private Compose network; no internal service publishes a host port. Only frontend and Vault bind host loopback ports. TLS remains optional in the reusable notification client/server for deployments that explicitly configure it; local runtime configuration contains no certificate settings.

Caddy behavior follows its internal TLS and automatic HTTPS documentation.

Mongo transactions and credential rotation

Mongo is an authenticated single-node replica set (rs0), supporting multi-document transactions. It has no failover or redundant data copy. A Mongo restart interrupts access even though application credential replacement does not.

The shared database wrapper authenticates with each service's AppRole and obtains an independent token/lease for each pool. At half the shorter token/lease lifetime, it verifies a replacement pool before publishing it. Active operations, cursors and full transactions retain the old pool until they finish, then it closes and its token is revoked. Nested stores and commit retries keep the same pool.

Failed refreshes retain valid credentials and retry with bounded backoff. After expiry, database operations fail closed until automatic recovery. Administrative revocation, infrastructure outages and operations longer than their lease can still interrupt work. Repository consumers use source.Source operation scopes; source.Fixed adapts externally owned static/test connections.

Migration and persistence

Bootstrap migrates the old flat bff, email and nats paths into the hierarchy above, and moves bootstrap/mongo to infrastructure/mongo/bootstrap. It copies with CAS protection, compares values, and refuses to overwrite conflicting destinations. After consumers use the new paths, it removes old KV metadata and obsolete tls, frontend-tls, notifications-tls entries. Readable old version history and metadata are archived privately in .local/dev/migrations/organized-v1/ before removal. That archive is administrative recovery material; applications do not mount it. Old deleted/destroyed versions are not recreated. The old PKI engine is not used or provisioned by this stack; an existing PKI mount is retained to avoid destroying its non-exportable signing key and history as part of a KV reorganization.

Vault, Mongo, JetStream and Caddy data use project-scoped named volumes. Private bootstrap material lives in .local/dev/. make down preserves both. Vault's local unseal file allows automatic unsealing on restart. This local convenience and loopback HTTP Vault endpoint are for development. Keep bootstrap files paired with their matching volumes; do not discard one side independently.

Verification

make ps
make logs
make down
make test-dev-tooling   # shell analysis, failure paths, migration conflict checks
make test-dev-infra     # policy isolation, real Mongo transactions and lease lifecycle
make test-dev-rotation  # temporarily deploy BFF/frontend and exercise short leases
make test-dev           # requires all six services; verified frontend/API HTTPS

Rotation validation temporarily shortens the BFF role TTL, recreates BFF/frontend, and requires three replacements with unchanged BFF PID and zero unexpected transaction-backed HTTP responses. It restores TTLs and removes services it started. No verification command sends email. Backend validation remains make test and make lint-backend, using the host SDKs documented in module guides.

Bootstrap archives and removes obsolete caller and services/notifications/callers/bff KV entries after updating Agent policies/templates. NATS authentication remains enabled.