tech/sxchnge
1
0 Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

Organize local infrastructure and Vault secrets by service ownership

Browse Source
This commit is contained in:
Stephan D committed 2026-09-12 23:30:18 +02:00
1 parent 1b8800ee73
commit 9dbf0bd80c
44 files changed
+496 -359
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/implementation.md
+38
View File
@@ -705,3 +705,41 @@ were restored and temporary BFF/frontend containers removed. `make up` exits at
the intentional missing-Resend check, with infrastructure healthy. Backend and
frontend code did not change in this revision; their earlier test/lint results
and supplementary legacy test limitations above still apply. No email was sent.
### Organized infrastructure and Vault ownership
This revision supersedes the earlier local Nginx/Vault-certificate arrangement.
Local Compose, runtime configuration, startup scripts and verification now live
under `infra/local/`, grouped by service and responsibility. Dockerfiles live in
`infra/images/`; the root Make targets remain the entry point. Product source
stays in backend/frontend. Generated state remains in ignored `.local/dev/`.
Vault KV now contains `services/bff/jwt`,
`services/notifications/callers/bff`, `services/notifications/email/resend`,
`infrastructure/nats/auth` and `bootstrap/mongo`. Dynamic database credentials
remain in the database engine. Migration copies with CAS protection, compares
existing values, archives readable old versions and metadata privately, and then
removes old flat/TLS KV entries. Conflicting destinations stop migration without
removing sources. Policies and Agent templates reference only the new paths.
The unused existing PKI mount is retained to avoid destroying its signing key;
new bootstrap neither creates nor uses it.
Caddy replaces Nginx, serving Flutter and proxying `/api` to BFF. Caddy owns browser
certificates and its CA in persistent volumes; only its public CA is exported to
`.local/dev/ca.pem`. Existing browser trust must be updated to that new CA. Local
BFF-to-notifications uses plaintext authenticated gRPC. Optional explicitly
configured TLS remains available in the reusable client/server; incomplete TLS
configuration is rejected. No local service consumes TLS secrets from Vault.
Validation passed: `make build`, `make test`, `make lint-backend`,
`make test-dev-tooling`, `make test-dev-infra`, and Compose/diff checks. Tests cover
plaintext caller-token transport, optional verified TLS, invalid certificate
configuration and secret migration conflicts. The existing Vault was migrated
and canonical paths/absence of old entries verified. An isolated fresh project
passed transaction/lease checks and repeated bootstrap; disposable volumes and
state were removed. The final rotation check also verified Caddy static serving,
SPA fallback and API health, then three pool replacements with unchanged BFF PID
and 1,083 expected HTTP responses, zero failures. Role TTLs were restored and
temporary BFF/frontend containers removed. `make up` intentionally stops at the
missing Resend credential check; no real email, optional security scan or merge
was performed. Earlier supplementary legacy test limitations remain recorded.
docs/local-development.md
+114 -170
View File
@@ -3,219 +3,163 @@
Task [#42](https://git.sendico.io/tech/sxchnge/issues/42),
[PR #43](https://git.sendico.io/tech/sxchnge/pulls/43), branch `dev-env-42`.
## Build and bootstrap
## Build and start
Requires Make, a POSIX shell and a local Docker engine with Compose v2 and
BuildKit. Docker builds Go, protobuf bindings and Flutter web; host Go, protoc
and Flutter installations are not required for `make build && make up`. The
tooling container includes Docker CLI, curl, jq, Git and OpenSSL. No host Python,
jq, curl or Git installation is needed for these Make targets. It mounts
the local Docker socket and workspace to orchestrate Compose; a remote Docker
daemon is not supported. The first Flutter SDK download can
take several minutes. Both ARM64 and AMD64 build from the same source definitions.
BuildKit. No host Python, Go, Flutter, jq or curl is required. Make runs bootstrap
tooling in Docker with the local Docker socket and workspace mounted. Remote
Docker engines are unsupported.
```sh
make build && make up
```
`make build` resolves the current stable upstream releases, pulls infrastructure
and builds NATS packaging, notifications, BFF and frontend images. Vault and Mongo
use upstream images directly. Resolution excludes prereleases and selects Nginx's
stable channel. The exact versions and manifest digests are saved in the ignored
`.local/dev/images.env`; Flutter's stable release is pinned by Git commit. Build
fails if upstream metadata cannot be resolved. Run `make build` again to refresh
versions. Review database release upgrade requirements before updating a populated
local volume across major versions; the tooling does not perform data migrations.
The build resolves the latest stable releases and pins their digests in ignored
`.local/dev/images.env`. Prereleases are excluded, resolution failures stop the
build, and previous pins are retained on failure. Flutter is pinned to the stable
release commit. Tooling uses a freshly pulled stable Docker CLI digest. Generated
protobuf bindings and Flutter output are built locally/in Docker and never tracked.
`make up` starts persistent Vault, initializes/unseals it, provisions secrets and
AppRoles, initializes an authenticated single-member MongoDB replica set, configures
Vault's MongoDB database plugin, and starts authenticated NATS JetStream. It then
starts notifications, BFF and the frontend when the required email settings exist.
Compose is at `infra/local/compose.yml`; Make supplies the repository root as its
project directory. Dockerfiles are in `infra/images/`. Runtime and startup files
are grouped by service under `infra/local/`. The stack consists of Vault, Mongo,
NATS, notifications, BFF and Caddy serving the frontend.
**On the first run, `make up` intentionally exits with an error after infrastructure
bootstrap because Resend credentials have not been supplied.** No placeholder API
key, mock delivery mode or embedded credential is installed. The three infrastructure
services remain running so you can configure Vault, then rerun `make up`.
On first startup, bootstrap initializes/unseals persistent Vault, provisions
AppRoles and secrets, initializes Mongo replica set `rs0`, configures Vault's Mongo
plugin, and starts authenticated NATS JetStream. Product startup fails explicitly
until Resend credentials exist. Infrastructure remains available for provisioning.
## Configure Resend in Vault
## Vault layout
The default email driver is `resend`. In the Vault UI at
[http://127.0.0.1:18200](http://127.0.0.1:18200), edit the KV v2 secret **`kv/email`**
(API path `kv/data/email`):
Paths below are inside the KV v2 mount `kv`; the API inserts `data/` after `kv/`.
```text
kv/
├── services/
│ ├── bff/jwt # jwt
│ └── notifications/
│ ├── callers/bff # token; read by BFF and notifications
│ └── email/resend # driver, api_key, sender_address, sender_name
├── infrastructure/
│ └── nats/auth # token; read by NATS and notifications
└── bootstrap/
└── mongo # local administrator and replica-set bootstrap material
```
Every secret has an owning service or infrastructure component. Service policies
permit only the exact paths they consume. Applications cannot read bootstrap
credentials. Mongo application users are dynamic secrets under
`database/creds/bff` and `database/creds/notifications`, not KV entries. They have
`readWrite` only on their respective `sxchnge` / `sxchnge_notifications` databases.
Vault uses `mongodb-database-plugin` and its own `vault-manager` account, whose
password is rotated immediately after initial configuration.
There are no frontend certificates, notification certificates or generic `tls`
entries in KV. Browser certificates belong to Caddy; internal gRPC is plaintext
on the private Docker network and retains caller-token authentication.
## Resend
Open [Vault](http://127.0.0.1:18200) and edit
`kv/services/notifications/email/resend`:
| Field | Value |
| --- | --- |
| `driver` | `resend` |
| `api_key` | Your Resend API key |
| `sender_address` | Your verified Resend sender email |
| `sender_name` | `SXCHNGE` or your desired display name |
| `sender_address` | Your verified sender address |
| `sender_name` | Your display name |
Local administrative login material is in the private, ignored
`.local/dev/admin/init.json` (`root_token`). Keep that file private. Product
containers never mount it. You can also provision the same secret with a hidden
terminal prompt, without placing the API key in shell history:
Local administrator login material is in private `.local/dev/admin/init.json`.
Product containers cannot access it. Alternatively use the hidden terminal prompt:
```sh
make email
make up
```
The startup check rejects an empty key or the bootstrap sender placeholder. It
does not validate the key against Resend or prove domain verification; an invalid
key fails at delivery. Vault Agent fetches the key and email settings and injects
them into the notifications process. There is no `.env` secret fallback. Changing
`kv/email` restarts the notification process automatically within about 10 seconds.
Bootstrapping again preserves your existing email secret.
No placeholder key or delivery fallback is installed. Startup validates presence;
it does not send an email or validate domain ownership with Resend. Updating email
settings causes Agent to restart notifications. Existing values survive bootstrap.
## Open the product
## Browser HTTPS and internal connections
After providing Resend credentials:
Caddy serves Flutter files and proxies `/api/*` to BFF over HTTP. Open
[https://localhost:5126](https://localhost:5126). HTTPS preserves Secure/HttpOnly
cookies and same-origin API access. Caddy uses `tls internal` and manages its local
CA, certificates and renewal in the `caddy-data` volume. The `caddy-config` volume
stores Caddy runtime configuration. Neither volume nor Caddy has Vault access.
- Frontend and proxied API: [https://localhost:5126](https://localhost:5126).
- BFF health: [https://localhost:5126/api/v1/health](https://localhost:5126/api/v1/health).
- Vault UI/API: [http://127.0.0.1:18200](http://127.0.0.1:18200).
Trust `.local/dev/ca.pem` in your local browser/OS certificate store to use HTTPS
without a certificate error. Trust installation is a separate local action, not
performed by the bootstrap. For a command-line check without changing trust:
After frontend starts, Make exports only the public Caddy CA to `.local/dev/ca.pem`.
Trust it in your browser/OS to avoid certificate errors. This is a new CA when
migrating from Vault-issued browser certificates; the old CA will not trust it.
Trust installation is not automated. A CLI check can use:
```sh
curl --cacert .local/dev/ca.pem https://localhost:5126/api/v1/health
```
HTTPS keeps the existing Secure/HttpOnly refresh cookie contract. Nginx serves
Flutter and proxies `/api/` to the BFF on the same origin. The bundle selects the
existing backend auth adapter with the public client ID `sxchnge-local-web`.
Bootstrap inserts this public browser client if absent; it has no client secret.
No frontend widget, service or provider architecture changes are needed.
BFF calls notifications over plaintext gRPC with its caller token. Mongo and NATS
also stay within the private Compose network; no internal service publishes a host
port. Only frontend and Vault bind host loopback ports. TLS remains optional in
the reusable notification client/server for deployments that explicitly configure
it; local runtime configuration contains no certificate settings.
The rest of the app retains its existing capabilities and limitations: wallet,
KYC and settings fixtures are not turned into production integrations by this
container setup. Full email signup/login requires a working Resend account.
Caddy behavior follows its [internal TLS](https://caddyserver.com/docs/caddyfile/directives/tls)
and [automatic HTTPS](https://caddyserver.com/docs/automatic-https) documentation.
## Secret and connection lifecycle
## Mongo transactions and credential rotation
| Consumer | Vault access |
| --- | --- |
| BFF | `database/creds/bff`, JWT key, notification caller token and CA |
| Notifications | `database/creds/notifications`, NATS token, caller token, TLS certificate/key and email settings |
| NATS | NATS authentication token only |
| Bootstrap | Infrastructure administration, PKI and initial identities |
Mongo is an authenticated single-node replica set (`rs0`), supporting multi-document
transactions. It has no failover or redundant data copy. A Mongo restart interrupts
access even though application credential replacement does not.
The database engine is configured at `database/config/mongo` with
`mongodb-database-plugin`. It uses its own `vault-manager` Mongo user, whose
password is rotated by Vault immediately after configuration. The manager has
`userAdminAnyDatabase`; applications never receive that credential. The independent
local Mongo administrator and replica-set key come from Vault's random generator
and are retained in `kv/bootstrap/mongo` for local provisioning.
The shared database wrapper authenticates with each service's AppRole and obtains
an independent token/lease for each pool. At half the shorter token/lease lifetime,
it verifies a replacement pool before publishing it. Active operations, cursors
and full transactions retain the old pool until they finish, then it closes and
its token is revoked. Nested stores and commit retries keep the same pool.
The BFF role has `readWrite` only on `sxchnge`; notifications has `readWrite` only
on `sxchnge_notifications`. Mongo URIs come from dynamic Vault credentials, not
static application users. Leases default to one hour with a 24-hour maximum.
The shared database wrapper obtains credentials directly from Vault using the
service's AppRole. At half the effective lifetime (the shorter of the database
lease and owning token TTL), it authenticates independently and prepares a new
Mongo pool. Only a successful primary ping permits publication. New operations
use that pool; active operations, cursors and complete transactions, including
nested stores and commit retries, retain the old pool until they finish. The old
pool then closes and its dedicated token is revoked. Mongo credential replacement
never restarts BFF or notifications. Each generation has a separate token so
cleanup cannot revoke the replacement or Vault Agent's identity.
Failed refreshes retain valid credentials and retry with bounded backoff. After
expiry, database operations fail closed until automatic recovery. Administrative
revocation, infrastructure outages and operations longer than their lease can
still interrupt work. Repository consumers use `source.Source` operation scopes;
`source.Fixed` adapts externally owned static/test connections.
Refresh failures keep the current valid pool and retry with bounded backoff.
Operations are bounded by credential expiry. If Vault remains unavailable beyond
that deadline, new database operations fail closed; refresh continues and access
recovers automatically without restarting the service. Administrative revocation,
operations exceeding their lease and infrastructure failures cannot be made
uninterrupted by pool rotation. Shutdown rejects new scopes and drains existing
work. Failed cleanup relies on Vault expiration as a backstop.
## Migration and persistence
Repositories and transaction factories share `source.Source` rather than retaining
concrete Mongo collections. Low-level consumers must acquire a scope, use its
returned context and release it after all work completes; never retain its client
or context beyond release. `source.Fixed` adapts externally owned test/static
connections. Database credentials are no longer Agent environment templates;
Agent continues to fetch non-database secrets and may restart children when those
secrets change.
Bootstrap migrates the old flat `bff`, `caller`, `email` and `nats` paths into the
hierarchy above. It copies with CAS protection, compares values, and refuses to
overwrite conflicting destinations. After consumers use the new paths, it removes
old KV metadata and obsolete `tls`, `frontend-tls`, `notifications-tls` entries.
Readable old version history and metadata are archived privately in
`.local/dev/migrations/organized-v1/` before removal. That archive is administrative
recovery material; applications do not mount it. Old deleted/destroyed versions
are not recreated. The old PKI engine is not used or provisioned by this stack;
an existing PKI mount is retained to avoid destroying its non-exportable signing
key and history as part of a KV reorganization.
Each agent authenticates using its own AppRole with a renewable periodic token.
Local SecretIDs are reusable for container restarts; each container mounts only
its own identity directory under `.local/dev/agents/`. Revoking an AppRole SecretID
requires removing its matching local `secret-id` file and rerunning `make up` to
issue a replacement. Revoking an active token/lease terminates its credentials;
The database wrapper retries with a fresh identity; Agent independently retries
its non-database secrets when Vault is reachable.
Vault, Mongo, JetStream and Caddy data use project-scoped named volumes. Private
bootstrap material lives in `.local/dev/`. `make down` preserves both. Vault's local
unseal file allows automatic unsealing on restart. This local convenience and
loopback HTTP Vault endpoint are for development. Keep bootstrap files paired
with their matching volumes; do not discard one side independently.
The notification gRPC connection uses a Vault-issued certificate for
`notifications` and the BFF validates the Vault CA and hostname. Frontend HTTPS
certificates also come from Vault PKI. Leaf certificates last 30 days; `make up`
refreshes them when less than seven days remain and reloads Nginx. Run it regularly
for a development stack kept online for weeks. Agents restart when their TLS KV
secret changes. Application secrets are never put into the frontend bundle.
Vault, Mongo and JetStream data use project-scoped named Docker volumes. Local
Vault unseal material, administrator login, per-service bootstrap identities and
frontend TLS material are under ignored `.local/dev/`, outside image build contexts.
The local Vault automatically unseals using its local key on container restart.
These convenience credentials and loopback HTTP Vault access are for this local
machine; this is not a production Vault deployment. Do not share `.local/dev/`.
Keep it paired with its matching Docker volumes for recovery.
Mongo and NATS expose no host ports. The `sxchnge-local` network, volumes and
container names are independent of existing Sendico stacks. Only the frontend
and Vault bind host loopback ports. No Sendico database, message bus or secrets
are consumed.
## Manage and verify
## Verification
```sh
make ps
make logs
make down # preserves named volumes and local bootstrap material
make up # idempotent provisioning and readiness checks
make test-dev-tooling # offline shell and failure-path checks
make test-dev-infra # works before Resend provisioning
make test-dev-rotation # real Vault rotations under BFF HTTP traffic
make test-dev # requires all six services running
make down
make test-dev-tooling # shell analysis, failure paths, migration conflict checks
make test-dev-infra # policy isolation, real Mongo transactions and lease lifecycle
make test-dev-rotation # temporarily deploy BFF/frontend and exercise short leases
make test-dev # requires all six services; verified frontend/API HTTPS
```
The infrastructure check exercises AppRole access restrictions, real Vault
MongoDB credential issuance, per-database access, lease renewal and user deletion
on revocation. The complete check also verifies service health and frontend/API
HTTPS with the generated CA. These checks never send a real email.
`test-dev-rotation` temporarily sets a 12-second BFF lease, deploys the built BFF
and frontend, and requires three pool replacements with unchanged process ID and
zero unexpected HTTP responses. Its invalid-token requests exercise Mongo-backed
transactions without sending email. The check restores the role TTL and removes
services it started; it recreates BFF/frontend at the beginning, so run it as a
development validation command, not against active work you need to preserve.
Race-enabled backend tests additionally hold a transaction and cursor across a
replacement and exercise failed candidate validation, outage recovery, rollback,
expiry rejection and graceful shutdown.
Backend regression tests use fresh `mongo:latest` and `nats:latest` official
stable images, pulled before `make test-backend`. Normal validation remains
`make test` and `make lint-backend` and requires the host SDKs described in the
repository guides.
Deleting Docker volumes removes local data; `make down` deliberately keeps them.
If starting from scratch, stop the stack and explicitly remove this project's
three volumes **and** its matching `.local/dev/` bootstrap state together. Never
reuse old unseal material with a newly initialized Vault volume, or retain Mongo
users while discarding the Vault data that manages their leases.
## Sources and checked releases
On 2026-09-12 the resolver selected Vault 2.1.0, MongoDB 8.3.9, NATS 2.14.6,
Go 1.27.1, Nginx stable 1.30.4, Ubuntu 26.04 and Flutter stable 3.47.4. These are
verification records, not fixed future defaults. Release selection follows the
[Vault releases](https://releases.hashicorp.com/vault/),
[MongoDB stable release notes](https://www.mongodb.com/docs/manual/release-notes/),
[NATS releases](https://github.com/nats-io/nats-server/releases/latest),
[Nginx downloads](https://nginx.org/en/download.html),
[Go releases](https://go.dev/dl/) and
[Flutter stable channel](https://github.com/flutter/flutter/tree/stable).
The implementation follows HashiCorp's
[MongoDB database plugin](https://developer.hashicorp.com/vault/docs/secrets/databases/mongodb)
and [Agent process supervisor](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent/process-supervisor)
contracts.
Rotation validation temporarily shortens the BFF role TTL, recreates BFF/frontend,
and requires three replacements with unchanged BFF PID and zero unexpected
transaction-backed HTTP responses. It restores TTLs and removes services it
started. No verification command sends email. Backend validation remains
`make test` and `make lint-backend`, using the host SDKs documented in module guides.