Run the local product stack with Vault-managed secrets #42

Closed
opened 2026-09-12 18:43:37 +00:00 by tech · 0 comments
Owner

Goal

Provide an organized local development stack with current stable containers, service-owned Vault secrets, Mongo transactions and seamless credential replacement.

Approach

make build && make up builds and bootstraps isolated Vault, transaction-capable Mongo replica set, NATS JetStream, notifications, BFF and a Caddy-served Flutter frontend. Infrastructure sources are grouped under infra/local/ by service/responsibility; Dockerfiles are under infra/images/. Make invokes containerized shell tooling, with no Python dependency. Releases are resolved to latest stable versions and pinned by digest.

Vault KV uses services/bff/jwt, services/notifications/email/resend, infrastructure/nats/auth and infrastructure/mongo/bootstrap. Migration preserves values with CAS-protected copying and equality checks, refuses conflicts, archives readable old versions privately, and removes old flat/TLS KV entries. Agent policies and templates use exact owned paths. Mongo credentials remain dynamic through mongodb-database-plugin; verified replacement pools drain active transactions/cursors without restarting services.

Caddy owns browser HTTPS certificates in persistent volumes, serves frontend files and proxies the API. BFF-to-notifications is plaintext gRPC without caller authentication on the private Docker network. Optional explicit TLS remains available outside the local configuration. Caddy's public CA is exported to .local/dev/ca.pem; browser trust must be updated from the old CA. The unused legacy PKI mount is retained to avoid destroying its signing key/history; new bootstrap does not provision it.

Acceptance criteria

  • Make builds/deploys infrastructure and product images without Python.
  • Repository infrastructure is grouped by responsibility and service.
  • Vault values are migrated safely into explicit ownership paths; old flat/TLS KV entries are absent.
  • Caddy manages browser HTTPS; internal local gRPC uses plaintext without caller authentication.
  • Mongo transactions, database-plugin credential lifecycle and uninterrupted planned rotation are verified.
  • Missing Resend credentials fail startup after infrastructure bootstrap.
  • Required validation, documentation, commit, push and PR update are complete.

Notes

Validation passed: make build, make test, make lint-backend, make test-dev-tooling, make test-dev-infra, Compose configuration and diff checks. The real Vault migration and canonical layout were verified. An isolated fresh project passed transaction/lease checks and repeated bootstrap; disposable resources were removed. make test-dev-rotation verified Caddy static serving, SPA fallback, API health and three replacements with unchanged BFF PID: 1,083 expected HTTP responses, zero failures. Role TTLs were restored and temporary product containers removed.

Resend defaults to resend; make up intentionally stops after healthy infrastructure bootstrap until the developer provides api_key at kv/services/notifications/email/resend (or make email) and configures the verified sender in infra/local/notifications/runtime.yml. No placeholder credentials or live email. Mongo is a single-node replica set; infrastructure failures and outages beyond credential expiry can interrupt service. Earlier supplementary integration-tag failures (legacy BSON-map and intermittent timestamp assertions) remain documented in docs/implementation.md. No optional security scan or merge. Branch dev-env-42, commit 9dbf0bd. See docs/local-development.md and infra/README.md.

PR #43: #43. Owner tech; project SXCHNGE; labels backend and frontend.

Mongo bootstrap credentials were subsequently migrated to kv/infrastructure/mongo/bootstrap; the old kv/bootstrap/mongo path is absent. Existing values were preserved and compared. Shell and live infrastructure checks passed, including transaction and lease lifecycle checks and application denial of administrative credentials. Latest commit: 9d57a65.

Email configuration is now outside Vault: the live kv/services/notifications/email/resend secret contains only api_key. Driver and sender settings live in infra/local/notifications/runtime.yml; the sender is intentionally empty until configured. make email prompts only for the key. Agent injects the key without generating YAML. CAS-checked migration preserves it and rejects unknown fields. Image build, shell/migration checks, live infrastructure checks (including key-only secret shape), and diff checks passed. Missing API key still blocks startup; sender configuration also requires rebuilding the image. No backend source change in that email-only correction, and no email delivery. Latest commit: 1fe1733.

Removed BFF-to-notifications bearer authentication, token configuration, Vault templates and policy access. The private gRPC endpoint retains the existing product storage namespace for receipt and idempotency compatibility. Bootstrap archives/removes obsolete caller secrets, and live Vault no longer contains them. NATS authentication remains enabled. Regression tests cover credential-free gRPC Send/Get and replay, no outgoing bearer metadata, and repeatable retirement preserving NATS authentication. make build, make test, make lint-backend, make test-dev-tooling, make test-dev-infra and diff checks passed. make up applied the migration, then intentionally failed on the missing Resend API key. No email sent; full product deployment awaits the key and sender configuration. Commit caad7a3.

Startup completion message (ce6583f): make up now ends with a Sendico-style success heading, frontend/API/health/Vault URLs, a public CA trust reminder, and make ps, make logs, and make down. It appears only after service health checks and CA export succeed, and prints no credentials. make test-dev-tooling, git diff --check, and a containerized shell smoke check passed; the smoke check previewed success and verified nonzero exits with no success message after simulated startup/CA export failures. UX review covered readable labels, actionable next steps and confirmed-success timing, reusing command exit status and Compose diagnostics without new analytics. Live startup was not rerun for this message-only change. The user's local sender configuration remains uncommitted and unchanged. Details: docs/local-development.md.

### Goal Provide an organized local development stack with current stable containers, service-owned Vault secrets, Mongo transactions and seamless credential replacement. ### Approach `make build && make up` builds and bootstraps isolated Vault, transaction-capable Mongo replica set, NATS JetStream, notifications, BFF and a Caddy-served Flutter frontend. Infrastructure sources are grouped under `infra/local/` by service/responsibility; Dockerfiles are under `infra/images/`. Make invokes containerized shell tooling, with no Python dependency. Releases are resolved to latest stable versions and pinned by digest. Vault KV uses `services/bff/jwt`, `services/notifications/email/resend`, `infrastructure/nats/auth` and `infrastructure/mongo/bootstrap`. Migration preserves values with CAS-protected copying and equality checks, refuses conflicts, archives readable old versions privately, and removes old flat/TLS KV entries. Agent policies and templates use exact owned paths. Mongo credentials remain dynamic through `mongodb-database-plugin`; verified replacement pools drain active transactions/cursors without restarting services. Caddy owns browser HTTPS certificates in persistent volumes, serves frontend files and proxies the API. BFF-to-notifications is plaintext gRPC without caller authentication on the private Docker network. Optional explicit TLS remains available outside the local configuration. Caddy's public CA is exported to `.local/dev/ca.pem`; browser trust must be updated from the old CA. The unused legacy PKI mount is retained to avoid destroying its signing key/history; new bootstrap does not provision it. ### Acceptance criteria - [x] Make builds/deploys infrastructure and product images without Python. - [x] Repository infrastructure is grouped by responsibility and service. - [x] Vault values are migrated safely into explicit ownership paths; old flat/TLS KV entries are absent. - [x] Caddy manages browser HTTPS; internal local gRPC uses plaintext without caller authentication. - [x] Mongo transactions, database-plugin credential lifecycle and uninterrupted planned rotation are verified. - [x] Missing Resend credentials fail startup after infrastructure bootstrap. - [x] Required validation, documentation, commit, push and PR update are complete. ### Notes Validation passed: `make build`, `make test`, `make lint-backend`, `make test-dev-tooling`, `make test-dev-infra`, Compose configuration and diff checks. The real Vault migration and canonical layout were verified. An isolated fresh project passed transaction/lease checks and repeated bootstrap; disposable resources were removed. `make test-dev-rotation` verified Caddy static serving, SPA fallback, API health and three replacements with unchanged BFF PID: 1,083 expected HTTP responses, zero failures. Role TTLs were restored and temporary product containers removed. Resend defaults to `resend`; `make up` intentionally stops after healthy infrastructure bootstrap until the developer provides `api_key` at `kv/services/notifications/email/resend` (or `make email`) and configures the verified sender in `infra/local/notifications/runtime.yml`. No placeholder credentials or live email. Mongo is a single-node replica set; infrastructure failures and outages beyond credential expiry can interrupt service. Earlier supplementary integration-tag failures (legacy BSON-map and intermittent timestamp assertions) remain documented in docs/implementation.md. No optional security scan or merge. Branch `dev-env-42`, commit `9dbf0bd`. See docs/local-development.md and infra/README.md. PR #43: https://git.sendico.io/tech/sxchnge/pulls/43. Owner tech; project SXCHNGE; labels backend and frontend. Mongo bootstrap credentials were subsequently migrated to `kv/infrastructure/mongo/bootstrap`; the old `kv/bootstrap/mongo` path is absent. Existing values were preserved and compared. Shell and live infrastructure checks passed, including transaction and lease lifecycle checks and application denial of administrative credentials. Latest commit: 9d57a65. Email configuration is now outside Vault: the live `kv/services/notifications/email/resend` secret contains only `api_key`. Driver and sender settings live in `infra/local/notifications/runtime.yml`; the sender is intentionally empty until configured. `make email` prompts only for the key. Agent injects the key without generating YAML. CAS-checked migration preserves it and rejects unknown fields. Image build, shell/migration checks, live infrastructure checks (including key-only secret shape), and diff checks passed. Missing API key still blocks startup; sender configuration also requires rebuilding the image. No backend source change in that email-only correction, and no email delivery. Latest commit: 1fe1733. Removed BFF-to-notifications bearer authentication, token configuration, Vault templates and policy access. The private gRPC endpoint retains the existing product storage namespace for receipt and idempotency compatibility. Bootstrap archives/removes obsolete caller secrets, and live Vault no longer contains them. NATS authentication remains enabled. Regression tests cover credential-free gRPC Send/Get and replay, no outgoing bearer metadata, and repeatable retirement preserving NATS authentication. `make build`, `make test`, `make lint-backend`, `make test-dev-tooling`, `make test-dev-infra` and diff checks passed. `make up` applied the migration, then intentionally failed on the missing Resend API key. No email sent; full product deployment awaits the key and sender configuration. Commit caad7a3. Startup completion message (ce6583f): `make up` now ends with a Sendico-style success heading, frontend/API/health/Vault URLs, a public CA trust reminder, and `make ps`, `make logs`, and `make down`. It appears only after service health checks and CA export succeed, and prints no credentials. `make test-dev-tooling`, `git diff --check`, and a containerized shell smoke check passed; the smoke check previewed success and verified nonzero exits with no success message after simulated startup/CA export failures. UX review covered readable labels, actionable next steps and confirmed-success timing, reusing command exit status and Compose diagnostics without new analytics. Live startup was not rerun for this message-only change. The user's local sender configuration remains uncommitted and unchanged. Details: docs/local-development.md.
tech added the backendfrontend labels 2026-09-12 18:43:37 +00:00
tech self-assigned this 2026-09-12 18:43:37 +00:00
tech added this to the SXCHNGE project 2026-09-12 18:43:37 +00:00
tech added reference dev-env-42 2026-09-12 18:43:57 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 20:20:21 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 21:03:12 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 21:15:04 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 21:18:16 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 21:30:53 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 21:37:03 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 21:46:23 +00:00
tech changed reference from dev-env-42 to dev-env-42 2026-09-12 22:00:09 +00:00
tech closed this issue 2026-09-13 08:35:02 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: tech/sxchnge#42