Run local infrastructure and product images with Vault-managed secrets #43

Merged
tech merged 12 commits from dev-env-42 into main 2026-09-13 08:35:02 +00:00
12 changed files with 84 additions and 25 deletions
Showing only changes of commit 1fe173397f - Show all commits

No files matched your search

+2
View File
@@ -181,3 +181,5 @@ belong to the receiving service at `services/<service>/callers/<caller>`.
Use Caddy for local browser HTTPS and certificate management. Treat the private Use Caddy for local browser HTTPS and certificate management. Treat the private
local Docker network as trusted; do not add internal gRPC TLS or Vault-managed local Docker network as trusted; do not add internal gRPC TLS or Vault-managed
local TLS certificates without an explicit requirement. local TLS certificates without an explicit requirement.
Store only secret values in Vault. Driver selection, sender identities, endpoints
and other non-secret runtime settings belong in service configuration files.
+2 -1
View File
@@ -40,7 +40,8 @@ make build && make up
This builds the product images and starts an isolated Vault, authenticated MongoDB This builds the product images and starts an isolated Vault, authenticated MongoDB
replica set and NATS JetStream. First startup intentionally stops after bootstrap replica set and NATS JetStream. First startup intentionally stops after bootstrap
until you set the Resend API key and verified sender in Vault's `kv/services/notifications/email/resend` secret. until you set `api_key` in Vault's `kv/services/notifications/email/resend` secret
and the verified sender in `infra/local/notifications/runtime.yml`.
Then rerun `make up` to start notifications, BFF and the frontend at Then rerun `make up` to start notifications, BFF and the frontend at
[https://localhost:5126](https://localhost:5126). [https://localhost:5126](https://localhost:5126).
+15
View File
@@ -751,3 +751,18 @@ checks read the new path; application identities are denied both paths. Shell
checks and live infrastructure validation (including Mongo transactions, lease checks and live infrastructure validation (including Mongo transactions, lease
renewal/revocation and obsolete-path absence) passed. `make up` still stops at the renewal/revocation and obsolete-path absence) passed. `make up` still stops at the
expected missing-Resend check. No backend code or product behavior changed. expected missing-Resend check. No backend code or product behavior changed.
Email configuration is now separated from credentials. The current Vault secret
`kv/services/notifications/email/resend` contains only `api_key`; `driver`,
`sender_address` and `sender_name` live in
`infra/local/notifications/runtime.yml`. Sender address is empty until configured.
Agent injects only `EMAIL_API_KEY`, and notifications loads its runtime YAML
directly. `make email` prompts only for the API key. The migration preserves the
key with a CAS-checked update, archives the prior readable version, and refuses
unknown fields rather than discarding them. The live Vault entry was normalized.
Validation passed: image build, shell/migration tests, live infrastructure checks
(including key-only secret shape and Mongo transactions/leases), and diff checks.
`make up` intentionally fails for the still-missing API key. Verified sender must
also be configured in runtime YAML before rebuilding and starting notifications.
No backend source changed and no email was sent.
+22 -7
View File
@@ -40,7 +40,7 @@ kv/
│ ├── bff/jwt # jwt │ ├── bff/jwt # jwt
│ └── notifications/ │ └── notifications/
│ ├── callers/bff # token; read by BFF and notifications │ ├── callers/bff # token; read by BFF and notifications
│ └── email/resend # driver, api_key, sender_address, sender_name │ └── email/resend # api_key only
└── infrastructure/ └── infrastructure/
├── nats/auth # token; read by NATS and notifications ├── nats/auth # token; read by NATS and notifications
└── mongo/bootstrap # local administrator and replica-set bootstrap material └── mongo/bootstrap # local administrator and replica-set bootstrap material
@@ -65,22 +65,37 @@ Open [Vault](http://127.0.0.1:18200) and edit
| Field | Value | | Field | Value |
| --- | --- | | --- | --- |
| `driver` | `resend` |
| `api_key` | Your Resend API key | | `api_key` | Your Resend API key |
| `sender_address` | Your verified sender address |
| `sender_name` | Your display name | The driver and sender settings belong in `infra/local/notifications/runtime.yml`:
```yaml
email:
driver: resend
settings:
api_key_env: EMAIL_API_KEY
sender_address: "your-verified-sender@example.com"
sender_name: "SXCHNGE"
```
Set your verified address in that file and rebuild. It is intentionally empty by
default, so notifications cannot start with an unconfigured sender. Vault Agent
injects only the API key; it does not generate email configuration.
Local administrator login material is in private `.local/dev/admin/init.json`. Local administrator login material is in private `.local/dev/admin/init.json`.
Product containers cannot access it. Alternatively use the hidden terminal prompt: Product containers cannot access it. Alternatively use the hidden terminal prompt:
```sh ```sh
make email make email
make up make build && make up
``` ```
No placeholder key or delivery fallback is installed. Startup validates presence; No placeholder key or delivery fallback is installed. Startup validates presence;
it does not send an email or validate domain ownership with Resend. Updating email it does not send an email or validate domain ownership with Resend. Updating the
settings causes Agent to restart notifications. Existing values survive bootstrap. API key causes Agent to restart notifications. Driver/sender changes require an
image rebuild. Existing API keys survive bootstrap. Migration removes legacy
driver/sender fields from the current Vault secret with a version-checked write;
the preceding version is preserved in Vault history and the private archive.
## Browser HTTPS and internal connections ## Browser HTTPS and internal connections
+6
View File
@@ -19,6 +19,12 @@ messaging:
templates: "/etc/sxchnge/templates.json" templates: "/etc/sxchnge/templates.json"
callers: callers:
product: "SXCHNGE_NOTIFICATIONS_PRODUCT_TOKEN" product: "SXCHNGE_NOTIFICATIONS_PRODUCT_TOKEN"
email:
driver: resend
settings:
api_key_env: EMAIL_API_KEY
sender_address: "" # Set your verified Resend sender here, then run make build.
sender_name: "SXCHNGE"
# Use replicas: 3 and environment: production for a three-node NATS cluster. # Use replicas: 3 and environment: production for a three-node NATS cluster.
# mode: ops validates resources without modifying them; app provisions them. # mode: ops validates resources without modifying them; app provisions them.
queue: queue:
+6 -8
View File
@@ -98,20 +98,19 @@ up() {
random | jq -Rs --arg field "$field" '{($field):rtrimstr("\n")}' | put "$path" random | jq -Rs --arg field "$field" '{($field):rtrimstr("\n")}' | put "$path"
fi fi
done done
printf '{"driver":"resend","api_key":"","sender_address":"notifications@localhost.test","sender_name":"SXCHNGE local"}' | ensure services/notifications/email/resend printf '{"api_key":""}' | ensure services/notifications/email/resend
provision_mongo provision_mongo
provision_agents provision_agents
compose up -d --wait --wait-timeout 120 nats compose up -d --wait --wait-timeout 120 nats
retire_secrets retire_secrets
kv services/notifications/email/resend | email_valid || fail 'Vault kv/services/notifications/email/resend needs a Resend api_key and verified sender_address. Run make email, then make up.' normalize_email_secret
kv services/notifications/email/resend | email_valid || fail 'Vault kv/services/notifications/email/resend needs api_key. Run make email; configure the verified sender in infra/local/notifications/runtime.yml, then make build && make up.'
compose up -d --wait --wait-timeout 180 compose up -d --wait --wait-timeout 180
export_ca export_ca
echo 'Local stack ready: https://localhost:5126; Vault: http://127.0.0.1:18200' echo 'Local stack ready: https://localhost:5126; Vault: http://127.0.0.1:18200'
} }
email() { email() {
existing existing
printf 'Verified Resend sender address: ' >&2
IFS= read -r address
printf 'Resend API key (hidden): ' >&2 printf 'Resend API key (hidden): ' >&2
tty_state=$(stty -g) tty_state=$(stty -g)
trap 'stty "$tty_state"; cleanup' EXIT trap 'stty "$tty_state"; cleanup' EXIT
@@ -120,9 +119,8 @@ email() {
stty "$tty_state" stty "$tty_state"
printf '\n' >&2 printf '\n' >&2
[ -n "$key" ] || fail 'A key is required' [ -n "$key" ] || fail 'A key is required'
case "$address" in *@*) ;; *) fail 'A sender email address is required';; esac
printf '%s' "$key" > "$WORK/email-key" printf '%s' "$key" > "$WORK/email-key"
printf '%s' "$address" > "$WORK/email-address" normalize_email_secret
jq -n --rawfile key "$WORK/email-key" --rawfile address "$WORK/email-address" '{driver:"resend",api_key:$key,sender_address:$address,sender_name:"SXCHNGE"}' | put services/notifications/email/resend jq -n --rawfile key "$WORK/email-key" '{api_key:$key}' | put services/notifications/email/resend
echo 'Email configuration saved in Vault. Run make up.' echo 'API key saved in Vault. Set the verified sender in infra/local/notifications/runtime.yml, then make build && make up.'
} }
+1 -1
View File
@@ -57,7 +57,7 @@ ensure() (
# Never forward mongosh errors: authentication/command errors may include secrets. # Never forward mongosh errors: authentication/command errors may include secrets.
mongo() { compose exec -T mongo mongosh --quiet --file /dev/stdin > "$WORK/mongo-result" 2> "$WORK/mongo-error" || return 1; cat "$WORK/mongo-result"; } mongo() { compose exec -T mongo mongosh --quiet --file /dev/stdin > "$WORK/mongo-result" 2> "$WORK/mongo-error" || return 1; cat "$WORK/mongo-result"; }
admin_auth() { kv infrastructure/mongo/bootstrap | jq -r '"const admin=db.getSiblingDB(\"admin\"); admin.auth(\"local-admin\","+(.password|tojson)+");"'; } admin_auth() { kv infrastructure/mongo/bootstrap | jq -r '"const admin=db.getSiblingDB(\"admin\"); admin.auth(\"local-admin\","+(.password|tojson)+");"'; }
email_valid() { jq -e '(.api_key|type=="string") and (.api_key|test("\\S")) and (.sender_address|type=="string") and (.sender_address|test("\\S")) and .sender_address!="notifications@localhost.test"' >/dev/null; } email_valid() { jq -e '(.api_key|type=="string") and (.api_key|test("\\S"))' >/dev/null; }
export_ca() { export_ca() {
compose exec -T frontend cat /data/caddy/pki/authorities/local/root.crt | write "$STATE/ca.pem" 644 compose exec -T frontend cat /data/caddy/pki/authorities/local/root.crt | write "$STATE/ca.pem" 644
} }
+13
View File
@@ -55,3 +55,16 @@ retire_secrets() (
# and history are retained until the owner explicitly chooses to destroy them. # and history are retained until the owner explicitly chooses to destroy them.
echo 'PASS: Vault secrets migrated and verified; obsolete KV entries retired' echo 'PASS: Vault secrets migrated and verified; obsolete KV entries retired'
) )
# Only credentials belong in Vault; driver/sender settings are runtime YAML.
normalize_email_secret() (
path=services/notifications/email/resend
vault "kv/data/$path" GET yes > "$WORK/email-current"
[ "$(cat "$WORK/email-current")" != null ] || exit 0
jq -e '(.data.data|keys)-["api_key","driver","sender_address","sender_name"]|length==0' "$WORK/email-current" >/dev/null || fail 'Unexpected Resend secret fields; refusing to discard them'
if jq -e '.data.data|keys==["api_key"]' "$WORK/email-current" >/dev/null; then exit 0; fi
archive_secret "$path"
jq '{data:{api_key:(.data.data.api_key // "")},options:{cas:.data.metadata.version}}' "$WORK/email-current" | vault "kv/data/$path" POST >/dev/null
vault "kv/data/$path" | jq -e --slurpfile original "$WORK/email-current" '.data.data|keys==["api_key"] and .api_key==($original[0].data.data.api_key // "")' >/dev/null || fail 'Resend migration verification failed'
echo 'Resend secret now contains only api_key; configure driver/sender in notifications runtime.yml'
)
+1 -4
View File
@@ -2,9 +2,6 @@
set -eu set -eu
umask 077 umask 077
if [ "$SXCHNGE_SERVICE" = notifications ]; then if [ "$SXCHNGE_SERVICE" = notifications ]; then
printf '%s\n' "$EMAIL_CONFIG" > /run/sxchnge/email.yml exec service -config.file=/etc/sxchnge/notifications.yml
unset EMAIL_CONFIG
cat /etc/sxchnge/notifications.yml /run/sxchnge/email.yml > /run/sxchnge/config.yml
exec service -config.file=/run/sxchnge/config.yml
fi fi
exec service -config.file=/etc/sxchnge/bff.yml exec service -config.file=/etc/sxchnge/bff.yml
+15
View File
@@ -59,3 +59,18 @@ printf '{"api_key":"different-fixture"}' > "$WORK/destination"
if (migrate_secrets) >/dev/null 2>&1; then fail 'Migration overwrote conflicting destination'; fi if (migrate_secrets) >/dev/null 2>&1; then fail 'Migration overwrote conflicting destination'; fi
jq -e '.api_key=="different-fixture"' "$WORK/destination" >/dev/null jq -e '.api_key=="different-fixture"' "$WORK/destination" >/dev/null
echo 'PASS: secret migration is repeatable and refuses conflicting destination values' echo 'PASS: secret migration is repeatable and refuses conflicting destination values'
# Removing non-secret fields retains the key and checks the current KV version.
vault() {
if [ "${2:-GET}" = GET ]; then cat "$WORK/email-fixture"; else
jq -e '.options.cas==42 and .data=={api_key:"preserved-fixture-key"}' > /dev/null || return 1
printf '{"data":{"metadata":{"version":43},"data":{"api_key":"preserved-fixture-key"}}}' > "$WORK/email-fixture"
fi
}
printf '{"data":{"metadata":{"version":42},"data":{"api_key":"preserved-fixture-key","driver":"resend","sender_address":"sender@example.test","sender_name":"Fixture"}}}' > "$WORK/email-fixture"
normalize_email_secret
normalize_email_secret
jq -e '.data.data=={api_key:"preserved-fixture-key"}' "$WORK/email-fixture" >/dev/null
printf '{"data":{"metadata":{"version":43},"data":{"api_key":"preserved-fixture-key","unexpected":"must-not-drop"}}}' > "$WORK/email-fixture"
if (normalize_email_secret) >/dev/null 2>&1; then fail 'Unknown secret field was silently dropped'; fi
if grep -q EMAIL_CONFIG infra/local/vault/notifications.hcl infra/local/scripts/start-service.sh; then fail 'Email configuration is still rendered from Vault'; fi
echo 'PASS: email migration preserves API key, removes configuration fields and rejects unknown fields'
+1
View File
@@ -50,6 +50,7 @@ product() {
echo 'PASS: six healthy services, frontend and proxied BFF use verified TLS; no email sent' echo 'PASS: six healthy services, frontend and proxied BFF use verified TLS; no email sent'
} }
layout() { layout() {
kv services/notifications/email/resend | jq -e 'keys==["api_key"]' >/dev/null || fail 'Non-secret email configuration remains in Vault'
for old in bootstrap/mongo bff caller email nats tls frontend-tls notifications-tls; do for old in bootstrap/mongo bff caller email nats tls frontend-tls notifications-tls; do
[ "$(vault "kv/metadata/$old" GET yes)" = null ] || fail "Obsolete KV entry remains: $old" [ "$(vault "kv/metadata/$old" GET yes)" = null ] || fail "Obsolete KV entry remains: $old"
done done
-4
View File
@@ -31,10 +31,6 @@ env_template "EMAIL_API_KEY" {
error_on_missing_key = true error_on_missing_key = true
} }
env_template "EMAIL_CONFIG" {
contents = "{{ with secret \"kv/data/services/notifications/email/resend\" }}email:\n driver: {{ .Data.data.driver | toJSON }}\n settings:\n api_key_env: EMAIL_API_KEY\n sender_address: {{ .Data.data.sender_address | toJSON }}\n sender_name: {{ .Data.data.sender_name | toJSON }}\n{{ end }}"
error_on_missing_key = true
}
exec { exec {
command = ["/usr/local/bin/start-service"] command = ["/usr/local/bin/start-service"]