Keep Mongo operations online across Vault credential rotation

This commit is contained in:
Stephan D committed 2026-09-12 22:19:36 +02:00
1 parent 80d6917aa7
commit 23f7b78c3b
30 files changed
+1290 -202

No files matched your search

+44 -4
View File
@@ -601,8 +601,9 @@ commands, current-stable backend test images and repository container policy.
committed. Root `AGENTS.md` now records the stable-container policy.
- Bootstrap configures Vault KV/PKI/AppRole and `mongodb-database-plugin`, rotates
its Mongo manager credential and issues separate database roles for the BFF
and notifications. Vault Agent renews credentials and restarts child processes
on changes. Existing shared database/messaging paths remain intact.
and notifications. The shared database wrapper replaces credential pools in
process and drains transactions/cursors before revoking retired credentials.
Agent handles non-database secrets. Shared messaging paths remain intact.
- Resend is the default. The user explicitly chose to provision its API key and
verified sender manually after bootstrap. `make up` leaves healthy infrastructure
running and intentionally exits nonzero before product startup while those
@@ -622,8 +623,8 @@ The initial image build exposed missing protobuf headers; installing
Additional live checks passed: a down/up cycle preserved a Mongo marker and Vault
secrets and automatically unsealed Vault; changing/restoring the NATS token
restarted the NATS child; a temporary 10-second/20-second BFF database role proved
replacement credentials restart a healthy BFF (normal TTLs restored afterward).
restarted the NATS child; the initial restart-based database credential lifecycle was superseded by
the uninterrupted pool rotation implementation described below.
Temporary BFF/frontend containers served the web bundle and proxied BFF health
successfully over HTTPS verified with the generated CA; those test containers
were removed. The notifications image rendered its Agent configuration and exited
@@ -636,3 +637,42 @@ NATS remain running. No optional security scan or merge was performed. See the
[local development guide](local-development.md) for exact provisioning fields,
certificate trust, restart behavior, local bootstrap-material handling and
verification commands.
### Uninterrupted Mongo credential rotation
BFF and notifications now acquire database operation scopes from the shared
wrapper. A replacement is issued at half the effective token/lease lifetime,
validated before publication, and independently revoked after draining. Whole
transactions and cursors retain their original pool. Failed refreshes retry while
the current credential remains valid. Expiry rejects new operations until recovery;
there is no scheduled service restart. Non-database Agent template changes still
restart their child process. This is planned-rotation continuity, not infrastructure
high availability or a guarantee beyond Vault credential expiry.
The repository and store constructor contract now uses `source.Source` rather
than a concrete Mongo database. Static test integrations use `source.Fixed`.
Application code continues through shared repositories and transaction factories.
The new `make test-dev-rotation` command proves real Vault plugin replacements
under BFF transaction traffic and checks process identity. It restores shortened
role TTLs and removes temporary product containers afterward.
Validation of the revision: `make build`, `make test`, `make lint-backend`,
`make test-dev-infra` (six setup tests and live checks), and
`make test-dev-rotation` passed. The live rotation check observed three replacements
with an unchanged BFF PID and 1,155 expected invalid-token HTTP responses, zero
failures. Product integration-tag packages also compiled. The race-enabled
rotation suite covers active transactions/cursors, rejected credentials, transient
and extended outages, automatic recovery, rollback and shutdown; issuer tests
cover effective lifetime, credential escaping and failed-issuance cleanup.
An additional `go test -race -tags integration ./database/internal/repository/...
./database/vault/...` run in `backend/shared` found two existing test issues: the
partial-index test expects nested BSON maps instead of ordered documents, and the
patch timestamp assertion can require a strict increase within the same Mongo
millisecond. The BSON mismatch reproduces against `origin/main`; five isolated baseline
repetitions of the timestamp test pass, consistent with its timing-dependent
assertion. Neither assertion nor the timestamp update logic changes in this task;
the supplementary failures are recorded and are not suppressed. The required
normal test and lint commands pass. No optional security scan or email delivery
was performed. Resend remains intentionally unconfigured; infrastructure stays up
and product startup awaits manual `kv/email` provisioning.