Keep Mongo operations online across Vault credential rotation

docs/implementation.md

@@ -601,8 +601,9 @@ commands, current-stable backend test images and repository container policy.
   committed. Root `AGENTS.md` now records the stable-container policy.
 - Bootstrap configures Vault KV/PKI/AppRole and `mongodb-database-plugin`, rotates
   its Mongo manager credential and issues separate database roles for the BFF
-  and notifications. Vault Agent renews credentials and restarts child processes
-  on changes. Existing shared database/messaging paths remain intact.
+  and notifications. The shared database wrapper replaces credential pools in
+  process and drains transactions/cursors before revoking retired credentials.
+  Agent handles non-database secrets. Shared messaging paths remain intact.
 - Resend is the default. The user explicitly chose to provision its API key and
   verified sender manually after bootstrap. `make up` leaves healthy infrastructure
   running and intentionally exits nonzero before product startup while those
@@ -622,8 +623,8 @@ The initial image build exposed missing protobuf headers; installing
 
 Additional live checks passed: a down/up cycle preserved a Mongo marker and Vault
 secrets and automatically unsealed Vault; changing/restoring the NATS token
-restarted the NATS child; a temporary 10-second/20-second BFF database role proved
-replacement credentials restart a healthy BFF (normal TTLs restored afterward).
+restarted the NATS child; the initial restart-based database credential lifecycle was superseded by
+the uninterrupted pool rotation implementation described below.
 Temporary BFF/frontend containers served the web bundle and proxied BFF health
 successfully over HTTPS verified with the generated CA; those test containers
 were removed. The notifications image rendered its Agent configuration and exited
@@ -636,3 +637,42 @@ NATS remain running. No optional security scan or merge was performed. See the
 [local development guide](local-development.md) for exact provisioning fields,
 certificate trust, restart behavior, local bootstrap-material handling and
 verification commands.
+
+### Uninterrupted Mongo credential rotation
+
+BFF and notifications now acquire database operation scopes from the shared
+wrapper. A replacement is issued at half the effective token/lease lifetime,
+validated before publication, and independently revoked after draining. Whole
+transactions and cursors retain their original pool. Failed refreshes retry while
+the current credential remains valid. Expiry rejects new operations until recovery;
+there is no scheduled service restart. Non-database Agent template changes still
+restart their child process. This is planned-rotation continuity, not infrastructure
+high availability or a guarantee beyond Vault credential expiry.
+
+The repository and store constructor contract now uses `source.Source` rather
+than a concrete Mongo database. Static test integrations use `source.Fixed`.
+Application code continues through shared repositories and transaction factories.
+The new `make test-dev-rotation` command proves real Vault plugin replacements
+under BFF transaction traffic and checks process identity. It restores shortened
+role TTLs and removes temporary product containers afterward.
+
+Validation of the revision: `make build`, `make test`, `make lint-backend`,
+`make test-dev-infra` (six setup tests and live checks), and
+`make test-dev-rotation` passed. The live rotation check observed three replacements
+with an unchanged BFF PID and 1,155 expected invalid-token HTTP responses, zero
+failures. Product integration-tag packages also compiled. The race-enabled
+rotation suite covers active transactions/cursors, rejected credentials, transient
+and extended outages, automatic recovery, rollback and shutdown; issuer tests
+cover effective lifetime, credential escaping and failed-issuance cleanup.
+
+An additional `go test -race -tags integration ./database/internal/repository/...
+./database/vault/...` run in `backend/shared` found two existing test issues: the
+partial-index test expects nested BSON maps instead of ordered documents, and the
+patch timestamp assertion can require a strict increase within the same Mongo
+millisecond. The BSON mismatch reproduces against `origin/main`; five isolated baseline
+repetitions of the timestamp test pass, consistent with its timing-dependent
+assertion. Neither assertion nor the timestamp update logic changes in this task;
+the supplementary failures are recorded and are not suppressed. The required
+normal test and lint commands pass. No optional security scan or email delivery
+was performed. Resend remains intentionally unconfigured; infrastructure stays up
+and product startup awaits manual `kv/email` provisioning.

docs/local-development.md

@@ -108,16 +108,39 @@ and are retained in `kv/bootstrap/mongo` for local provisioning.
 The BFF role has `readWrite` only on `sxchnge`; notifications has `readWrite` only
 on `sxchnge_notifications`. Mongo URIs come from dynamic Vault credentials, not
 static application users. Leases default to one hour with a 24-hour maximum.
-Vault Agent renews leases, obtains replacement credentials, and restarts its child
-process when credentials change. Apps retain their existing shared database and
-messaging connection wrappers. No new direct application database path is added.
+The shared database wrapper obtains credentials directly from Vault using the
+service's AppRole. At half the effective lifetime (the shorter of the database
+lease and owning token TTL), it authenticates independently and prepares a new
+Mongo pool. Only a successful primary ping permits publication. New operations
+use that pool; active operations, cursors and complete transactions, including
+nested stores and commit retries, retain the old pool until they finish. The old
+pool then closes and its dedicated token is revoked. Mongo credential replacement
+never restarts BFF or notifications. Each generation has a separate token so
+cleanup cannot revoke the replacement or Vault Agent's identity.
+
+Refresh failures keep the current valid pool and retry with bounded backoff.
+Operations are bounded by credential expiry. If Vault remains unavailable beyond
+that deadline, new database operations fail closed; refresh continues and access
+recovers automatically without restarting the service. Administrative revocation,
+operations exceeding their lease and infrastructure failures cannot be made
+uninterrupted by pool rotation. Shutdown rejects new scopes and drains existing
+work. Failed cleanup relies on Vault expiration as a backstop.
+
+Repositories and transaction factories share `source.Source` rather than retaining
+concrete Mongo collections. Low-level consumers must acquire a scope, use its
+returned context and release it after all work completes; never retain its client
+or context beyond release. `source.Fixed` adapts externally owned test/static
+connections. Database credentials are no longer Agent environment templates;
+Agent continues to fetch non-database secrets and may restart children when those
+secrets change.
 
 Each agent authenticates using its own AppRole with a renewable periodic token.
 Local SecretIDs are reusable for container restarts; each container mounts only
 its own identity directory under `.local/dev/agents/`. Revoking an AppRole SecretID
 requires removing its matching local `secret-id` file and rerunning `make up` to
 issue a replacement. Revoking an active token/lease terminates its credentials;
-Agent's retries and the container restart policy recover when Vault is reachable.
+The database wrapper retries with a fresh identity; Agent independently retries
+its non-database secrets when Vault is reachable.
 
 The notification gRPC connection uses a Vault-issued certificate for
 `notifications` and the BFF validates the Vault CA and hostname. Frontend HTTPS
@@ -147,6 +170,7 @@ make logs
 make down             # preserves named volumes and local bootstrap material
 make up               # idempotent provisioning and readiness checks
 make test-dev-infra   # works before Resend provisioning
+make test-dev-rotation # real Vault rotations under BFF HTTP traffic
 make test-dev         # requires all six services running
 ```
 
@@ -154,6 +178,15 @@ The infrastructure check exercises AppRole access restrictions, real Vault
 MongoDB credential issuance, per-database access, lease renewal and user deletion
 on revocation. The complete check also verifies service health and frontend/API
 HTTPS with the generated CA. These checks never send a real email.
+`test-dev-rotation` temporarily sets a 12-second BFF lease, deploys the built BFF
+and frontend, and requires three pool replacements with unchanged process ID and
+zero unexpected HTTP responses. Its invalid-token requests exercise Mongo-backed
+transactions without sending email. The check restores the role TTL and removes
+services it started; it recreates BFF/frontend at the beginning, so run it as a
+development validation command, not against active work you need to preserve.
+Race-enabled backend tests additionally hold a transaction and cursor across a
+replacement and exercise failed candidate validation, outage recovery, rollback,
+expiry rejection and graceful shutdown.
 
 Backend regression tests use fresh `mongo:latest` and `nats:latest` official
 stable images, pulled before `make test-backend`. Normal validation remains