Keep Mongo operations online across Vault credential rotation

docs/local-development.md

@@ -108,16 +108,39 @@ and are retained in `kv/bootstrap/mongo` for local provisioning.
 The BFF role has `readWrite` only on `sxchnge`; notifications has `readWrite` only
 on `sxchnge_notifications`. Mongo URIs come from dynamic Vault credentials, not
 static application users. Leases default to one hour with a 24-hour maximum.
-Vault Agent renews leases, obtains replacement credentials, and restarts its child
-process when credentials change. Apps retain their existing shared database and
-messaging connection wrappers. No new direct application database path is added.
+The shared database wrapper obtains credentials directly from Vault using the
+service's AppRole. At half the effective lifetime (the shorter of the database
+lease and owning token TTL), it authenticates independently and prepares a new
+Mongo pool. Only a successful primary ping permits publication. New operations
+use that pool; active operations, cursors and complete transactions, including
+nested stores and commit retries, retain the old pool until they finish. The old
+pool then closes and its dedicated token is revoked. Mongo credential replacement
+never restarts BFF or notifications. Each generation has a separate token so
+cleanup cannot revoke the replacement or Vault Agent's identity.
+
+Refresh failures keep the current valid pool and retry with bounded backoff.
+Operations are bounded by credential expiry. If Vault remains unavailable beyond
+that deadline, new database operations fail closed; refresh continues and access
+recovers automatically without restarting the service. Administrative revocation,
+operations exceeding their lease and infrastructure failures cannot be made
+uninterrupted by pool rotation. Shutdown rejects new scopes and drains existing
+work. Failed cleanup relies on Vault expiration as a backstop.
+
+Repositories and transaction factories share `source.Source` rather than retaining
+concrete Mongo collections. Low-level consumers must acquire a scope, use its
+returned context and release it after all work completes; never retain its client
+or context beyond release. `source.Fixed` adapts externally owned test/static
+connections. Database credentials are no longer Agent environment templates;
+Agent continues to fetch non-database secrets and may restart children when those
+secrets change.
 
 Each agent authenticates using its own AppRole with a renewable periodic token.
 Local SecretIDs are reusable for container restarts; each container mounts only
 its own identity directory under `.local/dev/agents/`. Revoking an AppRole SecretID
 requires removing its matching local `secret-id` file and rerunning `make up` to
 issue a replacement. Revoking an active token/lease terminates its credentials;
-Agent's retries and the container restart policy recover when Vault is reachable.
+The database wrapper retries with a fresh identity; Agent independently retries
+its non-database secrets when Vault is reachable.
 
 The notification gRPC connection uses a Vault-issued certificate for
 `notifications` and the BFF validates the Vault CA and hostname. Frontend HTTPS
@@ -147,6 +170,7 @@ make logs
 make down             # preserves named volumes and local bootstrap material
 make up               # idempotent provisioning and readiness checks
 make test-dev-infra   # works before Resend provisioning
+make test-dev-rotation # real Vault rotations under BFF HTTP traffic
 make test-dev         # requires all six services running
 ```
 
@@ -154,6 +178,15 @@ The infrastructure check exercises AppRole access restrictions, real Vault
 MongoDB credential issuance, per-database access, lease renewal and user deletion
 on revocation. The complete check also verifies service health and frontend/API
 HTTPS with the generated CA. These checks never send a real email.
+`test-dev-rotation` temporarily sets a 12-second BFF lease, deploys the built BFF
+and frontend, and requires three pool replacements with unchanged process ID and
+zero unexpected HTTP responses. Its invalid-token requests exercise Mongo-backed
+transactions without sending email. The check restores the role TTL and removes
+services it started; it recreates BFF/frontend at the beginning, so run it as a
+development validation command, not against active work you need to preserve.
+Race-enabled backend tests additionally hold a transaction and cursor across a
+replacement and exercise failed candidate validation, outage recovery, rollback,
+expiry rejection and graceful shutdown.
 
 Backend regression tests use fresh `mongo:latest` and `nats:latest` official
 stable images, pulled before `make test-backend`. Normal validation remains