Recover local Sendico session after BFF network restarts (#57)

This commit is contained in:
Stephan D committed 2026-10-19 12:34:37 +02:00
1 parent 021bd04324
commit 13cc9c302e
9 files changed
+158 -8

No files matched your search

+36
View File
@@ -105,6 +105,27 @@ docker compose --project-directory . -f infra/local/compose.yml \
-f infra/local/sendico/compose.yml --env-file .local/dev/images.env restart sendico
```
A BFF container restart can leave its session worker in the old shared network
namespace. The worker checks Vault DNS every ten seconds, with a five-second
lookup limit. Three consecutive failures log a credential-free recovery reason
and terminate the session process; Docker's restart policy then attaches it to
the BFF's current namespace. A successful lookup resets the counter, so transient
DNS failures do not immediately restart the worker.
Container recreation changes the BFF container ID as well as its namespace.
Bootstrap and credential-rotation tooling recreate BFF and its configured session
worker together. Compose also marks the session dependency for restart when BFF
is explicitly updated. For a manual forced recreation, select both services:
```sh
docker compose --project-directory . -f infra/local/compose.yml \
-f infra/local/sendico/compose.yml --env-file .local/dev/images.env \
up -d --no-deps --force-recreate --wait bff sendico
```
The parent process cleans up its bridge, DNS monitor and renewal delay on exit.
This recovery retains the configured provider URL, account and replay namespace.
`make test-dev` checks seven healthy services, verified browser TLS, the Vault
layout and least-privilege identities, Mongo transactions/lease rotation, and
authenticated Sendico currency and ledger reads through the same loopback
@@ -130,6 +151,21 @@ and the established CA. A subsequent verified TLS check must succeed.
## Acceptance and operator UX review
Recovery follow-up for [task #57](https://git.sendico.io/tech/sxchnge/issues/57) /
[PR #59](https://git.sendico.io/tech/sxchnge/pulls/59), validated on 2026-10-19:
`make test-dev-tooling`, `make test-dev-rotation`, Compose configuration validation,
`sh infra/local/tools.sh verify` and `git diff --check` pass. A BFF-only restart
triggered automatic session restart and restored matching network namespaces;
paired recreation also finished healthy. Rotation observed three real Vault
pool replacements, an unchanged BFF process and 534 transaction-backed requests
with zero failures. The final check verifies all seven SXCHNGE services, browser
HTTPS, Vault/database boundaries and authenticated Sendico currency/ledger reads.
An earlier check failed because Sendico's ledger announcement connection was
closed; restoring Discovery and its service announcers in dependency order
recovered that existing stand. No Sendico source or financial data was changed,
and no email or payment was sent. This follow-up changes shell/Compose tooling;
it does not rerun the earlier full Go/Flutter suite for unchanged product source.
Validated on 2026-10-16: `make build`, `make up`, `make test-dev-tooling`,
`make test-dev-infra`, `make test-dev`, and `git diff --check` passed. Both product
sites, Sendico Backoffice and both API health endpoints returned HTTP 200;
+27
View File
@@ -162,6 +162,33 @@ as passing. The exact response to the reported Safari attempt was not captured,
and the frontend's generic unavailable text also covers some HTTP 400 input
errors. A fresh user retry is needed to confirm the visible signup outcome.
On October 19 the failing API was directly observed returning HTTP 502. The BFF
had restarted while the session worker retained its old network namespace;
`/proc/self/ns/net` differed between them, and the worker could not resolve
Vault. Sendico's public API itself returned HTTP 200. Restarting only the worker
reattached its namespace and restored SXCHNGE API HTTP 200. This is a confirmed
cause of that 502, separate from the previous process leak and host-load finding.
The session now monitors bounded DNS failures and exits after three consecutive
failures so Docker can reattach it automatically. Bootstrap and rotation pair
BFF/session recreation, with Compose dependency restart enabled. Offline shell
fixtures cover transient-failure reset, sustained-failure recovery, safe parent
targeting and conditional service pairing. A live BFF-only restart automatically
restarted the session and restored matching namespaces, fresh session readiness
and API HTTP 200. A second live test recreated BFF and its session together and
confirmed healthy services with matching namespaces. No provider bindings,
customer records, credentials or Sendico source files were changed.
Final October 19 validation passes: `make test-dev-tooling`, Compose config
validation, `make test-dev-rotation` (three real rotations, unchanged BFF process,
534 transaction-backed requests and zero failures), and
`sh infra/local/tools.sh verify`. The last confirms all seven healthy services,
verified frontend/BFF HTTPS, Vault/database checks and authenticated Sendico
currency/ledger reads. An initial ledger-read failure was resolved by restoring
Sendico Discovery and its announcers in their documented dependency order after
closed NATS connections. No email was sent during validation. The recovery fix
and its validation are included in [PR #59](https://git.sendico.io/tech/sxchnge/pulls/59).
With `api.sendico` configured, the product BFF creates the local pending customer
and activation token atomically, then registers an independent individual through