Recover local Sendico session after BFF network restarts (#57)
This commit is contained in:
1 parent
021bd04324
commit
13cc9c302e
9 files changed
+158
-8
No files matched your search
@@ -105,6 +105,27 @@ docker compose --project-directory . -f infra/local/compose.yml \
|
||||
-f infra/local/sendico/compose.yml --env-file .local/dev/images.env restart sendico
|
||||
```
|
||||
|
||||
A BFF container restart can leave its session worker in the old shared network
|
||||
namespace. The worker checks Vault DNS every ten seconds, with a five-second
|
||||
lookup limit. Three consecutive failures log a credential-free recovery reason
|
||||
and terminate the session process; Docker's restart policy then attaches it to
|
||||
the BFF's current namespace. A successful lookup resets the counter, so transient
|
||||
DNS failures do not immediately restart the worker.
|
||||
|
||||
Container recreation changes the BFF container ID as well as its namespace.
|
||||
Bootstrap and credential-rotation tooling recreate BFF and its configured session
|
||||
worker together. Compose also marks the session dependency for restart when BFF
|
||||
is explicitly updated. For a manual forced recreation, select both services:
|
||||
|
||||
```sh
|
||||
docker compose --project-directory . -f infra/local/compose.yml \
|
||||
-f infra/local/sendico/compose.yml --env-file .local/dev/images.env \
|
||||
up -d --no-deps --force-recreate --wait bff sendico
|
||||
```
|
||||
|
||||
The parent process cleans up its bridge, DNS monitor and renewal delay on exit.
|
||||
This recovery retains the configured provider URL, account and replay namespace.
|
||||
|
||||
`make test-dev` checks seven healthy services, verified browser TLS, the Vault
|
||||
layout and least-privilege identities, Mongo transactions/lease rotation, and
|
||||
authenticated Sendico currency and ledger reads through the same loopback
|
||||
@@ -130,6 +151,21 @@ and the established CA. A subsequent verified TLS check must succeed.
|
||||
|
||||
## Acceptance and operator UX review
|
||||
|
||||
Recovery follow-up for [task #57](https://git.sendico.io/tech/sxchnge/issues/57) /
|
||||
[PR #59](https://git.sendico.io/tech/sxchnge/pulls/59), validated on 2026-10-19:
|
||||
`make test-dev-tooling`, `make test-dev-rotation`, Compose configuration validation,
|
||||
`sh infra/local/tools.sh verify` and `git diff --check` pass. A BFF-only restart
|
||||
triggered automatic session restart and restored matching network namespaces;
|
||||
paired recreation also finished healthy. Rotation observed three real Vault
|
||||
pool replacements, an unchanged BFF process and 534 transaction-backed requests
|
||||
with zero failures. The final check verifies all seven SXCHNGE services, browser
|
||||
HTTPS, Vault/database boundaries and authenticated Sendico currency/ledger reads.
|
||||
An earlier check failed because Sendico's ledger announcement connection was
|
||||
closed; restoring Discovery and its service announcers in dependency order
|
||||
recovered that existing stand. No Sendico source or financial data was changed,
|
||||
and no email or payment was sent. This follow-up changes shell/Compose tooling;
|
||||
it does not rerun the earlier full Go/Flutter suite for unchanged product source.
|
||||
|
||||
Validated on 2026-10-16: `make build`, `make up`, `make test-dev-tooling`,
|
||||
`make test-dev-infra`, `make test-dev`, and `git diff --check` passed. Both product
|
||||
sites, Sendico Backoffice and both API health endpoints returned HTTP 200;
|
||||
|
||||
@@ -162,6 +162,33 @@ as passing. The exact response to the reported Safari attempt was not captured,
|
||||
and the frontend's generic unavailable text also covers some HTTP 400 input
|
||||
errors. A fresh user retry is needed to confirm the visible signup outcome.
|
||||
|
||||
On October 19 the failing API was directly observed returning HTTP 502. The BFF
|
||||
had restarted while the session worker retained its old network namespace;
|
||||
`/proc/self/ns/net` differed between them, and the worker could not resolve
|
||||
Vault. Sendico's public API itself returned HTTP 200. Restarting only the worker
|
||||
reattached its namespace and restored SXCHNGE API HTTP 200. This is a confirmed
|
||||
cause of that 502, separate from the previous process leak and host-load finding.
|
||||
|
||||
The session now monitors bounded DNS failures and exits after three consecutive
|
||||
failures so Docker can reattach it automatically. Bootstrap and rotation pair
|
||||
BFF/session recreation, with Compose dependency restart enabled. Offline shell
|
||||
fixtures cover transient-failure reset, sustained-failure recovery, safe parent
|
||||
targeting and conditional service pairing. A live BFF-only restart automatically
|
||||
restarted the session and restored matching namespaces, fresh session readiness
|
||||
and API HTTP 200. A second live test recreated BFF and its session together and
|
||||
confirmed healthy services with matching namespaces. No provider bindings,
|
||||
customer records, credentials or Sendico source files were changed.
|
||||
|
||||
Final October 19 validation passes: `make test-dev-tooling`, Compose config
|
||||
validation, `make test-dev-rotation` (three real rotations, unchanged BFF process,
|
||||
534 transaction-backed requests and zero failures), and
|
||||
`sh infra/local/tools.sh verify`. The last confirms all seven healthy services,
|
||||
verified frontend/BFF HTTPS, Vault/database checks and authenticated Sendico
|
||||
currency/ledger reads. An initial ledger-read failure was resolved by restoring
|
||||
Sendico Discovery and its announcers in their documented dependency order after
|
||||
closed NATS connections. No email was sent during validation. The recovery fix
|
||||
and its validation are included in [PR #59](https://git.sendico.io/tech/sxchnge/pulls/59).
|
||||
|
||||
|
||||
With `api.sendico` configured, the product BFF creates the local pending customer
|
||||
and activation token atomically, then registers an independent individual through
|
||||
|
||||
Reference in new issue
Block a user