sxchnge/docs/local-sendico.md

# Local Sendico integration

Task [#52](https://git.sendico.io/tech/sxchnge/issues/52), branch `local-stand-52`,
based on `payments-50` / [PR #51](https://git.sendico.io/tech/sxchnge/pulls/51).
The local Sendico checkout uses `payments-2896` /
[PR #2897](https://git.sendico.io/tech/sendico/pulls/2897). Merge the prerequisites
before this infrastructure PR. No PR is merged by local startup.

## Running the configured stands

Start Sendico first from its checkout using its documented Make workflow. Its
Product API must respond at `http://localhost:8080/api/v1/health`. Then, here:

```sh
make build
make up
make test-dev
```

Open [SXCHNGE](https://localhost:5126),
[Sendico Product](http://localhost:3000), and
[Sendico Backoffice](http://localhost:4000).
SXCHNGE's Caddy CA is exported to `.local/dev/ca.pem`; trust that public CA in
the browser/OS if necessary. Bootstrap does not modify the system trust store.

The optional integration is selected by ignored `.local/dev/sendico/settings.env`.
Make and the containerized bootstrap both include
`infra/local/sendico/compose.yml` whenever that file exists. Standalone SXCHNGE
still uses six services. The connected stack adds one `sendico` session worker.
`make ps`, `make logs`, and `make down` include it automatically.

## Identity and configuration

The configured local stand has a dedicated `SXCHNGE Local` organization, active
integration account and account-bound confidential API client. The account is
an organization member with organization read/update, ledger read, payment
create/read, and recipient create/read/update permissions. Customer subjects
remain independent from that account. No pooled balances, customer funds or
payout destinations are seeded by this setup. Existing stand data is retained.
This identity was created through a one-time local administrative bootstrap;
runtime traffic and session creation use only the public HTTP API.

For another stand, an operator must first provision the organization, account,
client and grants in Sendico. Resolve actual IDs from that stand; do not copy
another environment's references. Initialize SXCHNGE's Vault using `make up`
before configuring the integration. Supply this JSON privately on stdin:

```json
{
  "organization_ref": "<organization object ID>",
  "source_account_ref": "<integration account object ID>",
  "client_id": "<confidential client ID>",
  "device_id": "sxchnge-local-bff",
  "login": "<integration login>",
  "password": "<integration password>",
  "client_secret": "<confidential client secret>"
}
```

```sh
# Save operator input outside tracked files, with mode 0600. Never echo it.
make sendico < /private/path/sendico-input.json
make up
make test-dev
```

`make sendico` validates references and identifiers, stores only login/password/
client_secret at Vault `kv/services/bff/sendico`, and writes the non-secret client,
device, organization and source references into ignored local settings. The BFF
runtime is generated from `infra/local/bff/runtime.yml` with the existing
`api.sendico` configuration. Re-run configuration after changing that base YAML.
Remove the temporary input file once credentials have been stored in Vault.
No Sendico configuration parameters or application transport rules change.

## Connection and renewal

The session worker shares BFF's network namespace and binds a private loopback
TCP bridge at `127.0.0.1:18081`, forwarding to
`host.docker.internal:8080`. This uses the local stand's existing public HTTP API
port; it requires a local Docker engine with `host.docker.internal` support, as
provided by Docker Desktop. The bridge publishes no new host port. The adapter's
existing HTTPS requirement and loopback-only HTTP exception remain intact.

The worker authenticates to Vault with its own AppRole, reads exactly
`services/bff/sendico`, and revokes its five-minute Vault token after reading.
It cannot read Mongo credentials, JWT signing keys, NATS auth or administrator
material. Its initial login uses Sendico's public `/accounts/login/api` grant,
validates the access token's source account, full-session status and expiry, and
atomically publishes the token in a volume mounted read-only by BFF. Credentials
and response bodies stay in private tmpfs files and never appear in logs or
command arguments. Flutter receives no platform credentials.

Successful login repeats every 20 minutes before the one-hour access token
expires. A failure retains the prior token, logs one generic actionable message,
and retries after five minutes. The worker becomes unhealthy after 25 minutes
without renewal or when the upstream health endpoint fails; the adapter
separately rejects expired or mismatched tokens.
BFF waits for a recent token at startup. Following credential changes, restart
the worker with `make up` followed by the Compose restart command below, or wait
for its next renewal. Never change the integration account on an existing
financial environment without reviewing resource and idempotency bindings.

```sh
docker compose --project-directory . -f infra/local/compose.yml \
  -f infra/local/sendico/compose.yml --env-file .local/dev/images.env restart sendico
```

A BFF container restart can leave its session worker in the old shared network
namespace. The worker checks Vault DNS every ten seconds, with a five-second
lookup limit. Three consecutive failures log a credential-free recovery reason
and terminate the session process; Docker's restart policy then attaches it to
the BFF's current namespace. A successful lookup resets the counter, so transient
DNS failures do not immediately restart the worker.

Container recreation changes the BFF container ID as well as its namespace.
Bootstrap and credential-rotation tooling recreate BFF and its configured session
worker together. Compose also marks the session dependency for restart when BFF
is explicitly updated. For a manual forced recreation, select both services:

```sh
docker compose --project-directory . -f infra/local/compose.yml \
  -f infra/local/sendico/compose.yml --env-file .local/dev/images.env \
  up -d --no-deps --force-recreate --wait bff sendico
```

The parent process cleans up its bridge, DNS monitor and renewal delay on exit.
This recovery retains the configured provider URL, account and replay namespace.

`make test-dev` checks seven healthy services, verified browser TLS, the Vault
layout and least-privilege identities, Mongo transactions/lease rotation, and
authenticated Sendico currency and ledger reads through the same loopback
connection and token used by BFF. It sends no email and executes no payment.
Signup and wallet access automatically ensure subject-bound RUB, USDT and USDC
ledgers as described in [account setup](payments.md#automatic-currency-accounts-72).
Deploy the provider prerequisite and grant ledger-create permission first.
Saved funding and payout methods remain separate from currency-account setup.

## Recovery

The existing stand had application processes stopped inside still-running Air
containers after a NATS disconnect and Mongo replica-state interruption. Restore
discovery first, then Treasury and the other service announcers. Wait until each
stage is registered before restarting dependent services: billing fees and
Ledger, then Limits/gateways/quotation/receiving routes, then payment orchestrator,
compliance assessment and Product BFF. The Sendico Makefile provides its ordered
backend startup workflow. Inspect current logs and health instead of assuming a
running development container has a live application process. Do not delete
volumes or reset financial data to recover service registration.

If an old Caddy leaf certificate expires while the stack is stopped, restart
frontend after startup so Caddy issues a fresh certificate. Preserve `caddy-data`
and the established CA. A subsequent verified TLS check must succeed.

## Acceptance and operator UX review

Recovery follow-up for [task #57](https://git.sendico.io/tech/sxchnge/issues/57) /
[PR #59](https://git.sendico.io/tech/sxchnge/pulls/59), validated on 2026-10-19:
`make test-dev-tooling`, `make test-dev-rotation`, Compose configuration validation,
`sh infra/local/tools.sh verify` and `git diff --check` pass. A BFF-only restart
triggered automatic session restart and restored matching network namespaces;
paired recreation also finished healthy. Rotation observed three real Vault
pool replacements, an unchanged BFF process and 534 transaction-backed requests
with zero failures. The final check verifies all seven SXCHNGE services, browser
HTTPS, Vault/database boundaries and authenticated Sendico currency/ledger reads.
An earlier check failed because Sendico's ledger announcement connection was
closed; restoring Discovery and its service announcers in dependency order
recovered that existing stand. No Sendico source or financial data was changed,
and no email or payment was sent. This follow-up changes shell/Compose tooling;
it does not rerun the earlier full Go/Flutter suite for unchanged product source.

Validated on 2026-10-16: `make build`, `make up`, `make test-dev-tooling`,
`make test-dev-infra`, `make test-dev`, and `git diff --check` passed. Both product
sites, Sendico Backoffice and both API health endpoints returned HTTP 200;
SXCHNGE's browser endpoint was checked with its exported CA. All Sendico services
with health checks were healthy. Forced worker restart issued a different valid
same-account token and preserved successful authenticated currency/ledger reads.
The existing notification sender edit remains outside this task's commit.

The setup journey starts with the operator's Make invocation and completes when
Compose confirms service readiness; `make test-dev` additionally confirms the
authenticated integration. Existing command exit codes and Compose progress /
health states distinguish configuration, dependency, authentication and readiness
failures. A renewed-session log records credential recovery, while stale renewal
or upstream outage changes container health. Each new operator invocation is a
new attempt; worker retries remain within that process's lifetime. Command wall
time measures setup duration. Interrupted commands are incomplete attempts, not
successful launches. These are local operator signals, with no retained analytics
or cross-invocation conversion/abandonment reporting added.

The UX review covered missing bootstrap material, malformed operator input,
authentication failure, retained retry context, stale TLS recovery, successful
startup and confirmed API access. Commands reuse existing Make/Compose patterns,
errors give a recovery action, and success follows readiness. No secret values
appear in terminal feedback. Product layouts, focus and screen-reader behavior
are unchanged; browser interaction and funded payment settlement are not claimed
by HTTP readiness checks. No email or payment was sent during validation.