Merge pull request 'Restore signup, email delivery and local recovery' (#59) from identity-facts-57 into main

Reviewed-on: #59

docs/local-development.md

@@ -27,10 +27,17 @@ project directory. Dockerfiles are in `infra/images/`. Runtime and startup files
 are grouped by service under `infra/local/`. The stack consists of Vault, Mongo,
 NATS, notifications, BFF and Caddy serving the frontend.
 
+The frontend container uses Compose `init: true` to reap orphaned `ssl_client`
+children created by BusyBox's HTTPS `wget` health probe. Keep this enabled:
+Caddy alone does not reap those children when it runs as PID 1, and repeated
+probes can accumulate thousands of zombie processes. Recreate the frontend
+container when applying this setting; restarting an old container does not
+change its init configuration.
+
 On first startup, bootstrap initializes/unseals persistent Vault, provisions
 AppRoles and secrets, initializes Mongo replica set `rs0`, configures Vault's Mongo
 plugin, and starts authenticated NATS JetStream. Product startup fails explicitly
-until Resend credentials exist. Infrastructure remains available for provisioning.
+until the SMTP mailbox password exists. Infrastructure remains available for provisioning.
 
 After all services pass their startup health checks and the public Caddy CA is
 exported, `make up` prints a Sendico-style completion message with the frontend,
@@ -56,7 +63,7 @@ kv/
 ├── services/
 │   ├── bff/jwt                         # jwt
 │   └── notifications/
-│       └── email/resend                # api_key only
+│       └── email/smtp                  # password only
 └── infrastructure/
     ├── nats/auth                       # token; read by NATS and notifications
     └── mongo/bootstrap                 # local administrator and replica-set bootstrap material
@@ -74,29 +81,35 @@ There are no frontend certificates, notification certificates or generic `tls`
 entries in KV. Browser certificates belong to Caddy; internal gRPC is plaintext
 on the trusted private Docker network without caller authentication.
 
-## Resend
+## Local SMTP mail
 
 Open [Vault](http://127.0.0.1:18200) and edit
-`kv/services/notifications/email/resend`:
+`kv/services/notifications/email/smtp`:
 
 | Field | Value |
 | --- | --- |
-| `api_key` | Your Resend API key |
+| `password` | Password for the configured SMTP mailbox |
 
 The driver and sender settings belong in `infra/local/notifications/runtime.yml`:
 
 ```yaml
 email:
-  driver: resend
+  driver: smtp
   settings:
-    api_key_env: EMAIL_API_KEY
-    sender_address: "your-verified-sender@example.com"
+    host: mail.sendico.io
+    port: 465
+    username: no-reply@sendico.io
+    password_env: SMTP_PASSWORD
+    network_timeout: 10
+    sender_address: no-reply@sendico.io
     sender_name: "SXCHNGE"
 ```
 
-Set your verified address in that file and rebuild. It is intentionally empty by
-default, so notifications cannot start with an unconfigured sender. Vault Agent
-injects only the API key; it does not generate email configuration.
+The local stack uses Sendico's existing `no-reply@sendico.io` mailbox with
+authenticated implicit TLS on port 465 and certificate verification enabled.
+Vault Agent injects only the password; host, username and sender are non-secret
+runtime settings. Credentials are provisioned into SXCHNGE's own Vault; the
+notification service does not read Sendico configuration or secrets at runtime.
 
 Local administrator login material is in private `.local/dev/admin/init.json`.
 Product containers cannot access it. Alternatively use the hidden terminal prompt:
@@ -106,12 +119,15 @@ make email
 make build && make up
 ```
 
-No placeholder key or delivery fallback is installed. Startup validates presence;
-it does not send an email or validate domain ownership with Resend. Updating the
-API key causes Agent to restart notifications. Driver/sender changes require an
-image rebuild. Existing API keys survive bootstrap. Migration removes legacy
-driver/sender fields from the current Vault secret with a version-checked write;
-the preceding version is preserved in Vault history and the private archive.
+No placeholder password or delivery fallback is installed. Startup validates
+configuration and secret presence; the SMTP connection is opened on demand.
+Updating the password causes Agent to restart notifications. Driver/sender
+changes require an image rebuild. Old Resend secrets remain in Vault history
+but are not used or readable by the notifications AppRole. HTTP providers remain
+available for other deployments; switching one requires matching Agent/policy
+and runtime configuration. SMTP has no provider idempotency guarantee: explicit
+4xx rejections retry, 5xx rejections fail, and ambiguous transport outcomes require
+operator reconciliation before another send.
 
 ## Browser HTTPS and internal connections
 

docs/local-sendico.md

@@ -105,6 +105,27 @@ docker compose --project-directory . -f infra/local/compose.yml \
   -f infra/local/sendico/compose.yml --env-file .local/dev/images.env restart sendico
 ```
 
+A BFF container restart can leave its session worker in the old shared network
+namespace. The worker checks Vault DNS every ten seconds, with a five-second
+lookup limit. Three consecutive failures log a credential-free recovery reason
+and terminate the session process; Docker's restart policy then attaches it to
+the BFF's current namespace. A successful lookup resets the counter, so transient
+DNS failures do not immediately restart the worker.
+
+Container recreation changes the BFF container ID as well as its namespace.
+Bootstrap and credential-rotation tooling recreate BFF and its configured session
+worker together. Compose also marks the session dependency for restart when BFF
+is explicitly updated. For a manual forced recreation, select both services:
+
+```sh
+docker compose --project-directory . -f infra/local/compose.yml \
+  -f infra/local/sendico/compose.yml --env-file .local/dev/images.env \
+  up -d --no-deps --force-recreate --wait bff sendico
+```
+
+The parent process cleans up its bridge, DNS monitor and renewal delay on exit.
+This recovery retains the configured provider URL, account and replay namespace.
+
 `make test-dev` checks seven healthy services, verified browser TLS, the Vault
 layout and least-privilege identities, Mongo transactions/lease rotation, and
 authenticated Sendico currency and ledger reads through the same loopback
@@ -130,6 +151,21 @@ and the established CA. A subsequent verified TLS check must succeed.
 
 ## Acceptance and operator UX review
 
+Recovery follow-up for [task #57](https://git.sendico.io/tech/sxchnge/issues/57) /
+[PR #59](https://git.sendico.io/tech/sxchnge/pulls/59), validated on 2026-10-19:
+`make test-dev-tooling`, `make test-dev-rotation`, Compose configuration validation,
+`sh infra/local/tools.sh verify` and `git diff --check` pass. A BFF-only restart
+triggered automatic session restart and restored matching network namespaces;
+paired recreation also finished healthy. Rotation observed three real Vault
+pool replacements, an unchanged BFF process and 534 transaction-backed requests
+with zero failures. The final check verifies all seven SXCHNGE services, browser
+HTTPS, Vault/database boundaries and authenticated Sendico currency/ledger reads.
+An earlier check failed because Sendico's ledger announcement connection was
+closed; restoring Discovery and its service announcers in dependency order
+recovered that existing stand. No Sendico source or financial data was changed,
+and no email or payment was sent. This follow-up changes shell/Compose tooling;
+it does not rerun the earlier full Go/Flutter suite for unchanged product source.
+
 Validated on 2026-10-16: `make build`, `make up`, `make test-dev-tooling`,
 `make test-dev-infra`, `make test-dev`, and `git diff --check` passed. Both product
 sites, Sendico Backoffice and both API health endpoints returned HTTP 200;

docs/signup-facts.md

@@ -5,8 +5,191 @@ Task [#46](https://git.sendico.io/tech/sxchnge/issues/46), branch
 customer creation, runtime, shared database source and Flutter error feedback.
 Delivery: [PR #47](https://git.sendico.io/tech/sxchnge/pulls/47), targeting `main`.
 
+## Registration response correction
+
+Follow-up [#57](https://git.sendico.io/tech/sxchnge/issues/57), branch
+`identity-facts-57`, delivered in [PR #59](https://git.sendico.io/tech/sxchnge/pulls/59)
+against `main` (implementation `abca15f`), corrects the registration success status
+to HTTP 201 Created. The PR remains unmerged.
+Sendico's individual creation handler returns 201; the original adapter and test
+fixture incorrectly required 200. A successful registration therefore left a
+pending customer with an unresolved claim and `signup_attention_required`.
+The adapter now checks the exact status per operation before validating receipts:
+201 for registration and 200 for Facts intake. Other statuses, including other
+2xx responses, remain failures. Malformed receipts and uncertain registrations
+still require the existing recovery rules.
+
+On 2026-10-17, the affected local signup was reconciled using its registration
+correlation in Sendico logs and an authenticated organization-scoped individual
+GET. A conditional majority write bound that confirmed individual to the existing
+local claim, preserving the saved facts, namespace and registration request.
+The rebuilt BFF resumed the public activation-resend flow: Facts intake returned
+a persisted receipt and the endpoint returned 202 after activation-email queue
+acceptance. No replacement individual was registered. The account remains
+pending until the customer confirms the email; queue acceptance does not prove
+inbox delivery.
+
+Recovery also exposed a notification worker lifetime issue: its NATS subscription
+retained the 20-second startup context and was automatically unsubscribed after
+that deadline. Resource provisioning now has its own bounded context, while the
+subscription follows the service lifetime and closes at shutdown. A regression
+test checks that releasing provisioning leaves the subscription context active,
+and service cancellation still ends it.
+
+The rebuilt worker processed the queued activation message. The live Resend
+provider rejected it with `provider_authentication_failed` (HTTP 401 or 403),
+leaving delivery in `failed` after one attempt. Later diagnostics below identify
+the unverified sender domain as the actual blocker. Correct domain authorization
+before activation resend with a new idempotency key. Credential rotation is not
+required by that error; if a separate rotation is needed, use `make email` and
+allow Vault Agent to refresh/restart the notification process. Do not replay the
+failed notification as a success or mark the customer active on their behalf.
+
+Validation passed: `GOFLAGS=-p=2 make test` (all backend race tests and canonical
+frontend suites), `make lint-backend`, focused provider/identity race tests and
+notification bus/app race tests. The HTTP 201 regression first failed against the
+old adapter and passed after correction. An earlier full run hit a database
+credential-rotation timing failure at `rotation_test.go:249` during the Docker
+build; the isolated race-test retry and both subsequent full runs passed.
+Local BFF and notifications images were rebuilt with fresh upstream manifests
+and resolved stable digest pins; the BFF, session sidecar and notifications
+services are healthy. NATS shows the worker still waiting for deliveries beyond
+the startup deadline, with no pending or unacknowledged jobs. Live email delivery
+is blocked by provider authorization as described above. No security scans were
+requested or run. Unrelated notifications runtime settings are excluded from
+the task commits.
+
+### Provider diagnostics follow-up
+
+The same task/PR adds transient provider diagnostics to the worker's existing
+structured outcome log: `provider_status`, `provider_error` and
+`provider_message`. The adapter extracts bounded Resend/SendGrid error fields and
+redacts credentials, request values, addresses, URLs and token/code fragments.
+Transport errors have a sanitized cause; invalid or oversized responses have a
+safe fallback. Diagnostics remain outside stored notification models and public
+gRPC responses. The outcome is logged even when its result write fails. See
+the [notification logging contract](../backend/notifications/README.md).
+
+Runtime inspection confirmed `email.driver: resend`, an API key in Vault, and
+sender `noreply@sxchnge.com`. A live diagnostic notification used Resend's
+documented `delivered+sxchnge-config@resend.dev` test mailbox, without retrying
+the customer's email. The rebuilt worker logged HTTP 403,
+`provider_error: validation_error`, and the sanitized message:
+`The sxchnge.com domain is not verified. Please, add and verify your domain on [redacted-url]`.
+The precise blocker is sender-domain verification, not missing local Resend
+configuration. A sending-only key is appropriate for sending; its inability to
+read Resend's logs API does not explain the original send failure. Verify the
+configured domain in Resend (and its required DNS records), or explicitly choose
+an already verified sender domain. No sender/domain/DNS setting was changed by
+this follow-up; the customer's account remains pending email confirmation.
+
+Follow-up validation passed: `GOFLAGS=-p=2 make test`, `make lint-backend`, and
+`go -C backend/notifications test -race ./email ./service -count=1`. Coverage
+includes Resend/SendGrid messages, malformed/trailing/oversized response bodies,
+credential/content/code redaction, Unicode-safe truncation, sanitized transport
+errors, and exactly one provider outcome log even when persistence fails. The
+notifications image was rebuilt from fresh stable digest pins and deployed
+healthy; the live test verified both the diagnostic log and the unchanged generic
+gRPC failure response. No security scan or real customer email retry was run.
+
 ## Behavior and boundaries
 
+### Local mail client
+
+The user subsequently selected Sendico's existing mailbox instead of Resend for
+local notifications. The authoritative Sendico configuration resolves to
+`no-reply@sendico.io`, `mail.sendico.io:465`, authenticated implicit TLS and a
+10-second network timeout. SXCHNGE's owned `smtp` adapter consumes rendered email
+models through the existing sender contract and queue; it does not import or
+access Sendico business modules, databases or messaging. The password was copied
+once, without printing it, from the running Sendico notification environment into
+`services/notifications/email/smtp` in SXCHNGE Vault. Only `password` is stored
+there; host, port, username, display name and sender remain in runtime YAML.
+
+The SMTP adapter verifies certificates, authenticates before submission, sends
+UTF-8 MIME alternatives, and requires the final DATA 250 reply for acceptance.
+SMTP 4xx replies retry through the existing worker policy; 5xx replies fail;
+ambiguous transport outcomes remain UNKNOWN and are not immediately resent.
+Message-ID is correlation only, not a provider idempotency guarantee. Existing
+failed Resend notifications are not replayed or rewritten. The unverified
+`sxchnge.com` Resend domain is no longer the selected local delivery path.
+
+Live verified-TLS SMTP authentication and NOOP succeeded with reply 250 using
+the password from SXCHNGE Vault, without submitting email. SMTP adapter race
+tests, `make lint-backend`, `make test-dev-tooling` and `make test-dev-infra`
+pass, including canonical Vault layout, service permission isolation, database
+credential issuance/renewal/revocation and transactions. The final
+`GOFLAGS=-p=2 make test` passes all backend race tests and frontend suites.
+
+Earlier attempts hit Docker archive-copy/reaper timeouts, Go proxy TLS failures
+and stalled cache extraction/container startup. The user authorized restarting
+Docker Desktop. After recovery, the normal Compose `build --pull notifications`
+succeeded with the resolved stable digest pins; no checks were disabled. The
+generator stage shares the existing BuildKit Go module/build caches and retains
+the pinned tools and canonical Make generation command. A discarded fallback
+compiler attempt also failed to access its checksum-state directory; it was not
+used for the deployed artifact.
+
+The matching Vault Agent policy/template were applied, and notifications was
+recreated healthy. Its running image matches the rebuilt image
+`sha256:d4cbc0cb9005e9cef9381abdf2a13f672104d160536ee6d77ac73a909c0a8638`,
+and its startup log confirms `email_driver: smtp`. No customer email was sent or
+failed Resend job replayed; SMTP authentication is not a recipient-delivery
+check. Sendico's Vault sealed during the Docker restart and was unsealed with
+its existing local key, without reinitialization or exposing credentials.
+Its affected services were restarted in dependency order. The local Sendico
+session sidecar shares the BFF network namespace, so it was restarted after the
+BFF to restore Docker DNS and the loopback bridge. Final
+`sh infra/local/tools.sh verify` passes: seven healthy services, verified HTTPS
+for frontend/proxied BFF, authenticated Sendico currency/ledger reads and the
+Vault/database checks. No Sendico repository files were changed.
+
+On October 18, the frontend had 15,897 processes, including 15,888 unreaped
+`ssl_client` zombie children of Caddy PID 1. The three-second HTTPS health
+probe used BusyBox `wget`, which left those children behind. This confirmed a
+recurring local infrastructure leak, independently of SMTP authentication.
+Compose now enables `init: true` for frontend, and only that container was
+recreated. PID 1 is now `docker-init`; 40 consecutive HTTPS probes completed
+with zero zombie processes and a bounded count of 15 processes afterward.
+Compose configuration validation and `make test-dev-tooling` pass. The API
+health endpoint returns HTTP 200; a diagnostic request using the existing
+account and a generated incorrect password returned HTTP 400 `email_in_use`
+without creating an account or sending email.
+
+The contemporaneous full local verification attempt is blocked by MongoDB's
+five-second health-check timeout under continuing host load; it is not recorded
+as passing. The exact response to the reported Safari attempt was not captured,
+and the frontend's generic unavailable text also covers some HTTP 400 input
+errors. A fresh user retry is needed to confirm the visible signup outcome.
+
+On October 19 the failing API was directly observed returning HTTP 502. The BFF
+had restarted while the session worker retained its old network namespace;
+`/proc/self/ns/net` differed between them, and the worker could not resolve
+Vault. Sendico's public API itself returned HTTP 200. Restarting only the worker
+reattached its namespace and restored SXCHNGE API HTTP 200. This is a confirmed
+cause of that 502, separate from the previous process leak and host-load finding.
+
+The session now monitors bounded DNS failures and exits after three consecutive
+failures so Docker can reattach it automatically. Bootstrap and rotation pair
+BFF/session recreation, with Compose dependency restart enabled. Offline shell
+fixtures cover transient-failure reset, sustained-failure recovery, safe parent
+targeting and conditional service pairing. A live BFF-only restart automatically
+restarted the session and restored matching namespaces, fresh session readiness
+and API HTTP 200. A second live test recreated BFF and its session together and
+confirmed healthy services with matching namespaces. No provider bindings,
+customer records, credentials or Sendico source files were changed.
+
+Final October 19 validation passes: `make test-dev-tooling`, Compose config
+validation, `make test-dev-rotation` (three real rotations, unchanged BFF process,
+534 transaction-backed requests and zero failures), and
+`sh infra/local/tools.sh verify`. The last confirms all seven healthy services,
+verified frontend/BFF HTTPS, Vault/database checks and authenticated Sendico
+currency/ledger reads. An initial ledger-read failure was resolved by restoring
+Sendico Discovery and its announcers in their documented dependency order after
+closed NATS connections. No email was sent during validation. The recovery fix
+and its validation are included in [PR #59](https://git.sendico.io/tech/sxchnge/pulls/59).
+
+
 With `api.sendico` configured, the product BFF creates the local pending customer
 and activation token atomically, then registers an independent individual through
 Sendico's public HTTP API and submits the sign-up assertions. It persists the
@@ -18,7 +201,7 @@ backfilled or blocked at login.
 
 The adapter consumes these existing Sendico contracts:
 
-- `POST /individuals/{organization_ref}`, empty body, HTTP 200 registration
+- `POST /individuals/{organization_ref}`, empty body, HTTP 201 registration
   receipt containing `subject_ref` and `registered_at`.
 - `POST /identity_facts/{organization_ref}`, HTTP 200 immutable intake receipt
   containing `submission_ref`, `subject_ref` and `accepted_at`.
@@ -73,9 +256,10 @@ the adapter rereads it on every request. Startup rejects missing, expired or
 mismatched credentials and rejects Sendico configuration without notifications.
 
 HTTPS is required except for loopback HTTP test/development endpoints. Redirects
-are rejected and each HTTP call has a four-second timeout. Non-200 responses,
-malformed/truncated receipts, invalid identifiers, wrong subjects and unsuccessful
-envelopes never count as acceptance. Upstream error bodies are not surfaced.
+are rejected and each HTTP call has a four-second timeout. Unexpected response
+statuses (registration requires 201; Facts intake requires 200), malformed/truncated
+receipts, invalid identifiers, wrong subjects and unsuccessful envelopes never
+count as acceptance. Upstream error bodies are not surfaced.
 
 Omitting `api.sendico` keeps existing local-only behavior. Enabling it also gates
 activation for previously pending customers; use activation resend to submit
@@ -85,6 +269,34 @@ adapter fingerprint and blocks existing bindings for explicit reconciliation.
 Token rotation for the same account retains that fingerprint. Plan such changes
 as migrations; never reset a binding just to bypass the check.
 
+## Local signup request diagnosis
+
+The October 19 follow-up screenshot still shows the generic unavailable message
+after the network recovery. That screenshot alone does not establish a 503:
+the frontend also maps some rejected inputs and browser transport failures to
+that message. At the follow-up check the health endpoint returned 200, and BFF
+signup counters contained only the operator's two deliberately invalid probes.
+The user's current browser failure has not yet been reproduced or resolved.
+
+Local Caddy now logs requests to exactly `/api/v1/accounts/signup` under
+`http.log.access.signup`, including HTTP status and duration. The complete
+request object, response headers and user identifier are removed before log
+encoding, so paths/queries, cookies, authorization, addresses and user-supplied
+headers cannot enter these access records. Bodies are never logged. Other
+requests, including health probes and email confirmation URLs, are skipped.
+Inspect with:
+
+```sh
+docker logs --since 5m sxchnge-local-frontend-1 2>&1 |
+  jq -Rrc 'fromjson? | select(.logger=="http.log.access.signup")'
+```
+
+Validation: Caddy configuration validation and reload passed; a deliberately
+invalid signup returned 400 and produced the expected status/duration record.
+Sentinel query, cookie and custom-header values were absent from its log.
+The frontend was rebuilt with freshly resolved stable image pins. No real
+signup or customer email was submitted during this diagnostic work.
+
 ## Persistence, retries and recovery
 
 The local `signup_facts` collection uses the customer ID as `_id`, so its unique