Merge branch 'main' into prod-vault-74

The internal name of the product is exch, like its production server.
Paths, the Compose project, the Docker network, raft node ids and the
transit seal key and policy use it before the first rollout, so nothing
has to be renamed once data exists.

AGENTS.md

@@ -13,14 +13,17 @@
 - Never commit autogenerated code to the repository. This applies to backend,
   frontend, shared packages, and every other module, including generated
   protobuf bindings, API clients, serializers, and framework-generated code.
-- Commit the source definitions, generator configuration, and reproducible
+- Use externally published schemas as generation inputs. Do not embed or commit
-  generation commands instead. Generate code locally or during build and CI
+  schema snapshots in the repository. Commit external schema references,
-  workflows before any dependent validation or compilation steps.
+  generator configuration, and reproducible generation commands instead.
+- Fetch the externally published schemas and generate code at build time, before
+  any dependent validation or compilation steps, including in CI.
 - Keep generated outputs ignored by Git. Do not force-add ignored generated
   files or copy generated code into tracked source files.
-- Change the source definitions or generator inputs rather than editing
+- Change schema sources in their owning project or update external schema
-  generated outputs. Document the generation workflow so a clean checkout can
+  references and generator inputs rather than editing generated outputs.
-  build without committed generated code.
+  Document the generation workflow so a clean checkout can build without
+  committed generated code or embedded schema snapshots.
 
 ## Local development tooling
 

Makefile

@@ -80,6 +80,12 @@ test-dev:
 	sh infra/local/tools.sh test
 	sh infra/local/tools.sh verify
 
+# Production pins (infra/prod/README.md): resolved once, copied to every node.
+.PHONY: prod-images
+prod-images:
+	sh infra/local/tools.sh build
+	sh infra/local/tools.sh prod-images
+
 .PHONY: test-dev-infra
 test-dev-infra:
 	sh infra/local/tools.sh test

docs/ux/photo-preview.md

@@ -0,0 +1,98 @@
+# Identity photo preview correction
+
+Task [#69](https://git.sendico.io/tech/sxchnge/issues/69), branch `kyc-69`.
+Delivery: [PR #71](https://git.sendico.io/tech/sxchnge/pulls/71).
+This frontend change builds on the [identity verification journey](kyc.md)
+and is independent of the birthday controls in task #67.
+
+## Cause and correction
+
+A synthetic valid PNG reproduced this Flutter 3.47.7 browser error:
+
+```text
+TypeError: Instance of '_UnmodifiableUint8ListView': type
+'_UnmodifiableUint8ListView' is not a subtype of type 'NativeUint8List'
+```
+
+The stack reached `Uint8ListToJSUint8Array.toJS` from CanvasKit's browser
+image decoder through `MemoryImage` and `ResizeImage`. File selection had
+successfully decoded the original typed bytes. The `Photo` constructor then
+copied them into an unmodifiable view, which was passed directly into the
+preview's web decoder. Native decoding accepted that view, hiding the bug in
+native-only tests. The generic image error builder presented the failure as a
+damaged document even though the selected bytes were valid.
+
+`PhotoPreview` now creates a native `Uint8List` copy for `MemoryImage` once per
+selected photo. The shared domain model stays immutable, and validation,
+providers, services and transport contracts remain unchanged. The preview still
+requests a decode width of at most 1000 pixels and evicts its decoded cache entry
+on replacement or disposal. The extra encoded-byte copy belongs to the preview
+and is released with its provider/cache references; no secure-erasure guarantee
+is made.
+
+An actual preview failure now says that the preview could not be displayed and
+suggests selecting the photo again. The English/Russian fallback keeps at least
+the preview's usual height, can grow for long text, and has a separate live-region
+semantic node. Existing replacement/removal controls stay available and selected
+input is retained. This is distinct from validation's corrupt-file message.
+
+## UX and measurement review
+
+The outcome remains selecting readable ID and selfie photos, checking both on
+review, and submitting for confirmed service acceptance. Existing KYC journey
+attempts, retries, failure, submission and abandonment signals are reused. No
+control-level events, image bytes, filenames, form values or raw exceptions are
+added to telemetry. A selection success measures validation; it does not establish
+that pixels were painted. Preview-only rendering failures are not separately
+counted by existing telemetry. Server review/approval measurement and native
+screen-reader QA retain the limitations documented in the original journey.
+
+Regression tests use synthetic JPEG/PNG content only. They start and await real
+image decoding outside the widget test's fake async zone, inspect the actual
+rendered images, and surface stream errors so an error fallback cannot conceal a
+failure. Checks cover portrait/landscape decoding, model immutability, bounded
+decode width, stable provider reuse, replacement/removal cache eviction, both
+review slots, returning to review, localized fallback/replacement, a 320px view
+at 1.6 text scale, and error announcements isolated from surrounding instructions.
+The `image` package is already a KYC runtime dependency; its direct app test
+dependency is used only to create synthetic fixtures in memory.
+
+## Validation and delivery
+
+The original browser regression failed before the fix while its native equivalent
+passed. The complete task diff was reviewed against frontend architecture, UX,
+measurement, lifecycle and privacy requirements. Final validation passed:
+
+- In `frontend/app`, `dart format --output=none --set-exit-if-changed
+  lib/kyc/preview.dart test/preview_test.dart`: two files, no changes.
+- In `frontend/app`, `flutter analyze --no-pub`: no issues.
+- In `frontend/app`, `flutter test --platform chrome test/preview_test.dart
+  --reporter expanded --no-pub --verbose`: all six regression tests passed.
+- At the repository root, `make test-frontend`: all 228 tests passed across
+  the shared packages, theme example and app, including the six native preview
+  regressions and existing KYC journey/analytics tests.
+- In `frontend/app`, `flutter build web --no-pub`: release build and Wasm
+  compatibility dry run succeeded.
+
+Browser regression coverage uses Chrome; Safari and native assistive-technology
+testing remain release checks. Generated localization and build code stays
+ignored; the tracked app lockfile changes only `image` from transitive to direct
+test dependency. The separate birthday worktree was preserved.
+
+## Follow-up contribution review
+
+The complete seven-file contribution was reviewed again before the authorized
+merge and frontend rollout. No additional defects were found in the byte-copy
+boundary, provider reuse, cache disposal, error semantics/localization, privacy,
+architecture or regression coverage. No source changes were needed.
+
+Current `main` (`ec4115d`, including the birthday, signup-delivery and account
+setup changes) was merged into `kyc-69` before final verification. The two-file
+format check and app analyzer passed; all six Chrome preview regressions and
+all 240 tests in `make test-frontend` passed on the combined code. The original
+228-test result above describes the earlier isolated branch.
+
+The authorized rollout rebuilds only `frontend` from clean, pulled `main`, using
+the repository's fresh stable-release resolver and digest-pinned local inputs.
+Runtime image, HTTPS and served-bundle verification are recorded in task #69
+after the rollout; generated local pins remain ignored.

frontend/app/lib/kyc/preview.dart

@@ -1,3 +1,5 @@
+import 'dart:typed_data';
+
 import 'package:flutter/material.dart';
 
 import 'package:sxchnge_kyc/sxchnge_kyc.dart';
@@ -15,8 +17,12 @@ class PhotoPreview extends StatefulWidget {
 
 class _PhotoPreviewState extends State<PhotoPreview> {
   late ImageProvider _image = _provider();
-  ImageProvider _provider() =>
+  ImageProvider _provider() => ResizeImage.resizeIfNeeded(
-      ResizeImage.resizeIfNeeded(1000, null, MemoryImage(widget.photo.bytes));
+    1000,
+    null,
+    // Web decoders require a native typed array, not the model's read-only view.
+    MemoryImage(Uint8List.fromList(widget.photo.bytes)),
+  );
   @override
   void didUpdateWidget(covariant PhotoPreview oldWidget) {
     super.didUpdateWidget(oldWidget);
@@ -38,7 +44,15 @@ class _PhotoPreviewState extends State<PhotoPreview> {
     height: 220,
     fit: BoxFit.contain,
     semanticLabel: widget.label,
-    errorBuilder: (_, _, _) =>
+    errorBuilder: (_, _, _) => ConstrainedBox(
-        Text(AppLocalizations.of(context)!.kycErrorUnreadable),
+      constraints: const BoxConstraints(minHeight: 220),
+      child: Center(
+        child: Semantics(
+          container: true,
+          liveRegion: true,
+          child: Text(AppLocalizations.of(context)!.kycErrorPreview),
+        ),
+      ),
+    ),
   );
 }

frontend/app/lib/l10n/en.arb

@@ -128,6 +128,7 @@
   "kycErrorFormat": "Choose a JPEG or PNG image. Other formats are not supported.",
   "kycErrorLarge": "This image is too large. Choose one under 10 MB and 40 megapixels.",
   "kycErrorUnreadable": "This image is damaged or cannot be read. Choose another photo.",
+  "kycErrorPreview": "The photo preview could not be displayed. Try choosing the photo again.",
   "kycErrorSmall": "The photo is too small. Use an image at least 600 pixels wide and high.",
   "kycErrorIncomplete": "Choose both photos and confirm they belong to you before submitting.",
   "kycErrorSession": "Your session has expired. Return to login, then check your verification status.",

frontend/app/lib/l10n/ru.arb

@@ -128,6 +128,7 @@
   "kycErrorFormat": "Выберите изображение JPEG или PNG. Другие форматы не поддерживаются.",
   "kycErrorLarge": "Изображение слишком большое. Выберите файл до 10 МБ и 40 мегапикселей.",
   "kycErrorUnreadable": "Изображение повреждено или не читается. Выберите другое фото.",
+  "kycErrorPreview": "Не удалось показать фото. Попробуйте выбрать его ещё раз.",
   "kycErrorSmall": "Фото слишком маленькое. Ширина и высота должны быть не менее 600 пикселей.",
   "kycErrorIncomplete": "Выберите оба фото и подтвердите, что на них вы и ваш документ.",
   "kycErrorSession": "Сессия истекла. Войдите снова и проверьте статус верификации.",

frontend/app/pubspec.lock

@@ -222,7 +222,7 @@ packages:
     source: hosted
     version: "4.1.2"
   image:
-    dependency: transitive
+    dependency: "direct dev"
     description:
       name: image
       sha256: "1976370a4df3091bb0f72409c187ad1f9132a818bc6b95ca59c0bae1c75c688e"

frontend/app/pubspec.yaml

@@ -29,6 +29,7 @@ dependencies:
   go_router: ^18.0.1
   country_picker: ^2.0.28
 dev_dependencies:
+  image: ^4.9.0
   flutter_test:
     sdk: flutter
   flutter_lints: ^6.0.0

frontend/app/test/preview_test.dart

@@ -0,0 +1,229 @@
+import 'dart:async';
+import 'dart:typed_data';
+
+import 'package:flutter/material.dart';
+
+import 'package:flutter_test/flutter_test.dart';
+
+import 'package:image/image.dart' as images;
+
+import 'package:sxchnge_kyc/sxchnge_kyc.dart';
+
+import 'package:sxchnge_app/kyc/photo.dart';
+import 'package:sxchnge_app/kyc/preview.dart';
+
+import 'package:sxchnge_app/generated/i18n/app_localizations.dart';
+
+Widget host(Widget child, {Locale locale = const Locale('en')}) => MaterialApp(
+  locale: locale,
+  localizationsDelegates: AppLocalizations.localizationsDelegates,
+  supportedLocales: AppLocalizations.supportedLocales,
+  home: Scaffold(body: SingleChildScrollView(child: child)),
+);
+
+// Start decoding outside FakeAsync as well as waiting outside it, including
+// browser codec callbacks that otherwise remain pending in the test zone.
+Future<void> show(WidgetTester tester, Widget widget) async {
+  await tester.runAsync(() => tester.pumpWidget(widget));
+}
+
+// pumpAndSettle alone does not wait for image codecs. Listen to the actual
+// preview streams so a hidden errorBuilder cannot make this regression pass.
+Future<void> decoded(WidgetTester tester, {bool succeeds = true}) async {
+  final providers = tester
+      .widgetList<Image>(find.byType(Image))
+      .map((image) => image.image)
+      .toList();
+  expect(providers, isNotEmpty);
+  for (final provider in providers) {
+    Object? renderError;
+    StackTrace? renderStack;
+    await tester.runAsync(() async {
+      final stream = provider.resolve(ImageConfiguration.empty);
+      final ready = Completer<void>();
+      final listener = ImageStreamListener(
+        (info, _) {
+          info.dispose();
+          if (!ready.isCompleted) ready.complete();
+        },
+        onError: (Object error, StackTrace? stack) {
+          renderError = error;
+          renderStack = stack;
+          if (!ready.isCompleted) ready.complete();
+        },
+      );
+      stream.addListener(listener);
+      try {
+        await ready.future.timeout(const Duration(seconds: 15));
+      } finally {
+        stream.removeListener(listener);
+      }
+    });
+    expect(
+      renderError,
+      succeeds ? isNull : isNotNull,
+      reason: succeeds ? '$renderError\n$renderStack' : null,
+    );
+  }
+  await tester.pumpAndSettle();
+  expect(tester.takeException(), isNull);
+}
+
+void main() {
+  late Uint8List png;
+  late Uint8List jpeg;
+  setUpAll(() {
+    // Synthetic, non-sensitive portrait and landscape photos in both formats.
+    png = images.encodePng(images.Image(width: 1200, height: 1600));
+    jpeg = images.encodeJpg(images.Image(width: 1600, height: 1200));
+  });
+
+  for (final format in ['PNG', 'JPEG']) {
+    testWidgets('accepted $format renders from an immutable photo model', (
+      tester,
+    ) async {
+      final source = Uint8List.fromList(format == 'PNG' ? png : jpeg);
+      final photo = (await tester.runAsync(
+        () => FilePhotoService.validate(source),
+      ))!;
+      final originalFirstByte = photo.bytes.first;
+      source[0] = 0;
+      expect(photo.bytes.first, originalFirstByte);
+      expect(() => photo.bytes[0] = 0, throwsUnsupportedError);
+      await show(
+        tester,
+        host(PhotoPreview(photo: photo, label: 'ID document photo')),
+      );
+      await decoded(tester);
+      final raw = tester.widget<RawImage>(find.byType(RawImage));
+      expect(raw.image, isNotNull);
+      expect(raw.image!.width, lessThanOrEqualTo(1000));
+      expect(
+        tester.widget<Image>(find.byType(Image)).semanticLabel,
+        'ID document photo',
+      );
+      expect(find.textContaining('damaged'), findsNothing);
+      expect(find.textContaining('could not be displayed'), findsNothing);
+    });
+  }
+
+  testWidgets('replacement and removal evict sensitive preview cache entries', (
+    tester,
+  ) async {
+    final first = Photo(png);
+    await show(tester, host(PhotoPreview(photo: first, label: 'ID')));
+    await decoded(tester);
+    final initialProvider = tester.widget<Image>(find.byType(Image)).image;
+    final initialKey = await initialProvider.obtainKey(
+      ImageConfiguration.empty,
+    );
+    final cache = PaintingBinding.instance.imageCache;
+    expect(cache.statusForKey(initialKey).keepAlive, isTrue);
+    await show(tester, host(PhotoPreview(photo: first, label: 'Document')));
+    await decoded(tester);
+    expect(
+      identical(
+        tester.widget<Image>(find.byType(Image)).image,
+        initialProvider,
+      ),
+      isTrue,
+    );
+    await show(tester, host(PhotoPreview(photo: Photo(jpeg), label: 'ID')));
+    await decoded(tester);
+    expect(cache.statusForKey(initialKey).untracked, isTrue);
+    final replacement = tester.widget<Image>(find.byType(Image)).image;
+    final replacementKey = await replacement.obtainKey(
+      ImageConfiguration.empty,
+    );
+    expect(cache.statusForKey(replacementKey).keepAlive, isTrue);
+    expect(tester.widget<RawImage>(find.byType(RawImage)).image!.width, 1000);
+    await show(tester, host(const SizedBox()));
+    await tester.pumpAndSettle();
+    expect(cache.statusForKey(replacementKey).untracked, isTrue);
+  });
+
+  testWidgets('ID and selfie both render on review and after returning', (
+    tester,
+  ) async {
+    final passport = Photo(png);
+    final selfie = Photo(jpeg);
+    for (var visit = 0; visit < 2; visit++) {
+      await show(
+        tester,
+        host(
+          Column(
+            children: [
+              PhotoPanel(
+                kind: PhotoKind.passport,
+                photo: passport,
+                review: true,
+              ),
+              PhotoPanel(kind: PhotoKind.selfie, photo: selfie, review: true),
+            ],
+          ),
+        ),
+      );
+      await decoded(tester);
+      final previews = tester.widgetList<RawImage>(find.byType(RawImage));
+      expect(previews.length, 2);
+      expect(previews.every((image) => image.image != null), isTrue);
+      expect(find.text('Choose a different photo'), findsNothing);
+      await show(tester, host(const SizedBox()));
+      await tester.pumpAndSettle();
+    }
+  });
+
+  for (final locale in ['en', 'ru']) {
+    testWidgets(
+      '$locale preview failure explains recovery and can be replaced',
+      (tester) async {
+        tester.view.physicalSize = const Size(320, 900);
+        tester.view.devicePixelRatio = 1;
+        tester.platformDispatcher.textScaleFactorTestValue = 1.6;
+        addTearDown(tester.view.resetPhysicalSize);
+        addTearDown(tester.view.resetDevicePixelRatio);
+        addTearDown(tester.platformDispatcher.clearTextScaleFactorTestValue);
+        var replacements = 0;
+        var removals = 0;
+        await show(
+          tester,
+          host(
+            PhotoPanel(
+              kind: PhotoKind.passport,
+              photo: Photo(Uint8List.fromList([1, 2, 3])),
+              onPick: () => replacements++,
+              onRemove: () => removals++,
+            ),
+            locale: Locale(locale),
+          ),
+        );
+        await decoded(tester, succeeds: false);
+        final l = AppLocalizations.of(tester.element(find.byType(PhotoPanel)))!;
+        expect(find.text(l.kycErrorPreview), findsOneWidget);
+        expect(find.text(l.kycErrorUnreadable), findsNothing);
+        expect(
+          tester.getSemantics(find.text(l.kycErrorPreview)),
+          matchesSemantics(isLiveRegion: true, label: l.kycErrorPreview),
+        );
+        final replace = find.byKey(const ValueKey('pick-passport'));
+        await tester.ensureVisible(replace);
+        await tester.tap(replace);
+        final remove = find.text(l.kycRemove);
+        await tester.ensureVisible(remove);
+        await tester.tap(remove);
+        expect(replacements, 1);
+        expect(removals, 1);
+        await show(
+          tester,
+          host(
+            PhotoPanel(kind: PhotoKind.passport, photo: Photo(jpeg)),
+            locale: Locale(locale),
+          ),
+        );
+        await decoded(tester);
+        expect(find.text(l.kycErrorPreview), findsNothing);
+        expect(tester.widget<RawImage>(find.byType(RawImage)).image, isNotNull);
+      },
+    );
+  }
+}

infra/local/scripts/dispatch.sh

@@ -5,6 +5,7 @@ set -o pipefail
 . infra/local/scripts/secrets.sh
 case "${1:-}" in
  images) . infra/local/scripts/images.sh; images;;
+ prod-images) TOOL_STATE="$PWD/.local"; . infra/local/scripts/images.sh; prod_images;;
  up) . infra/local/scripts/bootstrap.sh; up;;
  email) . infra/local/scripts/bootstrap.sh; email;;
  sendico) . infra/local/scripts/bootstrap.sh; . infra/local/scripts/sendico.sh; configure_sendico;;

infra/local/scripts/images.sh

@@ -22,3 +22,10 @@ images() (
  cat "$WORK/images.env" | write "$STATE/images.env"
  cat "$STATE/images.env"
 )
+# Production pins: one resolution for every node of a cluster, so all of them
+# run the same build. Written outside .local/dev; copied to the servers.
+prod_images() (
+ ref=$(image hashicorp/vault latest) || exit 1
+ printf 'VAULT_IMAGE=%s\n' "$ref" | write "$PWD/.local/prod/images.env"
+ cat "$PWD/.local/prod/images.env"
+)

infra/local/scripts/test.sh

@@ -3,7 +3,7 @@ set -eu
 set -o pipefail
 # Shell syntax and static analysis cover the actual container shell scripts.
 shellcheck -s ash -x -S warning infra/local/scripts/dispatch.sh infra/local/scripts/test.sh
-shellcheck -x -S warning infra/local/tools.sh infra/local/scripts/start-service.sh infra/local/mongo/start.sh infra/local/nats/start.sh infra/local/vault/start.sh
+shellcheck -x -S warning infra/local/tools.sh infra/local/scripts/start-service.sh infra/local/mongo/start.sh infra/local/nats/start.sh infra/local/vault/start.sh infra/prod/vault/start.sh
 shellcheck -s ash -x -S warning infra/local/sendico/session.sh infra/local/sendico/check.sh infra/local/sendico/network.sh
 . infra/local/scripts/common.sh
 . infra/local/scripts/images.sh

infra/prod/README.md

@@ -0,0 +1,97 @@
+# Production
+
+Production runs on two servers of the shared private network: `exch`
+(192.168.10.191), the production server, and `infra-gamma` (192.168.10.98), which
+keeps the third copy of the data. Each component runs two nodes on `exch` and
+one on `infra-gamma`: losing `infra-gamma` keeps the majority, losing one node on
+`exch` keeps two of three. Every server runs plain Docker Compose; files are
+deployed to `/srv/exch/<component>` on the server.
+
+Ports are published on the private addresses only. Traffic between the servers
+is not wrapped in TLS: the network is private and limited by security groups,
+and every component authenticates its peers.
+
+## Vault
+
+[#74](https://git.sendico.io/tech/sxchnge/issues/74). Three raft nodes:
+
+| Node | Server | API | Cluster | Compose |
+| --- | --- | --- | --- | --- |
+| `exch-vault-1` | exch | 192.168.10.191:8200 | :8201 | `vault/exch.yml` |
+| `exch-vault-2` | exch | 192.168.10.191:8210 | :8211 | `vault/exch.yml` |
+| `exch-vault-3` | infra-gamma | 192.168.10.98:8200 | :8201 | `vault/gamma.yml` |
+
+Containers on `exch` join the Docker network `exch` and use
+`http://vault:8200`; the alias belongs to both local nodes. A standby node
+forwards requests to the active one, so the leader may be on any server.
+
+Vault is not published to the internet. Its UI is opened through an SSH tunnel
+from an administrator's machine, for example
+`ssh -N -L 8200:192.168.10.191:8200 exch` and then `http://localhost:8200/ui`.
+
+Nodes unseal themselves with the `transit` seal on the platform Vault
+(`https://vault.sendico.io`, key `transit/keys/exch-vault-unseal`). Each server
+has its own periodic token in `secrets/transit-token` (root, mode 600, not in Git)
+that may only encrypt and decrypt with that key. If the platform Vault is
+unreachable while a node starts, the node stays sealed until it is reachable
+again; running nodes are not affected. `vault operator init` returns recovery
+keys, not unseal keys: they authorize operations such as generating a new root
+token but cannot unseal a node. Only the transit key can. Its backup
+(`transit/backup/exch-vault-unseal`), the recovery keys and the root token are
+kept outside the servers by their holders; see "Platform Vault lost".
+
+`make prod-images` resolves the latest stable Vault release and pins its digest in
+ignored `.local/prod/images.env`. All three nodes get the same file: a cluster
+runs one build, and an upgrade is a separate rollout of a new file, one node at a
+time.
+
+### First rollout
+
+1. On the platform Vault, with an administrator token:
+
+   ```sh
+   vault secrets list | grep -q '^transit/' || vault secrets enable transit
+   vault write transit/keys/exch-vault-unseal exportable=true allow_plaintext_backup=true
+   vault read -field=backup transit/backup/exch-vault-unseal   # store with the holders, then clear the terminal
+   vault policy write exch-vault-unseal - <<'EOF'
+   path "transit/encrypt/exch-vault-unseal" { capabilities = ["update"] }
+   path "transit/decrypt/exch-vault-unseal" { capabilities = ["update"] }
+   EOF
+   ```
+
+   Create one token per server and write it straight into that server's
+   `/srv/exch/vault/secrets/transit-token`, never to a terminal or a log:
+
+   ```sh
+   vault token create -orphan -period=768h -policy=exch-vault-unseal \
+     -display-name=exch-vault-<server> -field=token
+   ```
+
+2. Copy `vault/` and `.local/prod/images.env` to `/srv/exch/vault` on both
+   servers.
+3. Start the nodes: on `exch`
+   `docker compose --env-file images.env -f exch.yml up -d`, on `infra-gamma`
+   the same with `gamma.yml`.
+4. Initialize once, on `exch`:
+   `docker exec -it exch-vault-vault-1-1 vault operator init -recovery-shares=5 -recovery-threshold=3`.
+   The output is shown once; store the recovery keys and the root token with
+   their holders. Revoke the root token after the bootstrap; a new one is
+   generated with the recovery keys when needed.
+5. Check: `vault operator raft list-peers` lists three voters; `docker restart`
+   of each node in turn unseals it without a person and the raft committed index
+   matches on all three.
+
+### Replacing a node
+
+Stop the node, remove it with `vault operator raft remove-peer <node id>`,
+delete its volume and start it again: it joins through `retry_join` and copies the
+data from the leader. A node keeps its node id and addresses; moving it to
+another server is a change of its config file and of every `retry_join` list.
+
+### Platform Vault lost
+
+Running nodes keep working. To start a node without the platform Vault, start
+any temporary Vault, enable `transit`, load the stored key backup with
+`vault write transit/restore/exch-vault-unseal backup=<backup>`, issue a token
+with the policy above, and point `address` in the node configs at that Vault.
+Without the key backup the data cannot be unsealed again.

infra/prod/vault/config/vault-1.hcl

@@ -0,0 +1,30 @@
+# Production Vault, node 1. Only api_addr, cluster_addr and node_id
+# differ between the three node files.
+ui            = true
+disable_mlock = true
+api_addr      = "http://192.168.10.191:8200"
+cluster_addr  = "https://192.168.10.191:8201"
+
+listener "tcp" {
+  address         = "0.0.0.0:8200"
+  cluster_address = "0.0.0.0:8201"
+  # Plain HTTP on the private network; raft and request forwarding on the
+  # cluster port are encrypted by Vault itself.
+  tls_disable     = 1
+}
+
+storage "raft" {
+  path    = "/vault/file"
+  node_id = "exch-vault-1"
+  retry_join { leader_api_addr = "http://192.168.10.191:8200" }
+  retry_join { leader_api_addr = "http://192.168.10.191:8210" }
+  retry_join { leader_api_addr = "http://192.168.10.98:8200" }
+}
+
+# Auto-unseal with the platform Vault. The token comes from VAULT_TOKEN, set by
+# start.sh from the secret file; recovery keys stay with people.
+seal "transit" {
+  address    = "https://vault.sendico.io"
+  mount_path = "transit/"
+  key_name   = "exch-vault-unseal"
+}

infra/prod/vault/config/vault-2.hcl

@@ -0,0 +1,30 @@
+# Production Vault, node 2. Only api_addr, cluster_addr and node_id
+# differ between the three node files.
+ui            = true
+disable_mlock = true
+api_addr      = "http://192.168.10.191:8210"
+cluster_addr  = "https://192.168.10.191:8211"
+
+listener "tcp" {
+  address         = "0.0.0.0:8200"
+  cluster_address = "0.0.0.0:8201"
+  # Plain HTTP on the private network; raft and request forwarding on the
+  # cluster port are encrypted by Vault itself.
+  tls_disable     = 1
+}
+
+storage "raft" {
+  path    = "/vault/file"
+  node_id = "exch-vault-2"
+  retry_join { leader_api_addr = "http://192.168.10.191:8200" }
+  retry_join { leader_api_addr = "http://192.168.10.191:8210" }
+  retry_join { leader_api_addr = "http://192.168.10.98:8200" }
+}
+
+# Auto-unseal with the platform Vault. The token comes from VAULT_TOKEN, set by
+# start.sh from the secret file; recovery keys stay with people.
+seal "transit" {
+  address    = "https://vault.sendico.io"
+  mount_path = "transit/"
+  key_name   = "exch-vault-unseal"
+}

infra/prod/vault/config/vault-3.hcl

@@ -0,0 +1,30 @@
+# Production Vault, node 3. Only api_addr, cluster_addr and node_id
+# differ between the three node files.
+ui            = true
+disable_mlock = true
+api_addr      = "http://192.168.10.98:8200"
+cluster_addr  = "https://192.168.10.98:8201"
+
+listener "tcp" {
+  address         = "0.0.0.0:8200"
+  cluster_address = "0.0.0.0:8201"
+  # Plain HTTP on the private network; raft and request forwarding on the
+  # cluster port are encrypted by Vault itself.
+  tls_disable     = 1
+}
+
+storage "raft" {
+  path    = "/vault/file"
+  node_id = "exch-vault-3"
+  retry_join { leader_api_addr = "http://192.168.10.191:8200" }
+  retry_join { leader_api_addr = "http://192.168.10.191:8210" }
+  retry_join { leader_api_addr = "http://192.168.10.98:8200" }
+}
+
+# Auto-unseal with the platform Vault. The token comes from VAULT_TOKEN, set by
+# start.sh from the secret file; recovery keys stay with people.
+seal "transit" {
+  address    = "https://vault.sendico.io"
+  mount_path = "transit/"
+  key_name   = "exch-vault-unseal"
+}

infra/prod/vault/exch.yml

@@ -0,0 +1,59 @@
+# Production Vault on exch: nodes 1 and 2 of three (#74). Node 3 runs
+# on infra-gamma (gamma.yml). Ports are published on the private address only.
+name: exch-vault
+
+x-vault: &vault
+  image: ${VAULT_IMAGE:?Run make prod-images and copy images.env}
+  restart: unless-stopped
+  cap_add: [IPC_LOCK]
+  ulimits: {memlock: {soft: -1, hard: -1}}
+  environment:
+    VAULT_ADDR: http://127.0.0.1:8200
+    SKIP_SETCAP: 'true'
+  entrypoint: [/bin/sh, /etc/vault/start.sh]
+  secrets: [transit-token]
+  healthcheck:
+    test: [CMD-SHELL, 'vault status >/dev/null']
+    interval: 10s
+    timeout: 5s
+    retries: 6
+
+services:
+  vault-1:
+    <<: *vault
+    ports:
+      - 192.168.10.191:8200:8200
+      - 192.168.10.191:8201:8201
+    volumes:
+      - vault-1-data:/vault/file
+      - ./config/vault-1.hcl:/vault/config/vault.hcl:ro
+      - ./start.sh:/etc/vault/start.sh:ro
+    networks:
+      exch:
+        aliases: [vault]
+
+  vault-2:
+    <<: *vault
+    ports:
+      - 192.168.10.191:8210:8200
+      - 192.168.10.191:8211:8201
+    volumes:
+      - vault-2-data:/vault/file
+      - ./config/vault-2.hcl:/vault/config/vault.hcl:ro
+      - ./start.sh:/etc/vault/start.sh:ro
+    networks:
+      exch:
+        aliases: [vault]
+
+secrets:
+  transit-token:
+    file: ./secrets/transit-token
+
+volumes:
+  vault-1-data:
+  vault-2-data:
+
+# Applications on exch join this network and use http://vault:8200.
+networks:
+  exch:
+    name: exch

infra/prod/vault/gamma.yml

@@ -0,0 +1,34 @@
+# Production Vault on infra-gamma: node 3 of three (#74). It keeps a
+# copy of the data off exch; nodes 1 and 2 run on exch (exch.yml).
+name: exch-vault
+
+services:
+  vault-3:
+    image: ${VAULT_IMAGE:?Run make prod-images and copy images.env}
+    restart: unless-stopped
+    cap_add: [IPC_LOCK]
+    ulimits: {memlock: {soft: -1, hard: -1}}
+    environment:
+      VAULT_ADDR: http://127.0.0.1:8200
+      SKIP_SETCAP: 'true'
+    entrypoint: [/bin/sh, /etc/vault/start.sh]
+    secrets: [transit-token]
+    healthcheck:
+      test: [CMD-SHELL, 'vault status >/dev/null']
+      interval: 10s
+      timeout: 5s
+      retries: 6
+    ports:
+      - 192.168.10.98:8200:8200
+      - 192.168.10.98:8201:8201
+    volumes:
+      - vault-3-data:/vault/file
+      - ./config/vault-3.hcl:/vault/config/vault.hcl:ro
+      - ./start.sh:/etc/vault/start.sh:ro
+
+secrets:
+  transit-token:
+    file: ./secrets/transit-token
+
+volumes:
+  vault-3-data:

infra/prod/vault/start.sh

@@ -0,0 +1,8 @@
+#!/bin/sh
+# Hands the transit seal token to the Vault server process only: it is read
+# from the secret file here, so it is not part of the container environment.
+set -eu
+VAULT_TOKEN=$(cat /run/secrets/transit-token)
+[ -n "$VAULT_TOKEN" ] || { echo 'Empty transit seal token' >&2; exit 1; }
+export VAULT_TOKEN
+exec docker-entrypoint.sh vault server -config=/vault/config/vault.hcl