Run the SXCHNGE production Vault on exch and infra-gamma #74

New Issue

2026-10-29T16:37:02Z

protuberanets commented

2026-10-29 16:37:02 +00:00

Goal

Run the SXCHNGE production Vault as a three-node raft cluster: two nodes on exch (the SXCHNGE prod server) and one on infra-gamma. It is the first production component: Mongo, NATS and the applications get their secrets and dynamic Mongo credentials from it, so it has to exist and survive the loss of one server before anything else is deployed. Losing infra-gamma keeps the majority on exch; losing one node on exch keeps two of three.

Approach

Deployment code in this repository under infra/prod/, deployed by this repository's CI later (separate task); the first rollout is run by hand from the same files. Plain Docker Compose on each server, no Swarm: one Compose project on exch with two Vault services, one on infra-gamma with the third.
Raft integrated storage, retry_join between the three nodes. Each node publishes its API and cluster ports on the server's private address only (exch 192.168.10.191, infra-gamma 192.168.10.98); the two nodes on exch use different host ports. No internal TLS on the private network: raft cluster traffic on the cluster port is already encrypted by Vault itself.
Auto-unseal with the transit seal on the platform Vault (https://vault.sendico.io): a dedicated transit key and a policy that allows only encrypt/decrypt with that key, and a periodic orphan token for the SXCHNGE Vault. Recovery keys from vault operator init are kept by people, never on the servers or in the repository.
Image: the latest stable hashicorp/vault release, digest resolved at deploy time and recorded in the deploy output, as the repository rules require; nothing generated is committed.
Applications on exch reach Vault through one Docker network alias shared by the two local nodes.
Rollout: create the transit key and policy on the platform Vault, start the three nodes, vault operator init with recovery keys, then restart each node in turn and check that it unseals by itself and the raft index matches.

Out of scope here (separate tasks): KV/AppRole/policies and the Mongo database plugin, Mongo and NATS clusters, CI deployment, raft snapshots to SILO, monitoring and alerts.

Acceptance criteria

vault operator raft list-peers shows three voters: two on exch, one on infra-gamma.
After docker restart of any single node it unseals without a person and rejoins with the same raft committed index.
Stopping the infra-gamma node leaves the cluster writable from exch; starting it again rejoins it.
Vault ports listen only on the private addresses; nothing is reachable from the internet.
The transit token can only encrypt/decrypt with its own key on the platform Vault.
Recovery keys and the root token are not stored on any server, in the repository or in CI; the root token is revoked after bootstrap.
infra/prod/ documents the layout, ports, the rollout and how to replace a node.

Notes

Placement follows the agreed target scheme (SXCHNGE Vault 2 on exch + 1 on infra-gamma, the same rule as Vault/Mongo/NATS elsewhere: the majority where the consumers run).
Dependency on the platform Vault: if it is unreachable while a node restarts, that node stays sealed until the platform Vault is back or the node is unsealed with the recovery keys. Running nodes are not affected.
Firewall (OpenStack security groups, owned by the lead): exch <-> infra-gamma on the Vault cluster port; exch -> vault.sendico.io 443. To request before the rollout.
The public entry point and the domain for SXCHNGE are still being decided; Vault does not need them.
Mongo dynamic credentials (database/creds/*, 1 h TTL) make Vault availability a hard dependency of the applications; this is why it is deployed 2+1 first.

### Goal Run the SXCHNGE production Vault as a three-node raft cluster: two nodes on `exch` (the SXCHNGE prod server) and one on `infra-gamma`. It is the first production component: Mongo, NATS and the applications get their secrets and dynamic Mongo credentials from it, so it has to exist and survive the loss of one server before anything else is deployed. Losing `infra-gamma` keeps the majority on `exch`; losing one node on `exch` keeps two of three. ### Approach 1. Deployment code in this repository under `infra/prod/`, deployed by this repository's CI later (separate task); the first rollout is run by hand from the same files. Plain Docker Compose on each server, no Swarm: one Compose project on `exch` with two Vault services, one on `infra-gamma` with the third. 2. Raft integrated storage, `retry_join` between the three nodes. Each node publishes its API and cluster ports on the server's private address only (exch 192.168.10.191, infra-gamma 192.168.10.98); the two nodes on `exch` use different host ports. No internal TLS on the private network: raft cluster traffic on the cluster port is already encrypted by Vault itself. 3. Auto-unseal with the `transit` seal on the platform Vault (`https://vault.sendico.io`): a dedicated transit key and a policy that allows only `encrypt`/`decrypt` with that key, and a periodic orphan token for the SXCHNGE Vault. Recovery keys from `vault operator init` are kept by people, never on the servers or in the repository. 4. Image: the latest stable `hashicorp/vault` release, digest resolved at deploy time and recorded in the deploy output, as the repository rules require; nothing generated is committed. 5. Applications on `exch` reach Vault through one Docker network alias shared by the two local nodes. 6. Rollout: create the transit key and policy on the platform Vault, start the three nodes, `vault operator init` with recovery keys, then restart each node in turn and check that it unseals by itself and the raft index matches. Out of scope here (separate tasks): KV/AppRole/policies and the Mongo database plugin, Mongo and NATS clusters, CI deployment, raft snapshots to SILO, monitoring and alerts. ### Acceptance criteria - [ ] `vault operator raft list-peers` shows three voters: two on `exch`, one on `infra-gamma`. - [ ] After `docker restart` of any single node it unseals without a person and rejoins with the same raft committed index. - [ ] Stopping the `infra-gamma` node leaves the cluster writable from `exch`; starting it again rejoins it. - [ ] Vault ports listen only on the private addresses; nothing is reachable from the internet. - [ ] The transit token can only encrypt/decrypt with its own key on the platform Vault. - [ ] Recovery keys and the root token are not stored on any server, in the repository or in CI; the root token is revoked after bootstrap. - [ ] `infra/prod/` documents the layout, ports, the rollout and how to replace a node. ### Notes - Placement follows the agreed target scheme (SXCHNGE Vault 2 on exch + 1 on infra-gamma, the same rule as Vault/Mongo/NATS elsewhere: the majority where the consumers run). - Dependency on the platform Vault: if it is unreachable while a node restarts, that node stays sealed until the platform Vault is back or the node is unsealed with the recovery keys. Running nodes are not affected. - Firewall (OpenStack security groups, owned by the lead): exch <-> infra-gamma on the Vault cluster port; exch -> vault.sendico.io 443. To request before the rollout. - The public entry point and the domain for SXCHNGE are still being decided; Vault does not need them. - Mongo dynamic credentials (`database/creds/*`, 1 h TTL) make Vault availability a hard dependency of the applications; this is why it is deployed 2+1 first.

protuberanets added the backend label 2026-10-29 16:37:02 +00:00

protuberanets self-assigned this 2026-10-29 16:37:02 +00:00

protuberanets added this to the SXCHNGE project 2026-10-29 16:37:02 +00:00

protuberanets added reference prod-vault-74

2026-10-29 16:37:14 +00:00

protuberanets referenced this issue from a commit

2026-10-29 17:00:35 +00:00

Run the production Vault as three raft nodes on exch and infra-gamma

protuberanets referenced this issue

2026-10-29 17:00:46 +00:00

Run the production Vault as three raft nodes on exch and infra-gamma #75

protuberanets referenced this issue