sxchnge/infra/prod/README.md

Production

Production runs on two servers of the shared private network: exch (192.168.10.191), the SXCHNGE server, and infra-gamma (192.168.10.98), which keeps the third copy of the data. Each component runs two nodes on exch and one on infra-gamma: losing infra-gamma keeps the majority, losing one node on exch keeps two of three. Every server runs plain Docker Compose; files are deployed to /srv/sxchnge/<component> on the server.

Ports are published on the private addresses only. Traffic between the servers is not wrapped in TLS: the network is private and limited by security groups, and every component authenticates its peers.

Vault

#74. Three raft nodes:

Node	Server	API	Cluster	Compose
sxchnge-vault-1	exch	192.168.10.191:8200	:8201	vault/exch.yml
sxchnge-vault-2	exch	192.168.10.191:8210	:8211	vault/exch.yml
sxchnge-vault-3	infra-gamma	192.168.10.98:8200	:8201	vault/gamma.yml

Containers on exch join the Docker network sxchnge and use http://vault:8200; the alias belongs to both local nodes. A standby node forwards requests to the active one, so the leader may be on any server.

Vault is not published to the internet. Its UI is opened through an SSH tunnel from an administrator's machine, for example ssh -N -L 8200:192.168.10.191:8200 exch and then http://localhost:8200/ui.

Nodes unseal themselves with the transit seal on the platform Vault (https://vault.sendico.io, key transit/keys/sxchnge-vault-unseal). Each server has its own periodic token in secrets/transit-token (root, mode 600, not in Git) that may only encrypt and decrypt with that key. If the platform Vault is unreachable while a node starts, the node stays sealed until it is reachable again; running nodes are not affected. vault operator init returns recovery keys, not unseal keys: they authorize operations such as generating a new root token but cannot unseal a node. Only the transit key can. Its backup (transit/backup/sxchnge-vault-unseal), the recovery keys and the root token are kept outside the servers by their holders; see "Platform Vault lost".

make prod-images resolves the latest stable Vault release and pins its digest in ignored .local/prod/images.env. All three nodes get the same file: a cluster runs one build, and an upgrade is a separate rollout of a new file, one node at a time.

First rollout

1. On the platform Vault, with an administrator token:

   vault secrets list | grep -q '^transit/' || vault secrets enable transit
   vault write transit/keys/sxchnge-vault-unseal exportable=true allow_plaintext_backup=true
   vault read -field=backup transit/backup/sxchnge-vault-unseal   # store with the holders, then clear the terminal
   vault policy write sxchnge-vault-unseal - <<'EOF'
   path "transit/encrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
   path "transit/decrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
   EOF
   

   Create one token per server and write it straight into that server's /srv/sxchnge/vault/secrets/transit-token, never to a terminal or a log:

   vault token create -orphan -period=768h -policy=sxchnge-vault-unseal \
     -display-name=sxchnge-vault-<server> -field=token
   

2. Copy vault/ and .local/prod/images.env to /srv/sxchnge/vault on both servers.

3. Start the nodes: on exch docker compose --env-file images.env -f exch.yml up -d, on infra-gamma the same with gamma.yml.

4. Initialize once, on exch: docker exec -it sxchnge-vault-vault-1-1 vault operator init -recovery-shares=5 -recovery-threshold=3. The output is shown once; store the recovery keys and the root token with their holders. Revoke the root token after the bootstrap; a new one is generated with the recovery keys when needed.

5. Check: vault operator raft list-peers lists three voters; docker restart of each node in turn unseals it without a person and the raft committed index matches on all three.

Replacing a node

Stop the node, remove it with vault operator raft remove-peer <node id>, delete its volume and start it again: it joins through retry_join and copies the data from the leader. A node keeps its node id and addresses; moving it to another server is a change of its config file and of every retry_join list.

Platform Vault lost

Running nodes keep working. To start a node without the platform Vault, start any temporary Vault, enable transit, load the stored key backup with vault write transit/restore/sxchnge-vault-unseal backup=<backup>, issue a token with the policy above, and point address in the node configs at that Vault. Without the key backup the data cannot be unsealed again.