sxchnge/infra/prod/README.md

Production

Production runs on two servers of the shared private network: exch (192.168.10.191), the SXCHNGE server, and infra-gamma (192.168.10.98), which keeps the third copy of the data. Each component runs two nodes on exch and one on infra-gamma: losing infra-gamma keeps the majority, losing one node on exch keeps two of three. Every server runs plain Docker Compose; files are deployed to /srv/sxchnge/<component> on the server.

Ports are published on the private addresses only. Traffic between the servers is not wrapped in TLS: the network is private and limited by security groups, and every component authenticates its peers.

Vault

#74. Three raft nodes:

Node	Server	API	Cluster	Compose
sxchnge-vault-1	exch	192.168.10.191:8200	:8201	vault/exch.yml
sxchnge-vault-2	exch	192.168.10.191:8210	:8211	vault/exch.yml
sxchnge-vault-3	infra-gamma	192.168.10.98:8200	:8201	vault/gamma.yml

Containers on exch join the Docker network sxchnge and use http://vault:8200; the alias belongs to both local nodes. A standby node forwards requests to the active one, so the leader may be on any server.

Nodes unseal themselves with the transit seal on the platform Vault (https://vault.sendico.io, key transit/keys/sxchnge-vault-unseal). Each server has its own periodic token in secrets/transit-token (root, mode 600, not in Git) that may only encrypt and decrypt with that key. If the platform Vault is unreachable while a node starts, the node stays sealed until it is reachable again; running nodes are not affected. vault operator init returns recovery keys, not unseal keys: they authorize operations such as generating a new root token but cannot unseal a node. Only the transit key can. Its backup (transit/backup/sxchnge-vault-unseal), the recovery keys and the root token are kept outside the servers by their holders; see "Platform Vault lost".

make prod-images resolves the latest stable Vault release and pins its digest in ignored .local/prod/images.env. All three nodes get the same file: a cluster runs one build, and an upgrade is a separate rollout of a new file, one node at a time.

First rollout

1. On the platform Vault, with an administrator token:

   vault secrets list | grep -q '^transit/' || vault secrets enable transit
   vault write transit/keys/sxchnge-vault-unseal exportable=true allow_plaintext_backup=true
   vault read -field=backup transit/backup/sxchnge-vault-unseal   # store with the holders, then clear the terminal
   vault policy write sxchnge-vault-unseal - <<'EOF'
   path "transit/encrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
   path "transit/decrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
   EOF
   

   Create one token per server and write it straight into that server's /srv/sxchnge/vault/secrets/transit-token, never to a terminal or a log:

   vault token create -orphan -period=768h -policy=sxchnge-vault-unseal \
     -display-name=sxchnge-vault-<server> -field=token
   

2. Copy vault/ and .local/prod/images.env to /srv/sxchnge/vault on both servers.

3. Start the nodes: on exch docker compose --env-file images.env -f exch.yml up -d, on infra-gamma the same with gamma.yml.

4. Initialize once, on exch: docker exec -it sxchnge-vault-vault-1-1 vault operator init -recovery-shares=5 -recovery-threshold=3. The output is shown once; store the recovery keys and the root token with their holders. Revoke the root token after the bootstrap; a new one is generated with the recovery keys when needed.

5. Check: vault operator raft list-peers lists three voters; docker restart of each node in turn unseals it without a person and the raft committed index matches on all three.

Replacing a node

Stop the node, remove it with vault operator raft remove-peer <node id>, delete its volume and start it again: it joins through retry_join and copies the data from the leader. A node keeps its node id and addresses; moving it to another server is a change of its config file and of every retry_join list.

Platform Vault lost

Running nodes keep working. To start a node without the platform Vault, start any temporary Vault, enable transit, load the stored key backup with vault write transit/restore/sxchnge-vault-unseal backup=<backup>, issue a token with the policy above, and point address in the node configs at that Vault. Without the key backup the data cannot be unsealed again.