# Production

Production runs on two servers of the shared private network: `exch`
(192.168.10.191), the SXCHNGE server, and `infra-gamma` (192.168.10.98), which
keeps the third copy of the data. Each component runs two nodes on `exch` and
one on `infra-gamma`: losing `infra-gamma` keeps the majority, losing one node on
`exch` keeps two of three. Every server runs plain Docker Compose; files are
deployed to `/srv/sxchnge/<component>` on the server.

Ports are published on the private addresses only. Traffic between the servers
is not wrapped in TLS: the network is private and limited by security groups,
and every component authenticates its peers.

## Vault

[#74](https://git.sendico.io/tech/sxchnge/issues/74). Three raft nodes:

| Node | Server | API | Cluster | Compose |
| --- | --- | --- | --- | --- |
| `sxchnge-vault-1` | exch | 192.168.10.191:8200 | :8201 | `vault/exch.yml` |
| `sxchnge-vault-2` | exch | 192.168.10.191:8210 | :8211 | `vault/exch.yml` |
| `sxchnge-vault-3` | infra-gamma | 192.168.10.98:8200 | :8201 | `vault/gamma.yml` |

Containers on `exch` join the Docker network `sxchnge` and use
`http://vault:8200`; the alias belongs to both local nodes. A standby node
forwards requests to the active one, so the leader may be on any server.

Nodes unseal themselves with the `transit` seal on the platform Vault
(`https://vault.sendico.io`, key `transit/keys/sxchnge-vault-unseal`). Each server
has its own periodic token in `secrets/transit-token` (root, mode 600, not in Git)
that may only encrypt and decrypt with that key. If the platform Vault is
unreachable while a node starts, the node stays sealed until it is reachable
again; running nodes are not affected. `vault operator init` returns recovery
keys, not unseal keys: they authorize operations such as generating a new root
token but cannot unseal a node. Only the transit key can. Its backup
(`transit/backup/sxchnge-vault-unseal`), the recovery keys and the root token are
kept outside the servers by their holders; see "Platform Vault lost".

`make prod-images` resolves the latest stable Vault release and pins its digest in
ignored `.local/prod/images.env`. All three nodes get the same file: a cluster
runs one build, and an upgrade is a separate rollout of a new file, one node at a
time.

### First rollout

1. On the platform Vault, with an administrator token:

   ```sh
   vault secrets list | grep -q '^transit/' || vault secrets enable transit
   vault write transit/keys/sxchnge-vault-unseal exportable=true allow_plaintext_backup=true
   vault read -field=backup transit/backup/sxchnge-vault-unseal   # store with the holders, then clear the terminal
   vault policy write sxchnge-vault-unseal - <<'EOF'
   path "transit/encrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
   path "transit/decrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
   EOF
   ```

   Create one token per server and write it straight into that server's
   `/srv/sxchnge/vault/secrets/transit-token`, never to a terminal or a log:

   ```sh
   vault token create -orphan -period=768h -policy=sxchnge-vault-unseal \
     -display-name=sxchnge-vault-<server> -field=token
   ```

2. Copy `vault/` and `.local/prod/images.env` to `/srv/sxchnge/vault` on both
   servers.
3. Start the nodes: on `exch`
   `docker compose --env-file images.env -f exch.yml up -d`, on `infra-gamma`
   the same with `gamma.yml`.
4. Initialize once, on `exch`:
   `docker exec -it sxchnge-vault-vault-1-1 vault operator init -recovery-shares=5 -recovery-threshold=3`.
   The output is shown once; store the recovery keys and the root token with
   their holders. Revoke the root token after the bootstrap; a new one is
   generated with the recovery keys when needed.
5. Check: `vault operator raft list-peers` lists three voters; `docker restart`
   of each node in turn unseals it without a person and the raft committed index
   matches on all three.

### Replacing a node

Stop the node, remove it with `vault operator raft remove-peer <node id>`,
delete its volume and start it again: it joins through `retry_join` and copies the
data from the leader. A node keeps its node id and addresses; moving it to
another server is a change of its config file and of every `retry_join` list.

### Platform Vault lost

Running nodes keep working. To start a node without the platform Vault, start
any temporary Vault, enable `transit`, load the stored key backup with
`vault write transit/restore/sxchnge-vault-unseal backup=<backup>`, issue a token
with the policy above, and point `address` in the node configs at that Vault.
Without the key backup the data cannot be unsealed again.