protuberanets
f436b2528c
Two nodes on exch and one on infra-gamma, plain Compose per server, ports on the private addresses only. Nodes unseal through the transit seal on the platform Vault; the transit token reaches only the server process. make prod-images resolves the latest stable Vault release once and pins its digest in ignored .local/prod/images.env, so every node runs the same build. infra/prod/README.md records the layout, the first rollout, node replacement and recovery when the platform Vault is lost (#74).
94 lines
4.5 KiB
Markdown
94 lines
4.5 KiB
Markdown
# Production
|
|
|
|
Production runs on two servers of the shared private network: `exch`
|
|
(192.168.10.191), the SXCHNGE server, and `infra-gamma` (192.168.10.98), which
|
|
keeps the third copy of the data. Each component runs two nodes on `exch` and
|
|
one on `infra-gamma`: losing `infra-gamma` keeps the majority, losing one node on
|
|
`exch` keeps two of three. Every server runs plain Docker Compose; files are
|
|
deployed to `/srv/sxchnge/<component>` on the server.
|
|
|
|
Ports are published on the private addresses only. Traffic between the servers
|
|
is not wrapped in TLS: the network is private and limited by security groups,
|
|
and every component authenticates its peers.
|
|
|
|
## Vault
|
|
|
|
[#74](https://git.sendico.io/tech/sxchnge/issues/74). Three raft nodes:
|
|
|
|
| Node | Server | API | Cluster | Compose |
|
|
| --- | --- | --- | --- | --- |
|
|
| `sxchnge-vault-1` | exch | 192.168.10.191:8200 | :8201 | `vault/exch.yml` |
|
|
| `sxchnge-vault-2` | exch | 192.168.10.191:8210 | :8211 | `vault/exch.yml` |
|
|
| `sxchnge-vault-3` | infra-gamma | 192.168.10.98:8200 | :8201 | `vault/gamma.yml` |
|
|
|
|
Containers on `exch` join the Docker network `sxchnge` and use
|
|
`http://vault:8200`; the alias belongs to both local nodes. A standby node
|
|
forwards requests to the active one, so the leader may be on any server.
|
|
|
|
Nodes unseal themselves with the `transit` seal on the platform Vault
|
|
(`https://vault.sendico.io`, key `transit/keys/sxchnge-vault-unseal`). Each server
|
|
has its own periodic token in `secrets/transit-token` (root, mode 600, not in Git)
|
|
that may only encrypt and decrypt with that key. If the platform Vault is
|
|
unreachable while a node starts, the node stays sealed until it is reachable
|
|
again; running nodes are not affected. `vault operator init` returns recovery
|
|
keys, not unseal keys: they authorize operations such as generating a new root
|
|
token but cannot unseal a node. Only the transit key can. Its backup
|
|
(`transit/backup/sxchnge-vault-unseal`), the recovery keys and the root token are
|
|
kept outside the servers by their holders; see "Platform Vault lost".
|
|
|
|
`make prod-images` resolves the latest stable Vault release and pins its digest in
|
|
ignored `.local/prod/images.env`. All three nodes get the same file: a cluster
|
|
runs one build, and an upgrade is a separate rollout of a new file, one node at a
|
|
time.
|
|
|
|
### First rollout
|
|
|
|
1. On the platform Vault, with an administrator token:
|
|
|
|
```sh
|
|
vault secrets list | grep -q '^transit/' || vault secrets enable transit
|
|
vault write transit/keys/sxchnge-vault-unseal exportable=true allow_plaintext_backup=true
|
|
vault read -field=backup transit/backup/sxchnge-vault-unseal # store with the holders, then clear the terminal
|
|
vault policy write sxchnge-vault-unseal - <<'EOF'
|
|
path "transit/encrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
|
|
path "transit/decrypt/sxchnge-vault-unseal" { capabilities = ["update"] }
|
|
EOF
|
|
```
|
|
|
|
Create one token per server and write it straight into that server's
|
|
`/srv/sxchnge/vault/secrets/transit-token`, never to a terminal or a log:
|
|
|
|
```sh
|
|
vault token create -orphan -period=768h -policy=sxchnge-vault-unseal \
|
|
-display-name=sxchnge-vault-<server> -field=token
|
|
```
|
|
|
|
2. Copy `vault/` and `.local/prod/images.env` to `/srv/sxchnge/vault` on both
|
|
servers.
|
|
3. Start the nodes: on `exch`
|
|
`docker compose --env-file images.env -f exch.yml up -d`, on `infra-gamma`
|
|
the same with `gamma.yml`.
|
|
4. Initialize once, on `exch`:
|
|
`docker exec -it sxchnge-vault-vault-1-1 vault operator init -recovery-shares=5 -recovery-threshold=3`.
|
|
The output is shown once; store the recovery keys and the root token with
|
|
their holders. Revoke the root token after the bootstrap; a new one is
|
|
generated with the recovery keys when needed.
|
|
5. Check: `vault operator raft list-peers` lists three voters; `docker restart`
|
|
of each node in turn unseals it without a person and the raft committed index
|
|
matches on all three.
|
|
|
|
### Replacing a node
|
|
|
|
Stop the node, remove it with `vault operator raft remove-peer <node id>`,
|
|
delete its volume and start it again: it joins through `retry_join` and copies the
|
|
data from the leader. A node keeps its node id and addresses; moving it to
|
|
another server is a change of its config file and of every `retry_join` list.
|
|
|
|
### Platform Vault lost
|
|
|
|
Running nodes keep working. To start a node without the platform Vault, start
|
|
any temporary Vault, enable `transit`, load the stored key backup with
|
|
`vault write transit/restore/sxchnge-vault-unseal backup=<backup>`, issue a token
|
|
with the policy above, and point `address` in the node configs at that Vault.
|
|
Without the key backup the data cannot be unsealed again.
|