Deploy the production stacks from Woodpecker one stack at a time #76

New Issue

2026-10-30T12:52:06Z

protuberanets commented

2026-10-30 12:52:06 +00:00

Goal

The SXCHNGE production stacks on exch and infra-gamma (Vault, Mongo, NATS, the applications) are deployed from this repository by Woodpecker, one stack at a time: a manual "Run pipeline" with STACK=<name> in the Woodpecker UI or make deploy-infra-stack STACK=<name> on an operator's machine. The first rollout of each component (#74 for Vault) is done by hand from the same files; this task makes every later rollout and update a button.

Approach

Pipeline in .woodpecker/ that runs on manual events with STACK selecting one directory under infra/prod/; an empty or unknown value prints the list and does nothing. Application releases keep their own tag-driven pipeline.
The pipeline resolves the image digests the way make prod-images does, copies the stack's files to /srv/sxchnge/<component> on the target servers and runs docker compose up -d there. Same pattern as the platform pipeline (protuberanets/infra#62); reuse its conventions and the SSH deploy key approach decided there.
CI secrets are served by the Woodpecker Vault secret extension; whether they live in the platform Vault or in the SXCHNGE Vault is an open question for tech.
Clustered stacks (Vault, Mongo, NATS) are rolled one node at a time with a health check between nodes; the pipeline refuses to recreate all nodes at once.
make deploy-infra-stack STACK=<name> starts the pipeline through the Woodpecker API with the operator's personal token.

Acceptance criteria

"Run pipeline" with STACK=vault redeploys the three Vault nodes one at a time; each node unseals and rejoins before the next one is touched.
An empty or unknown STACK lists the stacks and changes nothing.
make deploy-infra-stack STACK=vault starts the same pipeline and prints its URL.
The pipeline holds no Vault credential as a Woodpecker secret.
A redeploy with unchanged image digests changes no container; a changed digest is reported in the log.
infra/prod/README.md documents the button, the make target and the stacks.

Notes

Depends on protuberanets/infra#62 (platform CI) (same pipeline pattern, SSH deploy keys, Vault secret extension) and on #74 for the first Vault rollout.
Open: location of SXCHNGE CI secrets (platform Vault or SXCHNGE Vault); one extension instance serves one Vault address.
Mongo and NATS stacks are added to the pipeline when their tasks exist.

### Goal The SXCHNGE production stacks on exch and infra-gamma (Vault, Mongo, NATS, the applications) are deployed from this repository by Woodpecker, one stack at a time: a manual "Run pipeline" with `STACK=<name>` in the Woodpecker UI or `make deploy-infra-stack STACK=<name>` on an operator's machine. The first rollout of each component (#74 for Vault) is done by hand from the same files; this task makes every later rollout and update a button. ### Approach 1. Pipeline in `.woodpecker/` that runs on manual events with `STACK` selecting one directory under `infra/prod/`; an empty or unknown value prints the list and does nothing. Application releases keep their own tag-driven pipeline. 2. The pipeline resolves the image digests the way `make prod-images` does, copies the stack's files to `/srv/sxchnge/<component>` on the target servers and runs `docker compose up -d` there. Same pattern as the platform pipeline (protuberanets/infra#62); reuse its conventions and the SSH deploy key approach decided there. 3. CI secrets are served by the Woodpecker Vault secret extension; whether they live in the platform Vault or in the SXCHNGE Vault is an open question for tech. 4. Clustered stacks (Vault, Mongo, NATS) are rolled one node at a time with a health check between nodes; the pipeline refuses to recreate all nodes at once. 5. `make deploy-infra-stack STACK=<name>` starts the pipeline through the Woodpecker API with the operator's personal token. ### Acceptance criteria - [ ] "Run pipeline" with `STACK=vault` redeploys the three Vault nodes one at a time; each node unseals and rejoins before the next one is touched. - [ ] An empty or unknown `STACK` lists the stacks and changes nothing. - [ ] `make deploy-infra-stack STACK=vault` starts the same pipeline and prints its URL. - [ ] The pipeline holds no Vault credential as a Woodpecker secret. - [ ] A redeploy with unchanged image digests changes no container; a changed digest is reported in the log. - [ ] `infra/prod/README.md` documents the button, the make target and the stacks. ### Notes - Depends on protuberanets/infra#62 (platform CI) (same pipeline pattern, SSH deploy keys, Vault secret extension) and on #74 for the first Vault rollout. - Open: location of SXCHNGE CI secrets (platform Vault or SXCHNGE Vault); one extension instance serves one Vault address. - Mongo and NATS stacks are added to the pipeline when their tasks exist.

protuberanets added the backend label 2026-10-30 12:52:06 +00:00

protuberanets self-assigned this 2026-10-30 12:52:06 +00:00

protuberanets added this to the SXCHNGE project 2026-10-30 12:52:06 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: tech/sxchnge#76