Infrastructure Management Checklist #312

New issue

Open

opened 2026-06-19 09:19:57 +00:00 by ryansquared · 1 comment

ryansquared commented

2026-06-19 09:19:57 +00:00

(Migrated from codeberg.org)

This is a living list of required infrastructure upgrades:

Standalone Components:

ECR repositories for enclave components (execd, locksmith, bootproof, etc.)
ECR repositories for platform components (api, email, metering, etc.)
S3 bucket for managing database backups
RDS instance, if we confirm we don't want it running in-cluster
Forgejo Actions runner machine and CI system, to build said components on-push
- Runner machine must have git verify-commit baked in. Might require a custom version of the runner.

Infrastructure should be deployed using Tofu with either an in-repository encrypted state, or a one-time ClickOps'd bucket. Non-cluster machines should be running Debian with Ansible configurations. Future upgrades to stagex-based single-use images can be considered later.

Cluster Components

Generic Kubernetes necessities, such as cert-manager, traefik ingress, external DNS, and whatever else is necessary
Mimir, Loki, etc. Grafana monitoring stack, with dashboards and alerts monitoring:
- HTTP 4xx/5xx rates
- Persistent volume usage
- Amount of deployed resources versus capacity (we'd need a custom metric for this, I imagine)
- In-cluster failed compute resources
Platform components running in-cluster
- No more make up on the boxen!
CronJob in the cluster, to automate database backups to aforementioned offsite S3 bucket

Applications should be deployed using Kustomize into their own namespaces. Secrets should be managed using ksops. Follow guidelines in https://git.distrust.co/public/stack for a reasonable base to get started, but we should also start building out kustomizations to be reusable for separate environments, so we can each have our own testing infrastructure without stepping on each others' toes.

Edit as necessary to add further components.

This is a living list of required infrastructure upgrades: ## Standalone Components: * ECR repositories for enclave components (execd, locksmith, bootproof, etc.) * ECR repositories for platform components (api, email, metering, etc.) * S3 bucket for managing database backups * RDS instance, if we confirm we don't want it running in-cluster * Forgejo Actions runner machine and CI system, to build said components on-push * Runner machine must have `git verify-commit` baked in. Might require a custom version of the runner. Infrastructure should be deployed using Tofu with either an in-repository encrypted state, or a one-time ClickOps'd bucket. Non-cluster machines should be running Debian with Ansible configurations. Future upgrades to stagex-based single-use images can be considered later. ## Cluster Components * Generic Kubernetes necessities, such as cert-manager, traefik ingress, external DNS, and whatever else is necessary * Mimir, Loki, etc. Grafana monitoring stack, with dashboards and alerts monitoring: * HTTP 4xx/5xx rates * Persistent volume usage * Amount of deployed resources versus capacity (we'd need a custom metric for this, I imagine) * In-cluster failed compute resources * Platform components running in-cluster * No more `make up` on the boxen! * CronJob in the cluster, to automate database backups to aforementioned offsite S3 bucket Applications should be deployed using Kustomize into their own namespaces. Secrets should be managed using ksops. Follow guidelines in https://git.distrust.co/public/stack for a _reasonable_ base to get started, but we should also start building out kustomizations to be reusable for separate environments, so we can each have our own testing infrastructure without stepping on each others' toes. --- Edit as necessary to add further components.

vkobel commented

2026-06-25 13:23:37 +00:00

(Migrated from codeberg.org)

ECR pull-through cache for stagex/* so base layers come from inside AWS, not Docker Hub over www

- ECR pull-through cache for stagex/* so base layers come from inside AWS, not Docker Hub over www

anton referenced this issue

2026-10-22 23:44:10 +00:00

Add setup to backup database and schedule weekly checks to ensure backups are valid #171

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

public/platform#312

No description provided.

Rows
Columns