Workload clusters

A workload cluster runs Products' zones. The management cluster stays in charge: Argo CD on the management cluster deploys to every workload cluster, from the gitops repo. Registering, removing and building clusters is for platform admins.

What happens when you register one:

1. Connected. Infrared reaches the cluster's API server and reads its Kubernetes version and node count.
2. Argo CD. For a cluster reached by kubeconfig, Infrared writes the Argo CD cluster Secret cluster-<name> in namespace argocd, so Argo CD can deploy to it.
3. Registered. Infrared opens and merges a pull request in the platform org's gitops repo that adds two files:
   • registry/clusters/<management cluster>/components/cluster-<name>.yaml: the app-of-apps registry-<name> (sync wave 50, AppProject default), which syncs every Application in registry/clusters/<name>/components.
   • registry/clusters/<name>/components/README.md, which explains the directory.

The Cluster's conditions Connected and Registered show each step, and the Clusters page in the UI shows both, with the Argo CD server and the gitops path.

Register a cluster

Infrared reaches a workload cluster in one of two ways.

Connection	When to use it	What Infrared does
inCluster	Zones should run on the management cluster itself, but you want them managed as a separate cluster (for example to allowlist it for one org).	Uses its own service account. Argo CD uses its built-in in-cluster destination, https://kubernetes.default.svc, so no cluster Secret is written.
kubeconfigSecretRef	Any other cluster.	Reads the kubeconfig from a Secret in namespace infrared, using its current context.

The kubeconfig's user must carry a bearer token or a client certificate inline. Exec plugins (such as aws eks get-token), auth-provider plugins and token files are refused, because Argo CD needs a credential it can store. A service account token bound to a ClusterRole that can deploy your zones works.

In the UI. Open Clusters, fill in Register a workload cluster: a name (lowercase letters, numbers and hyphens, at most 42 characters) and either Kubeconfig (paste it) or The management cluster itself.

Through the API.

curl -X POST https://<host>/api/v1/clusters \
  -H "Authorization: Bearer $INFRARED_TOKEN" -H 'Content-Type: application/json' \
  -d "$(jq -n --arg kc "$(cat edge.kubeconfig)" \
        '{name: "edge-us-east-1", kubeconfig: $kc, flavor: "k3s", provider: "aws", region: "us-east-1"}')"

Send "inCluster": true instead of kubeconfig for the management cluster itself. Infrared stores a kubeconfig in the Secret cluster-<name>-kubeconfig (key kubeconfig) in namespace infrared and never returns it.

As a resource.

apiVersion: infrared.darkshift.io/v1alpha1
kind: Cluster
metadata:
  name: edge-us-east-1
spec:
  type: workload
  flavor: k3s
  provider: aws
  region: us-east-1
  connection:
    kubeconfigSecretRef:
      name: edge-us-east-1-kubeconfig   # in namespace infrared
      key: kubeconfig

The Cluster becomes the owner of its kubeconfig Secret and its Argo CD cluster Secret, so removing the Cluster removes both.

Watch it:

kubectl get cluster edge-us-east-1 -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'

Set spec.server when Argo CD must reach the API server at a different address than the kubeconfig's.

Build a cluster with OpenTofu

A Cluster can build itself from a cluster template: Infrared runs the template's OpenTofu module in a Job on the management cluster.

apiVersion: infrared.darkshift.io/v1alpha1
kind: Cluster
metadata:
  name: edge-us-east-1
spec:
  type: workload
  flavor: k3s
  provider: aws
  region: us-east-1
  template:
    module: aws/k3s-node
    version: v0.2.3
  provision:
    action: plan                 # plan (the default) or apply
    variables:
      name: edge-us-east-1
      region: us-east-1
    credentialsSecretRef:
      name: edge-us-east-1-aws   # in namespace infrared

Field	Meaning
provision.action	plan shows what would be built. apply builds it. Default plan.
provision.source	The module source. Empty means git::https://github.com/darkshiftio/infrared-iac-modules//<template.module>?ref=<template.version>.
provision.variables	The module's inputs. A value that starts with [ or { is passed as JSON (a list or object); anything else is a string OpenTofu converts.
provision.credentialsSecretRef	A Secret in namespace infrared whose keys become the Job's environment, such as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION. Add GIT_TOKEN to read a private module source on GitHub. Only the name is read.
provision.image	The OpenTofu image. Default ghcr.io/opentofu/opentofu:1.10.

Infrared runs one Job per Cluster generation and action, tofu-<cluster>-<generation>-<action>, in namespace infrared with the service account infrared-tofu. The Job doesn't retry and is deleted a week after it finishes. OpenTofu keeps its state in the management cluster with the kubernetes backend: a Secret in namespace infrared with the suffix cluster-<name>. status.provision shows the Job, the action, its phase (Running, Succeeded or Failed) and how to read its logs:

kubectl -n infrared logs job/tofu-edge-us-east-1-1-plan

Read the plan, then change action to apply. Changing the spec is a new generation, so it starts a new Job.

:::caution Apply creates billable resources apply creates real cloud resources in the account the credentials belong to, and they cost money until you destroy them. Removing the Cluster does not destroy them. :::

When the apply succeeds, put the new cluster's kubeconfig in a Secret and set spec.connection.kubeconfigSecretRef; registration then continues as above. The API accepts template and provision on POST /v1/clusters too, without a connection. The UI's register form doesn't offer them yet.

Place zones on a cluster

A platform admin allows an org to use the cluster: tick the org under Orgs allowed on the cluster's panel in Clusters, or replace the org's allowlist with the API:

curl -X PUT -H "Authorization: Bearer $IR_TOKEN" -H 'Content-Type: application/json' \
  -d '{"clusters":["edge-us-east-1"]}' https://infrared.example.com/api/v1/orgs/acme/cluster-allowlist

Then the org sets cluster on the zone in its Product's spec.delivery.zones:

spec:
  delivery:
    zones:
      - name: checkout-rc
        environment: pre-release
      - name: checkout-prod
        environment: release
        cluster: edge-us-east-1

A zone without cluster runs on the gitops repo's management cluster. For a zone with one, a Release checks the cluster before it promotes:

The cluster is	The Release
Not on the org's cluster allowlist	Fails for that zone.
Not a Cluster Infrared knows	Fails for that zone.
Known but not Registered yet	Waits, and checks again every minute.
Registered	Promotes.

The zone's Application lives in that cluster's registry directory: registry/clusters/<zone cluster>/components/product-<zone>.yaml (set applicationPath on the zone to use another path). Point its destination at the cluster by name:

spec:
  project: products
  destination:
    name: edge-us-east-1
    namespace: checkout-prod

For an inCluster workload cluster, use server: https://kubernetes.default.svc instead.

:::note The AppProject must allow the destination The Application's AppProject decides which clusters and namespaces it may deploy to. The products AppProject from Deliver a single app allows only the management cluster, so add the workload cluster and the zone's namespace to its destinations, or Argo CD refuses to sync the Application. :::

The Clusters page lists each cluster's zones and the orgs that may use it.

Remove a cluster

Remove on the cluster's panel, or:

curl -X DELETE https://<host>/api/v1/clusters/edge-us-east-1 -H "Authorization: Bearer $INFRARED_TOKEN"

Kubernetes then removes the Argo CD cluster Secret and the kubeconfig Secret. Zones on the cluster stop deploying. The app-of-apps file stays in the gitops repo until you remove it with a pull request, and nothing on the cluster itself is deleted. The management cluster can't be removed.

Not yet

• Crossplane provider-terraform Workspaces as the provisioning path.
• Node access through SSM and Argo CD cluster Secrets delivered by ExternalSecret.
• Allowlist changes from the UI or API; today they're an edit to the Organization.
• Mapping org teams to Kubernetes RBAC on workload clusters.
• GCP and Linode cluster templates.

See the roadmap.