Skip to main content
Version: 0.1 (next)

Back up and restore

Infrared backs up the whole install to object storage outside its cluster. Each backup is one encrypted artifact holding Infrared's objects (orgs, Products, AgentRoles, AgentWorkflows, AgentWorkflowRuns and the rest), Gitea's dump when the install runs Gitea, and dumps of the platform's Postgres. Beside it, Infrared mirrors the buckets of the cluster's object store. A backup that a mirror run followed is a point: a restore brings a point back whole, on new servers.

Only platform admins see and change backups.

Set up backups​

Backups need three things before the first one is made: a destination, the destination's key, and at least one recipient to encrypt to. Set them in Settings → Backups, or in the setup wizard's optional Set up backups step, which shows the same form after the gitops repo is hydrated. Skip for now leaves the wizard step for later; the step then reads "Skipped for now: Settings, Backups."

Until all three are set, the panel says No backup is made yet and names what's missing.

Destination​

FieldWhat to enter
ProviderLinode Object Storage
RegionThe region in the bucket's endpoint, such as us-lax-4
BucketThe bucket's name. This install writes only under its prefix
PrefixThe path everything is kept under in the bucket. It defaults to the install's name
Access key ID and Secret access keyA key that can read and write the bucket

The Endpoint follows from the region (https://<region>.linodeobjects.com). The key is write-only: once it's set, the panel says when it was set, never what it is. Replace the access key enters a new pair, and Keep the key set cancels that.

Before saving a destination, Infrared checks it with a real write: it lists the prefix, then writes, reads and deletes one object with the key. Check destination runs the same check on the destination already set. A destination that fails the check isn't saved, and the panel lists each step (list, write, read, delete) with what the store answered.

Through the API or the chart's backup values, a destination can also be Google Cloud Storage (gcs), reached by the identity Infrared runs as through Workload Identity with no key, as on GKE, or another S3-compatible store (s3), such as Amazon S3. The form in Settings sets Linode Object Storage only, for now. See the API's PUT /v1/backup/destination.

Schedule and recipients​

FieldDefaultWhat it does
Backup schedule5 * * * *Five cron fields, in UTC: when a backup is made
Mirror schedule17 * * * *Five cron fields, in UTC: when the object store's buckets are mirrored. A backup becomes a point at the first mirror run after it
Retention7dWhole days, from 1d to 999d. Older backups are deleted; the newest complete one never is
RecipientsnoneUp to 16 age public keys (age1…). Each backup is encrypted to all of them, and any one of their private keys opens it
Archive Postgres's WALoffAlso keeps Postgres's write-ahead log in the backup bucket, beside the dump in each backup. Not available on Google Cloud Storage

Add recipient adds a line, up to 16. Save backups saves the destination and the settings together; a destination is checked first, and only what you changed is sent. Without a recipient, no backup is made. Paste only the public half of a key. If you paste a private key (AGE-SECRET-KEY-…), the form empties the line and says it wasn't kept.

To make a key pair, run age-keygen and keep its output in your vault, or run ir backup key new, which writes the identity to a file only you can read and prints its public half. Keep at least one private key somewhere that survives losing the cluster: without it, no backup can be opened.

The settings can also come from the chart's backup values at install. Infrared copies them into its own setting once, while the setting is empty; after that, change them in Settings → Backups.

See the backups​

Once backups are set, the panel shows What the operator last saw: whether backups are healthy (BackupsHealthy), the newest backup, the newest mirror run, the newest point and the operator's last message.

Backups in the destination lists the newest 24 backups, read straight from the bucket:

ColumnWhat it says
BackupIts stamp, such as 20261006T010500Z, and how long ago it was made
SizeThe artifact's size
PartsWhat it holds: Infrared's objects, Gitea's dump and the Postgres databases dumped
Mirror runThe mirror run that followed it
CompleteWhether the artifact and its manifest are both there and match
PointWhether a restore can bring it back, or why not yet

Back up now asks for a backup at once, followed by a mirror run, which makes it a point. The panel shows Backing up until both are done. It is off while a scheduled backup is running.

Restore an install​

A restore brings an install back from one point onto new servers, before Infrared runs there. You need:

  • a cluster with nothing of Infrared on it, reached by its own kubeconfig context;
  • the install's usual chart values, with the same backup block, so the restore reads the same bucket;
  • the age private key of one of the backup's recipients;
  • the bucket's key.

ir restore does the whole restore:

ir backup list # pick a point
ir restore --backup-artifact 20261006T010500Z --context <new-cluster> -f infrared.yaml

It checks the backup's manifest, that your key opens it and that a mirror run followed it. Then it creates the Secret infrared-backup-identity from your key file, runs the one helm install with restore.enabled=true and restore.point=<stamp>, and waits until the restore is complete and every Application is Synced and Healthy. It prints each change as it goes. --timeout (2 hours by default) bounds the wait.

Without ir, do the same by hand: create the Secret infrared-backup-identity (key identity) in the infrared namespace with kubectl, then helm install with your values plus restore.enabled=true and either restore.point=<stamp> or restore.from=<RFC 3339 time> (the newest complete backup at or before it). With Gitea, add gitea.replicaCount=0: Gitea starts once its data is back. Never pass the private key as a chart value: Helm keeps every value in its release record. The operator deletes the Secret once the restore is complete.

While a restore runs, the setup page shows it in place of the wizard: Plan the restore, Restore Gitea and Infrared's objects, Restore Postgres, then Bring back the stores. Every action that starts something, such as a run, a Release or a setup step, waits and says A restore is in progress, with Follow the restore. If the restore refuses to go on, for example because no complete backup matches, it stops and says so; a person decides what happens next.

To keep a copy outside the bucket, ir backup pull <stamp> --out <dir> -f infrared.yaml downloads the artifact and its manifest, and with --full the mirror as well. It reaches the bucket directly, so it works after the install is gone. ir restore --backup-artifact <dir>/<stamp>.irbackup then uploads whatever the bucket lacks before it restores.

API​

CallWhat it does
GET /v1/backupThe destination (without the key), the settings in force, what's missing, a backup in progress, the operator's view and the newest 24 backups
PUT /v1/backup/destinationSet where backups go, checked with a real write first
POST /v1/backup/destination/checkCheck the destination set
PUT /v1/backup/settingsSet the schedules, retention, recipients and WAL archive
POST /v1/backup/copiesBack up now
GET /v1/backup/copies/{stamp}One backup: its manifest and the mirror run that made it a point

Every call is for platform admins, and the calls that change something answer 409 while a restore is in progress.