Skip to main content
Version: 0.2 (next)

Back up and restore

Infrared backs up your whole install to object storage outside its cluster, on a schedule, so losing the cluster doesn't mean losing your work. Each backup is one encrypted artifact holding Infrared's objects (orgs, Products, AgentRoles, AgentWorkflows, AgentWorkflowRuns and the rest), Gitea's data when the install runs Gitea, and dumps of the platform's Postgres databases. Beside it, Infrared mirrors the buckets of the cluster's object store.

A backup that a mirror run has followed is a point. A restore brings a point back whole, onto new servers.

Only platform admins see and change backups, on Settings → Backups in the platform org.

Set up backups​

Before the first backup, Infrared needs three things: a destination, the destination's key, and at least one recipient to encrypt to. Set them on Settings → Backups, or in the setup wizard's optional Set up backups step, which shows the same form once the gitops repo is hydrated. Skip for now leaves it for later.

Until all three are set, the panel says No backup is made yet and names what's missing.

Destination​

FieldWhat to enter
ProviderLinode Object Storage, Amazon S3 or another S3-compatible store, or Google Cloud Storage
RegionFor Linode, the region in the bucket's endpoint, such as us-lax-4. For S3, the region requests are signed for, such as us-east-1. For Google Cloud Storage, the bucket's location; empty means us-central1
EndpointFor S3 only: the store's endpoint, https:// and a host with no path, such as https://s3.us-east-1.amazonaws.com. Linode's follows from the region, and Google Cloud Storage's is always https://storage.googleapis.com
BucketThe bucket's name. The install writes only under its prefix
PrefixThe path everything is kept under in the bucket. It defaults to the install's name
Access key ID and Secret access keyFor Linode and S3: a key that can list, read, write and delete in the bucket

Google Cloud Storage takes no key. Infrared reaches the bucket as its own Kubernetes ServiceAccounts through Workload Identity, as on GKE. Grant them roles/storage.objectUser on the bucket before you save. Postgres's WAL archive isn't available on Google Cloud Storage.

The key is write-only. Once it is set, the panel says when it was set, never what it is. Replace the access key lets you enter a new pair, and Keep the key set cancels that.

Before it saves a destination, Infrared checks it with a real write: it lists the prefix, then writes, reads and deletes one object. Check destination runs the same check on the destination already set. A destination that fails isn't saved, and the panel lists each step, list, write, read and delete, with what the store answered, so you can see exactly what the key is missing.

Schedule and recipients​

FieldDefaultWhat it does
Backup schedule5 * * * *When a backup is made: five cron fields, in UTC
Mirror schedule17 * * * *When the object store's buckets are mirrored: five cron fields, in UTC. A backup becomes a point at the first mirror run after it
Retention7dHow long backups are kept, in whole days from 1d to 999d. The newest complete backup is never deleted
RecipientsnoneUp to 16 age public keys (age1…). Each backup is encrypted to all of them, and any one of their private keys opens it
Archive Postgres's WALoffAlso keeps Postgres's write-ahead log in the bucket, beside the dump in each backup. Not available on Google Cloud Storage

Add recipient adds a line. Save backups saves the destination and the settings together, checking the destination first and sending only what you changed. Without a recipient, no backup is made.

Paste only the public half of a key. If you paste a private key (AGE-SECRET-KEY-…), the form empties the line and tells you it wasn't kept.

To make a key pair, run age-keygen and keep its output in your vault, or run ir backup key new <install>, which writes the private key to a file only you can read and prints its public half. Keep at least one private key somewhere that survives losing the cluster: without one, no backup can be opened.

The settings can also come from the chart's backup values at install. Infrared copies them into its own settings once, while those are empty. After that, change them on Settings → Backups.

See your backups​

Once backups are set up, the panel shows What the operator last saw: whether backups are healthy (BackupsHealthy), the newest backup, the newest mirror run, the newest point and the operator's last message. Settings → Installation shows the newest point too, so anyone in the platform org can see at a glance that the install is covered.

Backups in the destination lists the newest 24 backups, read straight from the bucket:

ColumnWhat it says
BackupIts stamp, such as 20261006T010500Z, and how long ago it was made
SizeThe artifact's size
PartsWhat it holds: Infrared's objects, Gitea's data and the Postgres databases dumped
Mirror runThe mirror run that followed it
CompleteWhether the artifact and its manifest are both there and match
PointWhether a restore can bring it back, or why not yet

Back up now makes a backup straight away, followed by a mirror run, so you have a fresh point before a risky change. The panel shows Backing up until both are done. The button is off while a scheduled backup runs.

From a terminal, ir backup status, ir backup list, ir backup get <stamp> and ir backup now do the same.

Restore an install​

A restore brings an install back from one point, onto new servers, before Infrared runs there. You need:

  • a cluster with nothing of Infrared on it, reached by its own kubeconfig context;
  • the install's usual install file and values, with the same backup block, so the restore reads the same bucket;
  • the age private key of one of the backup's recipients;
  • the bucket's key.

ir restore does the whole restore:

ir backup list # pick a point
ir restore --backup-artifact 20261006T010500Z --context <new-cluster> -f infrared.yaml

It checks the backup's manifest, that your key opens it, and that a mirror run followed it. Then it creates the Secret infrared-backup-identity from your key file, runs one helm install with restore.enabled=true and restore.point=<stamp>, and waits until the restore is complete and every Application is Synced and Healthy, printing each change as it goes. --timeout, 2 hours by default, bounds the wait.

Without ir, do the same by hand. Create the Secret infrared-backup-identity, key identity, in the infrared namespace with kubectl, then helm install with your values plus restore.enabled=true and either restore.point=<stamp> or restore.from=<RFC 3339 time>, which picks the newest complete backup at or before that time. With Gitea, add gitea.replicaCount=0: Gitea starts once its data is back. Never pass the private key as a chart value, because Helm keeps every value in its release record. The operator deletes the Secret once the restore is complete.

While a restore runs, the setup page shows it in place of the wizard: Plan the restore, Restore Gitea and Infrared's objects, Restore Postgres, then Bring back the stores. Anything that would start work, such as a run, a deployment or a setup step, waits and says A restore is in progress, with Follow the restore. If the restore can't go on, for example because no complete backup matches, it stops and says so, and you decide what happens next.

Keep a copy outside the bucket​

ir backup pull <stamp> --out <dir> -f infrared.yaml downloads a backup's artifact and manifest, and with --full the mirror as well. It reads the bucket directly, so it works even after the install is gone. ir restore --backup-artifact <dir>/<stamp>.irbackup then uploads whatever the bucket lacks before it restores.

API​

CallWhat it does
GET /v1/backupThe destination (without the key), the settings in force, what's missing, a backup in progress, the operator's view and the newest 24 backups
PUT /v1/backup/destinationSet where backups go, checked with a real write first
POST /v1/backup/destination/checkCheck the destination already set
PUT /v1/backup/settingsSet the schedules, retention, recipients and WAL archive
POST /v1/backup/copiesBack up now
GET /v1/backup/copies/{stamp}One backup: its manifest and the mirror run that made it a point

Every call is for platform admins. The calls that change something answer 409 while a restore is in progress.