Troubleshooting
Infrared tries to tell you what is wrong where you see it: a run's banner, a deployment's message, a condition on a cluster's page, a callout on a Settings panel. This page collects those messages by what you were doing when you saw them, with the cause and the fix. Look for the words you see on screen.
When a step of a Change stops and asks for a person, its own message says why, and When a step needs a human lists each one. When the whole install is paused for want of a license, see Licensing. When Slack doesn't post, see Slack.
Setting up Infrared
| What you see | Why | What to do |
|---|---|---|
| The wizard asks for the setup token again | This browser has never presented the token. The token lives in a cookie | Paste the token again. A new browser always asks |
| Password says "Use at least 20 characters" or "Use at most 72 characters", and Create platform admin stays off | The root password must be 20 to 72 characters (bytes, as bcrypt counts them) | Use one your break-glass vault's generator made, 32 characters or more |
| The admin step says "The root user exists. Rotate its password in Access, Break glass." | The root user was already made, in another browser or by ir setup run | Sign in as the root user, or have a platform admin signed in with Google rotate its password |
| GitHub says the redirect URI isn't associated with the App | The App has no callback URL, or names another host. An App created while Infrared was open on a local or private address, such as a port-forward, gets none | Add https://<host>/api/v1/auth/github/callback on the App's General tab. See Expose Infrared over HTTPS |
Every webhook delivery gets 401, and the API's log says GitHub webhook rejected | The App's webhook secret differs from the one Infrared holds. An App created with its webhook off has no secret at all | Make a secret and set it on both sides: Connect the webhook, step 2 |
The wizard's sync step, or an Application in Argo CD, shows ComparisonError mentioning the repo-server and connection refused | Argo CD's repo-server wasn't listening yet during the first sync | Wait for the next sync, or force one: kubectl --context <cluster> -n argocd annotate application <name> argocd.argoproj.io/refresh=hard --overwrite |
infrared (wave 40) stays Unknown with "error pulling OCI chart" | Argo CD can't reach the chart repository or sign in to it | Check the cluster can reach the chart repository and that its registry token is fresh (the chart's registryToken values) |
| A wave stays Progressing for a long time | A component is still pulling images or starting a database | In Argo CD, or with kubectl --context <cluster> get pods -A, find the pod that isn't Ready and read its events |
ir install refuses because the backup prefix already has something under it | Another install of the same name writes its backups there, and a fresh install would write over them | Restore with ir restore instead, or install under another name or prefix |
Signing in
Sign in with Google
See Sign in with Google.
| What you see | Why | What to do |
|---|---|---|
| The sign-in page has no Sign in with Google | The install has no client ID, no client secret or no allowed domain | Settings → Sign-in says what's missing under Status |
Google answers redirect_uri_mismatch | The OAuth client doesn't list the redirect URI of the address you signed in at | Add https://<host>/api/v1/auth/google/callback for that address under the client's Authorized redirect URIs. Settings → Sign-in shows the exact URI |
Google answers org_internal | The client's user type is Internal, and the account belongs to another Google Workspace, or to none | Sign in with an account of the Workspace that owns the client |
| The sign-in page says "Google refused this install's OAuth client" | Google answered invalid_client: the client secret Infrared holds is wrong, deleted or rotated away. Password sign-in comes back on until a Google sign-in works again | Put the client's current ID and secret in the keys google-client-id and google-client-secret of the Secret infrared-platform-tokens, then sign in with Google |
| "Your Google account is suspended: ask a Google Workspace admin." | Google Workspace reports the account suspended, in a group Infrared reads | A Google Workspace admin restores the account. Infrared sees it at its next read of the groups, within about 2 minutes |
| No root user form under Other ways to sign in, and Sign in with Google doesn't work | The install turns password sign-in off, and Google sign-in broke after it had worked | Delete the ConfigMap infrared-sign-in in Infrared's namespace: password sign-in works again within 15 seconds. See Sign in when Google can't |
| Password sign-in still works although your values turn it off | No Google sign-in has worked on the install yet, so Infrared keeps a way in open | Sign in with Google once. Settings → Sign-in explains under Password |
Sign in with GitHub
| What you see | Why | What to do |
|---|---|---|
You land back on the sign-in page with "<login> is not on any team: ask an org admin to add you" | The GitHub login is in no access group of any org | Sign in as the root user or with Google, and add the login under Workforce → Access groups |
"<login> is linked to a person who signs in with Google here" | Someone who signs in with Google linked that GitHub login, for git rather than for signing in | Sign in with Google. To sign in with GitHub again, unlink the login under Accounts on your You page, or a platform admin does in Access → People |
The root user
See Break glass.
| What you see | Why | What to do |
|---|---|---|
| Email or password is incorrect | The address or the password isn't the root user's | Take both from your break-glass vault. If the vault's password doesn't work, a platform admin signed in with Google rotates it in Access → Break glass |
HTTP 429, "too many failed sign-ins from this address: try again in <time>" | 5 failures in a minute, or 10 in 15 minutes, from your address | Wait the time it names. See Password sign-in limits |
HTTP 429, "Infrared is busy checking other passwords: try again in a few seconds" | 4 password checks were already running | Try again in a few seconds |
POST /v1/auth/login answers 403, "Password sign-in is turned off on this install" | The install turns password sign-in off, and Sign in with Google works | Sign in with Google. To break glass anyway, see Sign in when Google can't |
Signing in from ir
| What you see | Why | What to do |
|---|---|---|
| A sign-in link lands on the sign-in page: "The sign-in link expired or was used already: make a new one with ir open." | A link works once, within 60 seconds | Run ir open again |
| "The token that made this link was revoked: sign in another way." | The API token ir saved was revoked | Sign in with Google, make a new token and give it to ir (Use ir with an API token) |
| "The root user can't sign in on this install: password sign-in is off." | Root sessions aren't allowed while password sign-in is off | Sign in with Google |
| "The token's owner has no Infrared user: sign in with Google once." | The token's owner has never signed in with Google on this install | They sign in with Google once, then run ir open again |
ir login --context exits 2: "Infrared keeps no password in the cluster" | ir has no saved API token that works on that install | Sign in with Google, make a token on your You page and run ir login --token-stdin; or break glass with ir login --email <root user's address> --password-file <file> |
Someone can't see an org, or can't change something
Roles come from access groups. See Access groups and access. Access shows what Infrared last read of each Workspace group, and why a read failed.
| What you see | Why | What to do |
|---|---|---|
| A person signed in with Google sees no org, and their You page says "No role in any org yet" | They are in no access group's Workspace group, and no access group lists a GitHub login they linked | Add them to an access group's Workspace group in Google Workspace |
| An access group's Workspace group says "Not read for 10 minutes: its people hold no role until it is." | Infrared couldn't read the group lately. Access says why | Fix the cause, then choose Read now. The root password works meanwhile |
| Everyone who signs in with Google is still a platform admin | No platform-admin access group of the platform org has a Workspace group with an active person of an allowed domain yet | Map one, then choose Read now. See What a Google session can do |
| Access says "No Workspace directory is configured" | The chart value api.directory.serviceAccount is empty, so no Workspace group gives a role | Set it up: Read Workspace groups |
| Access says "The directory could not start", or every group's read fails with a permission error from Google | The API can't impersonate the service account: it has no Google credentials of its own, or they may not make the service account's tokens | On GKE, check the cluster uses Workload Identity, and grant the API's principal roles/iam.workloadIdentityUser on the service account (step 2 of Read Workspace groups) |
Saving an access group answers "Infrared may not read the group <group>" | The service account lacks the Workspace admin role Groups Reader | A super admin assigns Groups Reader to the service account in the Google Admin console (step 4) |
Saving an access group answers "there is no Workspace group <group>: check the address" | The address names no group | Check the group's address in Google Workspace |
| Saving an access group answers that the group "is outside the Google domains allowed to sign in here" | The group's domain isn't an allowed domain | Map a group of an allowed domain, or allow its domain in Settings → Sign-in |
| Saving an access group answers that the group "holds everyone in the domain", or that groups nest too deep inside it | Infrared reads named people only, and at most 5 levels of groups inside groups | Map a group of named people, with flatter nesting |
Saving the platform org's access groups answers "That would take platform admin from you: you are not in <group>." | The change would leave you, signed in with Google, without platform admin | Add yourself to the group in Google Workspace and choose Read now, or make the change as the root user |
A GitHub team says "The App <app> lacks Organization members (read and write)" | The org's GitHub App can't change the organization's GitHub teams | Give the App the permission, and accept it on its installation. See GitHub teams |
A login is skipped: "not a member of <organization>: invite them on GitHub" | Infrared never invites anyone to a GitHub organization | An owner of the organization invites them on GitHub. The next sync adds them once they accept |
| Someone in the group isn't in the GitHub team, listed as having no linked GitHub login | They haven't linked a GitHub login, and without one they have no git access | They link GitHub under Accounts on their You page |
| Linking says "That account is linked to another person here" | Someone else on the install linked that account | They unlink it on their You page, or a platform admin does in Access → People |
Linking says "<connector> can't link accounts yet" | The git connector's client is missing its ID or secret; the message says which | A platform admin fixes the connector in Access → Git connectors, then chooses Check |
| The forge refuses the link's redirect or callback address | The forge's OAuth application doesn't list the callback for the address Infrared was opened at | Register https://<host>/api/v1/me/links/<connector>/callback on the application. For the platform org's App, add its sign-in callback (Set it up) |
| A git connector is Not ready | The forge refused the client: a wrong client ID or secret, or on Gitea a callback the application doesn't list | Fix them on the forge and under Edit, then choose Check |
A Change doesn't start, or stops
A run's banner, at the top of its page, says what it is doing or waiting for.
| What you see | Why | What to do |
|---|---|---|
| Every page says Agent runs are stopped | The org's Anthropic key is out of credit or was rejected | Add credit, or set a working key in Settings → Model provider, choose Check now, then retry the step that stopped |
| A Change waits at Triage, in amber, and nothing builds | With human approvals on, triage waits for a person to decide | Choose Build it or Hand to a person on the Change's page |
| A Change ends Finished with no change | The builder changed no files, so nothing was merged or deployed | Read the builder's step. Its verdict says why, such as NO CHANGE NEEDED, or asks a question. Add detail to the issue and ask again if it should have changed something |
| Starting a Change answers "AgentWorkflow "change-quick" is disabled: enable it, then start the run", or opening an issue answers "no enabled AgentWorkflow hydrates new issues: enable change-quick" | The org switched change-quick off | Enable it under Workforce → Agents → AgentWorkflows |
| An issue opened on GitHub starts no Change | The org's GitHub App webhook isn't active, or change-quick is off | Activate the App's webhook on GitHub. See Triggers and scheduling |
| Start the change, Start run or Build it says A restore is in progress | The install is being restored from a backup, and no new work starts until the restore is complete | Choose Follow the restore, and try again once it's done |
A deployment waits or fails
A deployment's message, and each zone's, say what it is doing or waiting for. Retry on a failed deployment starts it again from where it failed; from a terminal, ir deployment retry <name>.
| What you see | Why | What to do |
|---|---|---|
"zone <zone> is on Cluster <cluster>, which is neither org <org>'s own nor on its cluster allowlist" | The zone names a cluster the org may not use | A platform admin ticks the org under Orgs allowed on the cluster's page. Then retry |
"zone <zone> is on Cluster <cluster>, which does not exist" | The zone's cluster names no cluster Infrared knows | Fix the zone's cluster, or register the cluster. Then retry |
The zone waits with "waiting for Cluster <cluster> to be registered" | The cluster isn't Registered yet. Infrared checks again every minute | See A cluster won't connect |
"<version> already tags <commit>: there is nothing new to release" | Nothing was committed since the last deployment's tag | Merge a Change first |
"kpack did not build <commit> within 45m0s" or "kpack build ... failed" | The build didn't finish, or failed | Look at the kpack Image and its Builds in the namespace builds, fix the build, then retry |
| The build log says "No log is kept for this build: kpack keeps the logs of its newest builds only." | kpack removes the pods of all but its newest builds, and the log goes with the pod | Read the steps on the deployment's page instead: their times and a failed step's reason stay there |
"Application <app> is <sync>/<health> and not yet on <version> after ...: check it in Argo CD, then retry the Release" | Argo CD didn't get the new version running in the zone in time | Look at the Application in Argo CD, fix what keeps it from syncing or becoming Healthy, then retry |
"<version> failed its release guard in <zone>: the smoke check failed 3 times" | A check in the zone's verify or guard.checks failed 3 times, or failed once more at the end of the hold. The zone keeps the new version | Check the URL and the expected status or text, fix the app or the check, then retry |
"<version> failed its release guard in <zone> and was rolled back to <previous>" | The zone failed its guard, and Infrared put back what it ran before through a pull request to the gitops repo. The zone runs <previous> again; its data is as <version> left it | Fix the app or the check, then retry or start a new deployment |
"the release guard failed (...) and the rollback to <previous> did not finish in 30m0s" | The rollback's pull request merged but the previous version didn't become Healthy in time, or the pull request wouldn't merge | Look at the zone's Application in Argo CD and at the rollback's pull request. The zone may still run the failed version |
"<version> failed its release guard in <zone>: Application ... was .../Degraded ... during the hold" | The zone ran the new version Healthy, then was off Synced and Healthy for a minute during its hold. The zone keeps the new version | Look at the zone's pods and events on its Live tab and in Argo CD, fix the app, then retry or start a new deployment |
| "pull request #N was closed without merging" | Someone closed the promotion's pull request on the gitops repo | Retry the deployment |
Go now didn't go through: the zone "is <phase>, not Holding", "has no hold", or its hold "has not started yet" or "ended" | Go now ends a hold that is running. The zone hasn't passed its smoke check yet, its hold is over, or it deploys fast and has none | Nothing to fix: the release guard moves on by itself. Use Go now while the zone shows Holding with time left |
Go now didn't go through: the zone "has not been Synced and Healthy since <time>" | The zone left Synced and Healthy during its hold, and Go now can't skip that | Look at the zone's pods and events, and choose Go now again once it is healthy |
| A promotion to Production waits and says a license is needed | The open beta has ended and no valid license is installed | Install a license. See Licensing |
Products, environments and domains
| What you see | Why | What to do |
|---|---|---|
A deleted Product stays, with deletionTimestamp set | Infrared holds it until its zones, build and chart are out of the gitops repo and the cluster. status.zoneFiles.message says what it waits for, such as a pull request GitHub won't merge yet | Fix what the message names. Don't remove the finalizer infrared.darkshift.io/zones by hand: the zones' Applications and namespaces would stay behind |
An environment's Settings tab shows Removing <zone> for a long time | Infrared is taking it out of the gitops repo and the cluster | Wait. The callout shows what blocks it, if anything |
A domain stays Pending: "No TXT record at _infrared.<domain> yet" | The ownership record isn't there, or hasn't propagated | Add the TXT record Infrared shows, then choose Check now. See Domains and addresses |
| A domain's traffic check says it "goes through Cloudflare's proxy (the orange cloud)" | Cloudflare's proxy answers instead of Infrared | Switch the record to DNS only |
| A domain is Failed: "... is already claimed by another org" | Another org claimed the name first | Choose another name |
| A managed domain is Pending: "the installation has no ingress host or DNS token set up for managed domains" | The Installation has no spec.previews.ingressHost or cloudflareTokenSecret | A platform admin sets them. See the CRD reference |
A cluster won't connect or build
A workload cluster's conditions say what's wrong, and its page shows them under Connection. See Workload clusters.
| What you see | Why | What to do |
|---|---|---|
Connected is False: "kubeconfig uses an exec or auth-provider plugin: use a service account token or client certificate" | The kubeconfig signs in through a plugin, such as aws eks get-token, which Argo CD can't store | Use a kubeconfig with an inline service account token or client certificate, and replace it under Details → Replace the kubeconfig on the cluster's page |
Registered is False: "kubeconfig reads its token from a file: inline the token" | The kubeconfig's user has a tokenFile | Put the token itself in the kubeconfig |
Connected is False: "cannot reach the cluster: ..." | The management cluster can't reach the API server, or the credential is wrong or expired | Check the address from inside the management cluster, and the credential's permissions. Set spec.server if Argo CD must use another address |
Registered is False: "GitHub will not merge pull request #N yet: ..." | Branch protection or a required check on the platform org's gitops repo blocks the cluster's pull request | Let the check pass, or merge the pull request yourself. Infrared notices on its next look |
Provisioning waits: "waiting for Secret cluster-<name>-credentials in namespace infrared with the cloud credentials" | The cluster's credentials Secret doesn't exist yet | Create it: kubectl --context <cluster> -n infrared create secret generic cluster-<name>-credentials --from-env-file=credentials.env |
Provisioning waits: "Cloud account <name> is Pending" or "doesn't exist" | The cluster's provision.cloudAccount isn't a verified cloud account of the platform org | Verify the account, or fix the name |
| Provisioning failed: "tofu plan failed: kubectl -n infrared logs job/..." | OpenTofu failed: wrong credentials, a missing variable, or a quota | Read the Job's logs with the command in the message, fix the cause, then save the cluster's Details again to start a new run |
The build succeeded but Connected stays False: "no connection: set spec.connection.inCluster or spec.connection.kubeconfigSecretRef" | Infrared doesn't connect by itself to a cluster it built | Put the new cluster's kubeconfig in a Secret and set spec.connection.kubeconfigSecretRef |
Cloud accounts and registries
| What you see | Why | What to do |
|---|---|---|
| Add a cloud account offers no Google Cloud, or no AWS | The install has no identity for that cloud: the chart's cloudIdentity values are empty | Ask whoever installed Infrared to set them. See Infrared's own identity |
| The check failed, so the account wasn't kept | Infrared couldn't act as the account: the grant under What to grant is missing, or the project, service account or role is wrong | Make the grant exactly as shown, then add the account again |
| An account turns Failed after it was verified | Its daily check failed: the grant was removed or the role changed | Restore the grant. The next check settles it |
| A registry can't be added: "add and verify one first" | The org has no verified cloud account for the registry's cloud | Add the cloud account first |
| A registry's Delete is off | It is the org's build registry | Choose another build registry, or Install default, first |
Upgrades and the gitops repo
| What you see | Why | What to do |
|---|---|---|
A helm upgrade you ran was undone | Argo CD manages Infrared and puts back what the gitops repo says | Upgrade with a pull request. See Upgrades |
| Your edit to a file in the gitops repo disappeared after a template upgrade | The template writes that file, and hydration replaced your edit | Keep your changes in files the template doesn't write, such as your values file |
| The chart refuses a value in your pull request | The chart checks values against its schema, and the value is misspelt, misplaced or new in a later release | Check the key and its nesting under infrared-core:, and the release notes of the version you move to |
| A tab shows A new version of Infrared is available | Infrared was upgraded since the tab loaded | Choose Reload. Nothing you saved is lost |
Backups and restores
| What you see | Why | What to do |
|---|---|---|
| Settings → Backups says No backup is made yet | A destination, its key or a recipient is missing; the panel names which | Set what it names. Without a recipient, no backup is made |
The destination wasn't saved, with a step failed | The check's real write failed: a wrong region or bucket, or a key that can't list, write, read or delete under the prefix | Fix what the failed step's message says, then save again |
| A backup's Point says Not yet | No mirror run has followed it yet | Wait for the next mirror run, or choose Back up now, which runs one right after the backup |
| Back up now is off | A backup is already running, on schedule or asked for | Wait for it to finish |
| The restore says The restore refused to go on | No complete backup matches the point or time asked for, or your key doesn't open it | Read the restore's message, pick another point with ir backup list, and start again. See Back up and restore |