Telemetry
Every Infrared install carries a metrics stack of its own, VictoriaMetrics, vmagent and Grafana, which measures Infrared's agent steps, API and components for its own dashboards and alerts. With install telemetry on, the stack's vmagent also writes a small, fixed set of those series to Darkshift's telemetry plane, under a name the install registers itself, so that Darkshift sees that installs come up and run, how agent steps behave and on which model provider, what fails and how often, without asking you and without anyone at Darkshift reaching your cluster. It is numbers only: no events, no content, and nothing that names a person. This page is the record of what leaves an install; it changes in the same change as the code that changes it.
Install telemetry is new in Infrared 0.1.3-rc.1 and sits behind the rollout flag installTelemetry. With the flag off, which is the default, nothing registers and nothing is sent, as before.
How an install registers
Once ir install --setup --wait sees the Installation Ready, it calls the plane once, POST /v1/telemetry/installs, with no sign-in and no account: the body is the install's id, which is its Installation's UID, the release it runs and its cloud, nothing else. The plane answers with the install's platform name, install-<id>, its write path, and a telemetry token bound to that platform, shown once: the token can write metrics under that one platform and do nothing else, so a token that leaked could only write numbers under your install's name. ir keeps the token in the Secret monitoring/telemetry-token on your cluster, records the platform and the write path on the Installation (spec.observability.telemetry), and your gitops repo renders the write into the metrics stack's vmagent: a second remote write beside the stack's own store, with a disk buffer of up to 800 MiB for a plane that is unreachable for a while. The plane stamps every series with the platform the token was minted for, whatever the sender set.
The id is stable for the life of the cluster and new for a rebuilt one. Registering again, which ir telemetry connect does, rotates the token; the old one stops working at once. The plane the install registers with is the chart value telemetry.plane, https://infrared.internal.darkshift.io by default; an install that sets it to a plane of its own sends everything there instead, and one that sets it empty registers nowhere.
What is sent
The metrics stack's vmagent keeps exactly these series for the plane and drops the rest of what it scrapes, and the plane refuses a write that carries any other series name. Every series carries two labels of the install's: platform, the name it registered as, and cluster, the management cluster's name.
| Series | Labels | What it counts |
|---|---|---|
infrared_step_attempts_total | org, workflow, role, outcome, provider | ended attempts of agent steps |
infrared_step_duration_seconds (a histogram, 30 seconds to 2 hours) | org, workflow, role | how long an attempt took |
infrared_step_tokens_total | org, workflow, role, model, kind, provider | tokens, by input, output, cache read and cache write |
infrared_step_cost_usd_total | org, workflow, role, model, provider | what the agent reported each step cost, at list prices |
infrared_step_agent_oom_killed_total | org, workflow, role, product | agent containers killed at their memory limit |
infrared_runs_active | org, workflow | runs in progress |
infrared_model_calls_total, infrared_model_call_seconds_total | org, workflow, role, model, provider, status, error | every model call a step's runner forwarded, by provider (anthropic, bedrock or claude-code), its outcome (2xx, 403, 429, 4xx, 5xx, network) and the provider's error type, such as ThrottlingException, and the seconds they took |
infrared_install_info | version, cloud, edge, bedrock | the release the install runs, its cloud, its edge and whether Bedrock is on |
infrared_installation_ready, infrared_installation_created_timestamp_seconds | whether the install is Ready, and when it was made | |
infrared_orgs_total, infrared_products_total, infrared_humans_total | how many orgs, Products and people the install holds: counts, no names | |
infrared_cloud_accounts | org, provider, state | the org's cloud accounts by cloud and state |
infrared_org_model_provider | org, provider | which provider each org's agent steps reach Claude through |
infrared_incidents_open | org, product, severity | open Incidents |
infrared_lease_lost_total, infrared_agentvm_peak_memory_bytes, infrared_agentvm_cpu_seconds, infrared_agentvm_phase | pool, template, phase | runner pools, where an install has them |
infrared_api_requests_total, infrared_api_request_duration_seconds | op, status | the API's requests by operation id and HTTP status, refusals included, and their latency |
infrared_telemetry_writes_total, infrared_api_client_address_unproven_total, infrared_api_edge_proof_armed | platform, status | the API's own telemetry write path and its proof of the client address, where the install serves them |
controller_runtime_reconcile_total, controller_runtime_reconcile_errors_total, controller_runtime_reconcile_time_seconds, workqueue_depth, workqueue_adds_total, process_resident_memory_bytes, process_cpu_seconds_total, go_memstats_heap_inuse_bytes, go_goroutines | controller, job | the operator's and the API's own process figures: reconciles, queues, memory, CPU and goroutines, from Infrared's own pods alone |
up, scrape_samples_scraped, scrape_duration_seconds | job, instance | whether each of the stack's scrape targets answered |
vmagent_remotewrite_*, vm_persistentqueue_*, vmagent_rows_inserted_total, vm_rows_inserted_total, vm_app_version | url | the vmagent's own figures for the write itself: what it sent, what was accepted, what waits in its buffer |
container_memory_working_set_bytes, container_cpu_usage_seconds_total | namespace, pod, container, node | the memory and CPU of agent step pods, those in the ir-org-* namespaces, read from the kubelet; the id, name and image labels are dropped first |
The labels carry the names you gave things: orgs, AgentWorkflows, AgentRoles, Products, models, operation ids and the step pods' names. Nothing in a label is a token, an email or a path, and the vmagent drops user.email, user.id and organization.id before anything leaves, in both spellings.
What is never sent
- Events: no record of who did what. The Changes log, the audit lines and the alert payloads stay on your install.
- Prompts, transcripts, file content, diffs, evidence, paths and repository names.
- Tokens, keys and any other secret; the telemetry token itself is the only credential involved, and it opens nothing of yours.
- A run's, step's or issue's record: no identifiers as labels on Infrared's own series, and no step's log.
- A person's name, email or login.
Nothing else leaves an install for Darkshift: there is no version check, no install count beyond this registration, and no crash reporting, and the license is checked offline. An agent step's model calls go to the model provider your org chose, the Anthropic API or Amazon Bedrock in your account, and to your forge, never through Darkshift.
Turn it off
Four switches, each remembered where it is set, so nothing turns itself back on:
| Switch | Who | What it does | Remembered |
|---|---|---|---|
ir install --no-telemetry | the person installing | Registers nothing: the install never gets a token, and the Installation names no plane | On the Installation, which stays unregistered until someone connects it |
| Settings → Telemetry, the switch Send this install's metrics to the telemetry plane | a platform admin, in the platform org | Asks you to confirm in place, then revokes the token at the plane, deletes the Secret monitoring/telemetry-token and clears the Installation's field; the stack's vmagent stops the write at its next render | On the Installation; the plane's token is gone |
ir telemetry disconnect | a platform admin, from a workstation | The same, through the API | On the Installation |
ir config set telemetry off | you, for every install and stack you run ir against | ir install registers nothing, and ir local up and ir local work connect nothing on a Mac, each saying so in one line. on puts it back | In ir's own config file, ~/.config/infrared/config.yaml or the --config file, as the key telemetry |
A disconnected install sends nothing: the check script's opt-out test (observability-check.sh --opt-out-test) proves the plane's count of its writes stays flat. Turning it on again, in Settings or with ir telemetry connect, registers the install anew under the same id with a new token. Darkshift can also revoke an install's token on the plane, after which the install's writes are refused within a minute and its buffer fills and drops.
See where it stands
ir telemetry status
shows Telemetry on or off, the Platform the install registered as, its Write URL, the Plane it registers with, Since when, and the plane's Last accepted write; -o json for scripts. Settings → Telemetry, for platform admins in the platform org, shows the same facts with the switch and the two lists above, and is marked New while the flag is on. The last accepted write is read from the install's own metrics store, from the vmagent's count of writes the plane answered with a 2xx; it shows a dash until the first one lands, within a minute or two of registering.
The platform name is also what the plane's dashboards group by, so if you ask Darkshift about your install, it is the name to give.
The flag
installTelemetry is a rollout flag: a bool, off by default, declared in Infrared 0.1.3-rc.1, to be retired by January 4, 2027. Set it in the install's values, at install or in your gitops repo afterwards (see Install configuration):
infrared-core:
api:
flags:
installTelemetry: true
With it off, ir install prints one line saying so and registers nothing, ir telemetry and the API's GET and PUT /v1/telemetry answer 404 naming the flag, and Settings shows no Telemetry panel. With it on, everything above applies. When the flag is retired, registering becomes the default for every new install, and ir install --no-telemetry, Settings, ir telemetry disconnect and ir config set telemetry off stay the ways to say no; the release notes of the release that retires it will say so.