CI Runners (Self-Hosted / ARC)
Where CI runs, what it costs, and how to fix it when it breaks.
Almost all CRED CI runs on self-hosted GitHub Actions runners on GKE, managed by Actions Runner Controller (ARC) in gha-runner-scale-set mode. GitHub-hosted runners are used only for a few meta-workflows that must not depend on the pool they manage.
Where CI runs
| Thing | Value |
|---|---|
| GKE cluster | cred-platform-prod-us-central1 |
| GCP project | cred-platform-prod |
| Region | us-central1 |
| Runner pods namespace | arc-runners |
| Controller + listeners namespace | arc-systems |
| Controller | gha-rs-controller (ARC gha-runner-scale-set chart, v0.14.x) |
| IaC | Terraform — cred-agent-ai/infra/runners/terraform (GCS backend cred-platform-tf-state, prefix runners/) |
| Runner images | cred-agent-ai/infra/runners/images/Dockerfile.scale-set (+ legacy Dockerfile) |
Get cluster access:
gcloud container clusters get-credentials cred-platform-prod-us-central1 \
--region us-central1 --project cred-platform-prod
kubectl get autoscalingrunnerset -n arc-runners
The runner pools
Each pool is an AutoscalingRunnerSet. Jobs route to a pool by putting the pool's exact name in runs-on: — the scale-set name is the routing key (not the legacy arc-runner-* labels). A wrong runs-on: value queues forever.
Pool (runs-on:) |
Node type | Capacity | Runners/node | Purpose |
|---|---|---|---|---|
arc-ss-small |
e2-standard-4 spot | 0 → 2500 | 2 | Default / bulk pool — most jobs |
arc-ss-large |
e2-standard-8 spot | 0 → 200 | 1 | Opt-in, memory-heavy jobs (>8 GiB) |
arc-ss-large-stable |
e2-standard-8 on-demand | 0 → 24 | 1 | Long memory-heavy jobs needing preemption resilience |
arc-ss-stable |
e2-standard-4 on-demand | 0 → 36 | 1 | Long non-spot jobs (e.g. author-agent) |
ubuntu-latest |
e2-standard-4 | 0 → 100 | — | Floating ubuntu-latest tag |
Scale-to-zero
Every pool has minRunners: 0. When no jobs are queued, the pool drains to zero runners and the cluster-autoscaler reclaims the nodes, so idle cost is ~just the small always-on baseline (kube-system + warm-pool overhead). Runners are ephemeral — one job per runner, then the pod exits and is replaced.
Cost
CI cost is billed as GKE node time on the cluster above. Spot pricing on the two bulk pools (arc-ss-small, arc-ss-large) keeps it low; the on-demand stable pools are the pricier per-node tiers.
Current run-rate (trailing 30 days): ≈ 3.4k gross / ~110/day average, but very bursty — it tracks PR/merge volume:
- Quiet weekday: ~$15–40/day
- Busy weekday: ~$60–85/day
- Weekend: ~$15–35/day
- A GitHub Actions outage day spiked to ~$242 (see troubleshooting) — outliers like this inflate the 30-day average.
Steady-state is closer to ~2–2.5k/mo**. Historically this was **~9.2k/mo before the 2026-07 ARC optimization (spot migration, scale-to-zero, VCE pool decommission, log-exclusion + networkmanagement API off).
How to check cost
Cost lives in the BigQuery billing export. Filter by the cluster label with a semi-join — never cross-join UNNEST(labels).
The 6× cost bug
FROM export, UNNEST(labels) fans every row out by its label count (~6.66 labels/row) and inflates the reported cost ~6×. Always use the WHERE EXISTS (…) semi-join form below.
-- Daily net CI cluster cost (net = cost + credits)
SELECT DATE(usage_start_time) AS day,
ROUND(SUM(cost + IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0)), 2) AS net_cost
FROM `cred-1556636033881.cred_costs.gcp_billing_export_resource_v1_013F1F_E02088_0DA499`
WHERE _PARTITIONTIME >= TIMESTAMP(DATE_SUB(CURRENT_DATE(), INTERVAL 33 DAY))
AND DATE(usage_start_time) >= DATE_SUB(CURRENT_DATE(), INTERVAL 31 DAY)
AND EXISTS (
SELECT 1 FROM UNNEST(labels) l
WHERE l.key = "goog-k8s-cluster-name"
AND l.value = "cred-platform-prod-us-central1"
)
GROUP BY day ORDER BY day;
The billing export lags ~1 day, so the most recent day under-reports until it settles.
How runners get deployed
Runners change via two coupled paths in cred-agent-ai, and order matters.
infra/runners/images/** → runner-image-publish.yml → builds + pushes image,
opens bot PR that pins the
new digest in
runner_image.auto.tfvars
(merge bot PR) → digest live on develop
infra/runners/** → arc-terraform-apply.yml → terraform apply rolls the
change onto the pools
- Image change (Dockerfile / baked scripts): merge to
develop→runner-image-publish.ymlrebuilds and opens an autobot/runner-image-digest-*PR that updatesrunner_image.auto.tfvars. The new image is only live once that bot PR merges. Never hand-editrunner_image.auto.tfvars. - Terraform change (pool sizing, node config, container command/env): merge to
develop→arc-terraform-apply.ymlplans + applies. Manual break-glass: run the workflow viaworkflow_dispatch(action: planthenapply).
Sequencing rule
If a Terraform change references something new in the image (e.g. repointing the container command to a new script), the image must be published and its digest live first. Applying the Terraform first makes every runner exec a file that isn't in the running image yet → all pools fail to start (org-wide CI outage).
The arc-terraform-apply workflow deliberately runs on GitHub-hosted ubuntu-22.04, not the ARC pool, so it can fix a broken pool without deadlocking on itself.
Runner wedge-guard (outage self-healing)
Every runner launches through /home/runner/wedge-guard.sh, a watchdog that self-terminates a runner stuck fetching a job during a GitHub outage (GetJobMessageAsync failing for 15 min with no job progress) so the pod exits, ARC reaps it, and its node drains — no human, no cost pileup.
- It is not a job-duration limit. A runner running a job never logs
GetJobMessageAsyncerrors, so normal jobs of any length (30 min, multi-hour) are unaffected. - Fails open: any internal error just runs
run.shnormally. - Knobs (Terraform env, no image rebuild):
WEDGE_GUARD_ENABLED(defaulttrue— setfalseto disable),WEDGE_GUARD_TIMEOUT(default900s).
Troubleshooting
Symptom: node count / cost spiking, hundreds of runners "Running" for hours
Cause: a GitHub Actions outage. Runners get assigned a job but can't fetch it (GetJobMessageAsync retry loop), never exit, and pin their nodes (safe-to-evict=false blocks the autoscaler).
First check — is Actions actually down?
curl -s https://www.githubstatus.com/api/v2/components.json \
| python3 -c "import sys,json; d=json.load(sys.stdin); print(next(c['status'] for c in d['components'] if c['name']=='Actions'))"
Now auto-handled by the wedge-guard (15-min self-terminate). If it's disabled or you need to act immediately, the manual playbook:
# 1. Reap wedged runners (assigned a job but stuck on GetJobMessageAsync)
for p in $(kubectl get pods -n arc-runners --no-headers | grep arc-ss-small | awk '{print $1}'); do
kubectl logs -n arc-runners "$p" -c runner 2>/dev/null | grep -q GetJobMessageAsync && echo "$p"
done | xargs -r kubectl delete ephemeralrunner -n arc-runners --wait=false
# 2. Stop the re-wedge bleed during the outage (cap to zero; RESTORE after recovery)
kubectl patch autoscalingrunnerset arc-ss-small -n arc-runners --type merge -p '{"spec":{"maxRunners":0}}'
# ...restore with maxRunners:2500 once Actions is operational
Cancelling the stuck runs on the GitHub side (gh api -X POST repos/OWNER/REPO/actions/runs/ID/cancel) returns 502 while Actions is down — retry after recovery; DELETE only works on runs in a terminal state.
Symptom: one pool has no runners and its jobs never start; others are fine
Cause: that pool's AutoscalingRunnerSet is stuck phase: Outdated — usually an outage interrupted an image/spec recreate and it lost its runner-scale-set-id annotation, so the controller loops on teardown and never re-creates the listener.
Check:
kubectl get autoscalingrunnerset <pool> -n arc-runners -o jsonpath='phase={.status.phase} id={.metadata.annotations.runner-scale-set-id}{"\n"}'
kubectl get autoscalinglistener -n arc-systems | grep <pool> # missing = wedged
Fix: recreate the CR so the controller mints a fresh scale-set + listener. A controller bounce (kubectl -n arc-systems rollout restart deploy/gha-rs-controller) does not fix this.
# Preferred: terraform apply -replace='helm_release.gha_rs_<pool>[0]' (keeps state consistent)
# Fast kubectl-only (keeps Helm labels so Terraform stays a no-op):
kubectl get autoscalingrunnerset <pool> -n arc-runners -o json \
| python3 -c 'import sys,json; d=json.load(sys.stdin); m=d["metadata"]; [m.pop(k,None) for k in ("resourceVersion","uid","creationTimestamp","generation","managedFields")]; m.get("annotations",{}).pop("kubectl.kubernetes.io/last-applied-configuration",None); d.pop("status",None); print(json.dumps(d))' \
> /tmp/ars.json
kubectl delete autoscalingrunnerset <pool> -n arc-runners --wait=true --timeout=90s
kubectl apply -f /tmp/ars.json
# verify: phase=Running, non-empty runner-scale-set-id, listener CR present
Symptom: a run is stuck "queued", 500s when you try to cancel
Stuck/orphaned queued runs (common during/after an outage) return 500/502 on cancel. DELETE on the run often works and unwedges it. These can also poison ARC's desired-count math — clear them once the API is healthy.
Symptom: runners won't start after an image change
Check the image actually published and the digest is live: runner-image-publish run is green, the bot/runner-image-digest-* PR merged, and runner_image.auto.tfvars shows the new sha256. Then confirm the pool rolled: kubectl get autoscalingrunnerset <pool> -n arc-runners -o jsonpath='{.spec.template.spec.containers[0].image}'.
Quick reference
# pool sizing
kubectl get autoscalingrunnerset -n arc-runners
# what's running right now
kubectl get pods -n arc-runners | grep -c Running
# listeners (one per pool)
kubectl get autoscalinglistener -n arc-systems
# controller
kubectl get pods -n arc-systems | grep controller
# a runner's logs
kubectl logs -n arc-runners <pod> -c runner --tail=50
- Owner repo for all runner config:
cred-agent-ai→infra/runners/ - Never hand-edit
runner_image.auto.tfvars(auto-generated by the publish workflow) - Never cross-join
UNNEST(labels)in cost queries (6× inflation)