Introduction
Ascender deployed on K3s stores its container images on the node running the cluster. On long-lived deployments these images accumulate and can consume a significant amount of disk space. This article covers how to reclaim that space immediately and how to configure K3s to manage it automatically.
Problem
A node running Ascender on K3s gradually fills its disk. Each time the execution environment image is pulled and the tag resolves to a new digest, the previously pulled image becomes untagged rather than being replaced.
By default, K3s removes these untagged images only under disk pressure. The kubelet scans for unused images every five minutes and compares usage on the filesystem holding the image store against two thresholds. Collection begins once usage reaches imageGCHighThresholdPercent, which defaults to 85 percent. The kubelet then deletes unused images, least recently used first, until usage drops below imageGCLowThresholdPercent, which defaults to 80 percent, and then stops. Images younger than imageMinimumGCAge, two minutes by default, are never considered. A fourth setting, imageMaximumGCAge, removes images that have gone unused for a given length of time regardless of disk usage, but it defaults to 0s, meaning disabled.
The practical effect is that no cleanup happens at all below 85 percent, and the larger the disk, the more space that represents. On a 500 GB disk roughly 75 GB of unused images can accumulate before the kubelet does anything, and once cleanup does begin, usage settles between the two thresholds rather than returning to a baseline.
This happens most quickly when a job template or execution environment references a floating tag such as latest combined with a pull policy that always pulls the image, because every pull of an updated image leaves the previous one behind.
Running the Ascender cleanup jobs does not help here. Those jobs prune database records, not container images.
Symptoms
Disk usage on the K3s node climbs steadily over weeks or months with no corresponding growth in job or inventory data.
Listing images shows a large number of untagged entries:
sudo /usr/local/bin/k3s crictl images | grep ascender-ee | awk '{print $2}' | sort | uniq -c | sort -rn
150 <none>
1 latest
When measuring usage, note that /run/k3s/containerd contains bind mounts of running container filesystems. Tools such as du will follow those mounts and report space that physically resides elsewhere. The image content itself is stored under /var/lib/rancher/k3s/agent/containerd, so identify the filesystem holding that path before tuning any thresholds:
df -h /var/lib/rancher/k3s
Resolution
Reclaim space immediately
Remove all images that are not referenced by a container:
sudo /usr/local/bin/k3s crictl rmi --prune
This only removes unused images. Images backing running containers are left in place. On a node that has accumulated images for several months this typically recovers the majority of the consumed space.
Option 1: Scheduled pruning with cron
The simplest ongoing approach is to run the same prune command on a schedule. Create /etc/cron.d/k3s-image-prune:
0 3 * * 0 root /usr/local/bin/k3s crictl rmi --prune >> /var/log/k3s-image-prune.log 2>&1
This runs weekly at 03:00 on Sunday. It requires no changes to the cluster configuration and is easy to verify, but it removes unused images on a fixed schedule regardless of whether they were needed, and it does nothing between runs.
Option 2: Kubelet image garbage collection
The kubelet offers two independent controls. Either can be used on its own, or both together.
The first is a time limit. Setting imageMaximumGCAge removes images that have gone unused for longer than the configured duration, regardless of how much disk is free. This is the closest equivalent to the cron approach, but the kubelet applies it continuously rather than on a schedule. A value of 168h keeps an unused image for one week, 336h for two. The default of 0s means disabled rather than immediate, and the value must be greater than imageMinimumGCAge, so very short durations are not available.
The second is the disk threshold. Lowering imageGCHighThresholdPercent and imageGCLowThresholdPercent does not change how collection works, only when it starts and how far it goes. This keeps the ceiling behavior but moves it somewhere useful on a large disk.
Use the time limit if the goal is to stop images accumulating in the first place. Lower the thresholds if the goal is to leave more headroom before the disk fills. Setting both gives continuous trimming with an earlier safety net under sudden growth.
The current values can be confirmed on a running node:
sudo /usr/local/bin/k3s kubectl get --raw "/api/v1/nodes/$(hostname)/proxy/configz"
"imageMinimumGCAge": "2m0s",
"imageMaximumGCAge": "0s",
"imageGCHighThresholdPercent": 85,
"imageGCLowThresholdPercent": 80,
Create /etc/rancher/k3s/kubelet-config.yaml. All four settings are shown together below, with the time limit set to one week and the thresholds lowered. Any setting left out of the file keeps its default, so a file containing only imageMaximumGCAge is valid and leaves the thresholds at 85 and 80.
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
imageMinimumGCAge: 2m
imageMaximumGCAge: 168h
imageGCHighThresholdPercent: 75
imageGCLowThresholdPercent: 60
Reference it from /etc/rancher/k3s/config.yaml:
kubelet-arg:
- "config=/etc/rancher/k3s/kubelet-config.yaml"
If the file already exists, add the kubelet-arg entry to it rather than replacing it.
Restart K3s to apply the change:
sudo systemctl restart k3s
Confirm K3s returned to service and the values took effect:
systemctl is-active k3s
sudo /usr/local/bin/k3s kubectl get --raw "/api/v1/nodes/$(hostname)/proxy/configz"
"imageMinimumGCAge": "2m0s",
"imageMaximumGCAge": "168h0m0s",
"imageGCHighThresholdPercent": 75,
"imageGCLowThresholdPercent": 60,
The kubelet evaluates images every five minutes and logs each removal:
image_gc_manager.go:529] "Removing image to free bytes" imageID="sha256:b116e155..." size=2217006
On K3s releases built on Kubernetes 1.35 and later, no additional configuration is required. On earlier releases the feature is gated and must also be enabled in the same file:
featureGates:
ImageMaximumGCAge: true
Check the release in use with k3s --version. Including this block on a newer release logs a warning that the gate is scheduled for removal, so omit it once the cluster is on 1.35 or later.
Notes
imageMaximumGCAge must be greater than imageMinimumGCAge, which defaults to two minutes. If it is not, the kubelet fails validation, exits, and K3s enters a restart loop, leaving the node unavailable:
Error: failed to validate kubelet configuration, error: invalid configuration:
imageMaximumGCAge 1m0s must be greater than imageMinimumGCAge 2m0s
References & related articles
Installing Ascender
Update the Ascender Execution Environment on Kubernetes
Building a Custom Execution Environment for Ascender
CIQ Ascender Documentation
Ascender Community Documentation
Kubernetes Garbage Collection
Kubernetes Kubelet Configuration Reference
K3s Agent Configuration Reference
Rocky Linux Cron Jobs Guide