kind load docker-image on ubuntu-latest (Ubuntu 26.04): "ctr: content digest ... not found" and Docker 29's containerd image store
ubuntu-latest is moving from Ubuntu 24.04 to 26.04 between October 19 and November 19, 2026 (what else changes). The 26.04 image has Docker 29.4.2, and 24.04 has 28.0.4. Docker 29 uses the containerd image store by default (docker info shows io.containerd.snapshotter.v1), and a job that preloads images into a kind cluster with kind load docker-image stops in its setup step.
The message
From api7-ingress-controller #504, where every e2e and conformance job failed before any test ran:
ERROR: failed to load image: command "docker exec --privileged -i kind-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1 Command Output: ctr: rpc error: code = NotFound desc = content digest sha256:08d3b285700dffd91925e9d6328690d3336ac3adf6539aab94988bcecd727715: not found
When you pull a multi-platform image, such as nginx or busybox, the containerd image store keeps the whole index but only the runner's platform. kind load docker-image saves the image and imports it with --all-platforms, so the import looks for the other platforms and fails on the first missing blob. The kind issue is kind #3795, still open on November 4. api7 found that kind v0.33.0 and kind's main branch import the same way, so upgrading kind does not help. datum-cloud/test-infra #84 found that single-platform images built on the runner still load; the multi-platform images pulled from a registry are the ones that break.
Fix 1: load a single-platform archive
kind's known issues give this as the targeted fix. Save only the platform the nodes run, then load the archive:
docker image save --platform linux/amd64 --output /tmp/image.tar my-image:tag kind load image-archive /tmp/image.tar --name my-cluster
On an Arm64 runner, use linux/arm64. docker image save --platform needs Docker 28 or later, so a script that must also run on older Docker can keep kind load docker-image when the store is off; api7's Makefile does this. One catch from datum-cloud: for an image built on the runner, the archive from Docker 29.4.2 also carries the build attestation, and kind imports that as an extra image.
Fix 2: turn the containerd image store off
This puts Docker back the way it was on 24.04. Merge the setting into the runner's /etc/docker/daemon.json rather than overwriting it, restart Docker, and check that the store changed. From datum-cloud's step:
- name: Use Docker's classic image store
run: |
set -euo pipefail
sudo mkdir -p /etc/docker
cfg='{}'
if sudo test -s /etc/docker/daemon.json; then cfg=$(sudo cat /etc/docker/daemon.json); fi
echo "$cfg" | jq '.features["containerd-snapshotter"] = false' | sudo tee /etc/docker/daemon.json >/dev/null
sudo systemctl restart docker
if docker info --format '{{ .DriverStatus }}' | grep -q io.containerd.snapshotter; then
echo "the containerd image store is still on"
exit 1
fi
Put the step before anything pulls or builds an image. Images in the containerd store are not visible in the classic store after the restart: skywalking-helm #204 describes a pinned test tool whose restart like this hid a locally built image in SkyWalking's Java E2E, which then failed with "pull access denied". For the same reason datum-cloud kept the step in its own job and out of a shared action that callers may call after preparing images. spiderpool #5854 uses the same setting in its e2e jobs. It also fixed a retry that ran systemctl restart docker without sudo, which fails with "Access denied" on the new runners.
Telling it apart
docker info --format '{{ .DriverStatus }}'namesio.containerd.snapshotter.v1when the containerd image store is on.- The failing command is the
ctr ... images import --all-platformsthat kind runs inside the node, and the digest it cannot find belongs to another platform of the image. - Look at "Set up job": the Runner Image group says
ubuntu-26.04orubuntu-24.04. Until November 19 a re-run can land on 24.04 and pass, so a green re-run proves nothing.runs-on: ubuntu-24.04holds until GitHub retires that image. - The containerd image store also takes more disk, under
/var/lib/containerd: "No space left on device".
Other 26.04 failures
- "client version ... is too old" on Ubuntu 26.04: Docker 29 refuses clients below API 1.40, so older Testcontainers fails with "Could not find a valid Docker environment".
- /etc/sysctl.conf on Ubuntu 26.04: kind clusters also run out of inotify instances, with "too many open files".
- bubblewrap on Ubuntu 26.04: a sandbox inside
bwrapfails under AppArmor'sbwrap-userns-restrict. - "coreutils: unknown program": most core tools are uutils.
- sudo -E on Ubuntu 26.04:
sudois sudo-rs, which ignores-E. - apt on Ubuntu 26.04: renamed and removed packages.
See failed jobs from IntelliJ IDEA
Pipeline Inbox for GitHub Actions Workflows, a plugin by Lesstep for JetBrains IDEs 2024.1 or later, lists the runs of all your repositories in an IDE tool window, with their jobs and steps, and your current branch's runs are in the status bar. Reading is free; re-running failed jobs, cancelling, running workflows and reviewing deployments are Pro (US$1.50 a month, with a 30-day trial).
GitHub and GitHub Actions are trademarks of GitHub, Inc.; Ubuntu is a trademark of Canonical Ltd.; Docker is a trademark of Docker, Inc.; Kubernetes is a registered trademark of The Linux Foundation; IntelliJ IDEA is a trademark of JetBrains s.r.o. This page is not affiliated with, sponsored or endorsed by GitHub, Inc., Canonical, Docker, Inc., The Linux Foundation or JetBrains.
More: "The ubuntu-latest label will migrate to Ubuntu 26" · /etc/sysctl.conf on Ubuntu 26.04 · "No space left on device" · GitHub Actions plugins for IntelliJ IDEA compared · 日本語