ansible-test 2.22 and devel: "mount -o remount,rw /sys/fs/cgroup/ && pkill sleep" returned exit status 1 on GitHub's Ubuntu runners
ansible-test runs sanity, unit and integration tests of a collection in its default test container. It starts that container with env sleep 60 ; exec /sbin/init, and a privileged helper container then remounts the cgroup filesystem inside it and wakes it with pkill sleep. On GitHub's ubuntu-24.04 and ubuntu-26.04 runners, now on kernel 7.0.0-1017-azure, that helper often exits 1, and ansible-test stops before any test runs (ansible/ansible #87731, open).
What it looks like
Starting new "ansible-test-controller-ENwvCcfN" container. Run command: docker run ... --cgroupns private -dt ... quay.io/ansible/default-test-container:v2.23-2 sh -c 'env sleep 60 ; exec /sbin/init' Run command: docker run --pid host --privileged --name ansible-test-init-controller-ENwvCcfN --rm quay.io/ansible/ansible-test-utility-container:3.6.0 nsenter -t 2937 -m -p sh -c 'mount -o remount,rw /sys/fs/cgroup/ && pkill sleep' FATAL: Host job(s) failed. See previous error(s) for details. ERROR: Host DockerConfig(...) job failed: Command "docker run --pid host --privileged ... 'mount -o remount,rw /sys/fs/cgroup/ && pkill sleep'" returned exit status 1.
The log is from a run of the SAP collections (ansible-test-gh-action #138), shortened. The random suffix of the container names and the process ID after nsenter -t change from run to run.
Who it hits
- ansible-test from ansible-core 2.22 and devel. In the same workflows, 2.18 to 2.21 pass.
- Through the ansible-test GitHub Action, antsibull-nox or a plain
ansible-test ... --docker: the action and the tools only start ansible-test, so they all fail the same way. - Reported for many community and partner collections, among them community.dns, community.docker, community.routeros, community.sops, community.hashi_vault, community.openwrt, amazon.aws, the Checkmk and SAP collections and lowlydba.sqlserver (#417). Early on a re-run sometimes passed; since November 1, maintainers report jobs that fail after 5 to 10 attempts.
Why the kernel is the suspect
The failing jobs all ran on 7.0.0-1017-azure. The issue opened on October 21, after the 24.04 image moved from 6.17.0-1022-azure to that kernel. For a few days in late October the jobs passed again on images with the 6.17 kernel, and they started failing again when the 20261026 image brought 7.0 back. A maintainer of ansible-test saw the same errors on the project's own Azure Pipelines VMs after moving them to that kernel, and moved them back.
What in the kernel causes it is not known yet. One explanation in the issue is a race: pkill sleep runs before env has started sleep, finds nothing to wake and returns 1. A pull request that retries the wake for up to 30 seconds (ansible/ansible #87864) was closed without being merged, after a maintainer asked for the cause instead of a retry.
Fixes until there is a real one
Run the 2.22 and devel jobs on ubuntu-22.04, whose kernel is 6.8.0-1070-azure, and keep the others on ubuntu-latest. Use the name of your own matrix variable:
jobs:
sanity:
runs-on: ${{ contains(fromJSON('["stable-2.22", "devel"]'), matrix.ansible) && 'ubuntu-22.04' || 'ubuntu-latest' }}
A pull request in amazon.ai does this for its devel jobs (amazon.ai #90). Pinning ubuntu-24.04 does not help: that image has the new kernel too.
Or let the devel jobs fail without failing the workflow, as community.sonic and vmware.vmware did (continue-on-error: ${{ matrix.ansible == 'devel' }} on the job). That keeps the stable jobs, which still gate merges, but hides real devel failures until the issue is fixed.
Telling it apart
- The command that failed is the
ansible-test-init-controller-...helper withnsenterandpkill sleep, before any test output. uname -rin the job prints 7.0.0-1017-azure (or another 7.0 version).- The jobs for ansible-core 2.21 or older in the same workflow pass.
- Self-hosted runners and other machines on an Ubuntu 7.0 kernel can show it too, as ansible-test's own Azure Pipelines VMs did.
Other 26.04 failures
- "conda: command not found" on Ubuntu 26.04: the image has no Miniconda and
CONDAis empty, so steps that add$CONDA/binto the PATH, as GitHub's starter workflow and Microsoft's Azure Pipelines guide do, find no conda; install it with setup-miniconda or Miniforge. - "MongoDB cannot start: Linux kernel versions 6.19 and newer": mongo:8.0 and mongo:8.3 read the 7.0 azure kernel of the 24.04 and 26.04 images as 7.0.0 and exit at startup; use mongo:9.0, or
ubuntu-22.04until 8.0.37 and 8.3.16. - "Unable to locate package linux-modules-extra-7.0.0-1017-azure": Ubuntu publishes no extras package for the 7.0 azure kernel, on the 24.04 and 26.04 images; modules such as
vcanare inlinux-modules, already installed, so skip the install and runmodprobe. - BPF on syscalls:sys_enter_* on GitHub's Ubuntu runners: kernel 7.0.0-1017-azure, on the 24.04 and 26.04 images, runs these programs on one CPU only, so eBPF tests lose events without an error;
ubuntu-22.04is not affected. - Docker-in-Docker on Ubuntu 26.04: the legacy iptables modules are not loaded, so a dockerd inside a container that uses
iptables-legacyfails with "can't initialize iptables table `nat'".
See failed jobs from IntelliJ IDEA
Pipeline Inbox for GitHub Actions Workflows, a plugin by Lesstep for JetBrains IDEs 2024.1 or later, lists the runs of all your repositories in an IDE tool window, with their jobs and steps, so a matrix where only the 2.22 and devel jobs went red stands out, and your current branch's runs are in the status bar. Reading is free; re-running failed jobs, cancelling, running workflows and reviewing deployments are Pro (US$1.50 a month, with a 30-day trial).
Get it on JetBrains Marketplace
GitHub and GitHub Actions are trademarks of GitHub, Inc.; Ansible is a registered trademark of Red Hat, Inc.; Ubuntu is a trademark of Canonical Ltd.; Linux is the registered trademark of Linus Torvalds; IntelliJ IDEA is a trademark of JetBrains s.r.o. This page is not affiliated with, sponsored or endorsed by GitHub, Inc., Red Hat, Canonical, the Linux Foundation or JetBrains.
More: "The ubuntu-latest label will migrate to Ubuntu 26" · MongoDB on kernel 7.0.0-1017-azure · GitHub Actions plugins for IntelliJ IDEA compared · 日本語