BPF on syscalls:sys_enter_* misses events on GitHub's Ubuntu runners (kernel 7.0.0-1017-azure)
The GitHub-hosted ubuntu-24.04 image moved from kernel 6.17.0-1022-azure to 7.0.0-1017-azure with image 20261018, and ubuntu-26.04 has had 7.0.0-1017-azure since about image 20261012.163.1. ubuntu-latest is one of the two either way (it is moving to 26.04). That kernel carries a backport of "tracing: perf: Fix stale head for perf syscall tracing" (2c2b322acdcc, from Linux 7.2) without the change it depends on. It returns early from perf_syscall_enter() before it checks for BPF programs, so a BPF program on a syscalls:sys_enter_* tracepoint, for syscalls such as execve, openat and write, runs only on the CPUs where a perf event for that tracepoint was opened (runner-images #15003).
What it looks like
Nothing fails to load or attach. The program runs, but only for calls made on one CPU, usually CPU 0, so on a 4-CPU runner about three quarters of the events are missing. Ring buffer drop counters and recursion_misses stay at 0. Tests that wait for an event time out, and which ones fail changes from run to run, because it depends on the CPU a thread happened to run on.
Where the reports came from
- Inspektor Gadget:
TestTraceExecandTestTraceOpenfailed on every integration job that landed on the new 24.04 image, against 7 unrelated failures in 111 jobs on the previous one. The project moved those jobs toubuntu-22.04(inspektor-gadget #5891). - tinytap:
writeevents were lost on every CPU but CPU 0 on 7.0.0-1017-azure.bpftracealone counted 2000writecalls on CPU 0 and 0 on CPUs 1 to 3, while ftrace counted all of them;readandwritevwere complete (tinytap #440). It also moved its capture jobs toubuntu-22.04.
Fix: run the eBPF jobs on ubuntu-22.04
jobs:
ebpf-tests:
runs-on: ubuntu-22.04
The 22.04 image is on kernel 6.8.0-1070-azure (image readme), which does not have the backport. Pinning ubuntu-24.04 does not help: that is the image with the new kernel. Move only the jobs that load BPF programs, until a fixed kernel reaches the images or GitHub retires 22.04.
The runner-images maintainers reproduced it, and said their build process cannot pin an older kernel and they will not pause image updates for it. The fix has to come from Ubuntu: a revert has been sent for Ubuntu's kernel (SRU patch, October 26), and Launchpad bug 2170225 was still New on November 4. The upstream 7.0 and 7.1 stable series are at end of life, so they will not get the fix.
Telling it apart
- Print
uname -rin the job. 7.0.0-1017-azure has the bug; tinytap also saw missingwriteevents on 7.0.0-1019-azure and 7.0.0-1014-aws, and linux-azure 7.0.0-1020.21 does not have the revert either, as of November 4 (tinytap #444). - Count one syscall with
bpftraceontracepoint:syscalls:sys_enter_*while pinning the caller to different CPUs withtaskset. On an affected kernel only one CPU's calls are counted, while ftrace still sees all of them. - A re-run does not help: every current 24.04 and 26.04 image has this kernel.
- It is not only GitHub's runners: a self-hosted runner or a server on one of Ubuntu's 7.0 kernels can have the same backport, and the tool misses events there without any error.
Other 26.04 failures
- PyCQA/bandit-action on Ubuntu 26.04: the action sets up Python 3.8 (3.9 on v1.0.1) with no input to change it, and setup-python has builds for 26.04 only from 3.10; run Bandit in your own steps.
- Postgres never starts in Docker on Ubuntu 26.04: newer kernels such as 7.0.0-42 run container processes as
docker-default//&unconfined, so they cannot signal each other andgosuhangs at 100% CPU; reported on self-hosted runners so far. - wg-quick "Permission denied" on Ubuntu 26.04: new AppArmor profiles let
wg-quickandwgread configuration and key files only under/etc/wireguard/, so awg0.confin the workspace fails even withsudo. - apt on Ubuntu 26.04: renamed and removed packages.
- "coreutils: unknown program": most core tools are uutils.
See failed jobs from IntelliJ IDEA
Pipeline Inbox for GitHub Actions Workflows, a plugin by Lesstep for JetBrains IDEs 2024.1 or later, lists the runs of all your repositories in an IDE tool window, with their jobs and steps, and your current branch's runs are in the status bar. Reading is free; re-running failed jobs, cancelling, running workflows and reviewing deployments are Pro (US$1.50 a month, with a 30-day trial).
Get it on JetBrains Marketplace
GitHub and GitHub Actions are trademarks of GitHub, Inc.; Ubuntu is a trademark of Canonical Ltd.; Linux is the registered trademark of Linus Torvalds; IntelliJ IDEA is a trademark of JetBrains s.r.o. This page is not affiliated with, sponsored or endorsed by GitHub, Inc., Canonical, the Linux Foundation or JetBrains.
More: "The ubuntu-latest label will migrate to Ubuntu 26" · Postgres never starts in Docker on Ubuntu 26.04 · GitHub Actions plugins for IntelliJ IDEA compared · 日本語