"MongoDB cannot start: Linux kernel versions 6.19 and newer" on GitHub's Ubuntu runners
A mongo:8.0 or mongo:8.3 service container stops a second after it starts, and the tests that need it fail. MongoDB 8.0 and later crash on Linux 6.19 to 7.0.13 because of an rseq bug in the TCMalloc they ship (SERVER-121912), so newer builds refuse to start on those kernels, and from 8.0.30, 8.3.9 and 9.0 they let 7.0.14 and later through (SERVER-125742). The ubuntu-24.04 and ubuntu-26.04 images now boot kernel 7.0.0-1017-azure, which is built from upstream 7.0.14 and has the fix. But Ubuntu keeps the release string at 7.0.0 for the whole series, and mongod reads that string, so it refuses anyway (SERVER-136848).
What it looks like
The mongod log:
{"s":"F","c":"CONTROL","id":12257600,"msg":"MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information."}
In the job, depending on how MongoDB is started:
- A service container: "Initialize containers" fails with
##[error]Failed to initialize container mongo:8.0, or the runner reportsmongo service is healthy.and the tests then fail withMongoServerSelectionError: connect ECONNREFUSED 127.0.0.1:27017(runner-images #15005). - Docker Compose:
dependency failed to start: container ... is unhealthyhalf a second after the start (RMF-Block #229). - MongoDB from apt: DevStack installs 8.0.34, mongod exits with status 1 and Zaqar fails with
ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused(gophercloud #4080). - Testcontainers and similar tools: the container exits and the readiness check times out. golang-migrate's
mongo:8.0tests failed in every run since October 29, while 5.0, 6.0 and 7.0 passed in the same runs (golang-migrate #1463).
A re-run does not help any more: the runner-images maintainers said on November 3 that the latest 24.04 image, 20261101.360, has the same kernel, and the 26.04 images have had it since October.
Which MongoDB versions start
- The fix that reads Ubuntu's real kernel version, SERVER-131779, lists 9.0.4, 8.3.16 and 8.0.37 as fixed versions (on November 4; on October 20 it listed 8.3.14 and 8.0.35).
- On Docker Hub on November 4:
mongo:9.0andmongo:latestare 9.0.4, which passed 90 seconds of repeated ping and CRUD operations on the affected VMs in the runner-images maintainers' test.mongo:8.0is 8.0.34 andmongo:8.3is 8.3.13, both still refusing. One project tested 8.3.9, 8.3.11, 8.3.13 and 8.0.34 on the runners, and all exited with id 12257600 (keyset-kit #2). - MongoDB 7.0 is not affected.
Fixes
Move to 9.0, if your code and data allow a major upgrade:
services:
mongo:
image: mongo:9.0
ports:
- 27017:27017
Or run the jobs that need MongoDB on ubuntu-22.04, the workaround the runner-images maintainers give. Its kernel is 6.8.0-1070-azure, where 8.0 and 8.3 start:
jobs:
tests:
runs-on: ubuntu-22.04
Pinning ubuntu-24.04 does not help: that image has the new kernel. The maintainers said they will not pin or downgrade the kernel for this. When mongo:8.0 reaches 8.0.37 (or mongo:8.3 reaches 8.3.16), the 8.x tags start again on the new kernel; check the version a tag points to before you move back, since the listed fix versions have changed once.
Two projects report that mongo:8.0 starts on these kernels with GLIBC_TUNABLES=glibc.pthread.rseq=1 in the container's environment (debezium dbz #1945, maple-mcp #1). It changes how memory allocation runs inside mongod, so treat it as a stopgap for tests, not for production.
Telling it apart
- Print
uname -randcat /proc/version_signaturein the job. A 7.0.0 release string with 7.0.14 at the end of the signature is this case. Containers share the host's kernel, so the image of the container does not matter. - Look for id 12257600 in mongod's log. For a service container, the Stop containers step at the end of the job prints the container's log; for Docker Compose, run
docker compose logsin a step withif: failure(). - It is not only GitHub's runners: a self-hosted runner or a server on Ubuntu 26.04's own 7.0 kernel (for example 7.0.0-38-generic) refuses the same way.
Other 26.04 failures
- "conda: command not found" on Ubuntu 26.04: the image has no Miniconda and
CONDAis empty, so steps that add$CONDA/binto the PATH, as GitHub's starter workflow and Microsoft's Azure Pipelines guide do, find no conda; install it with setup-miniconda or Miniforge. - ansible-test 2.22 and devel: "mount -o remount,rw /sys/fs/cgroup/ && pkill sleep" returned exit status 1: on the 7.0 azure kernel of the 24.04 and 26.04 images the default test container often fails to start, and re-runs rarely help; run those jobs on
ubuntu-22.04. - "Unable to locate package linux-modules-extra-7.0.0-1017-azure": Ubuntu publishes no extras package for the 7.0 azure kernel, on the 24.04 and 26.04 images; modules such as
vcanare inlinux-modules, already installed, so skip the install and runmodprobe. - BPF on syscalls:sys_enter_* on GitHub's Ubuntu runners: kernel 7.0.0-1017-azure, on the 24.04 and 26.04 images, runs these programs on one CPU only, so eBPF tests lose events without an error;
ubuntu-22.04is not affected. - "client version ... is too old" on Ubuntu 26.04: Docker 29 refuses clients below API 1.40, so older Testcontainers fails with "Could not find a valid Docker environment".
- apt on Ubuntu 26.04: renamed and removed packages.
See failed jobs from IntelliJ IDEA
Pipeline Inbox for GitHub Actions Workflows, a plugin by Lesstep for JetBrains IDEs 2024.1 or later, lists the runs of all your repositories in an IDE tool window, with their jobs and steps, and your current branch's runs are in the status bar. Reading is free; re-running failed jobs, cancelling, running workflows and reviewing deployments are Pro (US$1.50 a month, with a 30-day trial).
Get it on JetBrains Marketplace
GitHub and GitHub Actions are trademarks of GitHub, Inc.; MongoDB is a trademark of MongoDB, Inc.; Ubuntu is a trademark of Canonical Ltd.; Linux is the registered trademark of Linus Torvalds; IntelliJ IDEA is a trademark of JetBrains s.r.o. This page is not affiliated with, sponsored or endorsed by GitHub, Inc., MongoDB, Inc., Canonical, the Linux Foundation or JetBrains.
More: "The ubuntu-latest label will migrate to Ubuntu 26" · BPF on kernel 7.0.0-1017-azure · GitHub Actions plugins for IntelliJ IDEA compared · 日本語