apt-get stuck for minutes on GitHub-hosted Ubuntu runners: a slow azure.archive.ubuntu.com, and why apt does not move to the next mirror
On GitHub-hosted Ubuntu runners, apt asks azure.archive.ubuntu.com first. When that mirror sends archives slowly, at tens of kB/s, apt keeps downloading from it: the runner image's timeout fires only when nothing arrives at all, so an install that usually takes a minute runs until timeout-minutes cancels the job. The image maintainers track it in runner-images #14823, open since September 30, 2026. It is the mirror, not the image: ubuntu-24.04 and ubuntu-26.04 runners see the same.
What it looks like
The package lists come down at full speed, and the archives do not (RustCrypto/crypto-bigint, ubuntu-26.04, October 25, 2026):
Fetched 3043 kB in 0s (10.8 MB/s) ... Need to get 21.3 MB of archives. Get:2 http://azure.archive.ubuntu.com/ubuntu resolute-updates/main amd64v3 libc6-dev-i386 amd64 2.43-2ubuntu2.4 [1605 kB] ... Fetched 21.3 MB in 8min 11s (43.3 kB/s)
- With a time limit on the step or the job, the log ends at a
Get:line fromazure.archive.ubuntu.com, then##[error]The operation was canceled. - There are no
Err:orIgn:lines and no retries, and a re-run usually passes. apt-get updatecan stop too. In libtmux/libtmux-go #109 (October 31), apt left the Azure mirror (Ign), went on tohttps://archive.ubuntu.com/ubuntu, and printed nothing afterGet:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]until the 10-minutetimeout-minutescancelled five jobs (also runner-images #15040).
Why apt does not move on
- The image writes three mirrors to
/etc/apt/apt-mirrors.txtand points the Ubuntu sources atmirror+file:/etc/apt/apt-mirrors.txt(configure-apt-sources.shin actions/runner-images):http://azure.archive.ubuntu.com/ubuntu/ priority:1 https://archive.ubuntu.com/ubuntu/ priority:2 https://security.ubuntu.com/ubuntu/ priority:3
- It sets
Acquire::Retries "1"andAcquire::http::Timeout "15"(and the same for https) in/etc/apt/apt.conf.d/zz-retries, so a mirror that stops answering is left after 15 seconds (configure-apt.sh). The 22.04 and 24.04 Arm64 images keep 3 and 30: ports.ubuntu.com serves them, and it is not in the list. - That timeout fires only when no data arrives. A mirror that keeps sending, slowly, never trips it, and apt has no setting for a minimum speed, so it never tries
archive.ubuntu.com(#14823). - On October 26 and 27, 2026, the westus and westus3 endpoints of
azure.archive.ubuntu.comserved archives at 40 to 75 kB/s from every runner region the maintainers tried, while the other regional endpoints served at 20 to 90 MB/s, and 24.04 runners there saw the same (runner-images #15142). GitHub said the problem was the mirror, and on October 27 that Canonical had it healthy again. Public issues from October 31 to November 3 still report apt steps that stall, on the Azure mirror and after leaving it, and #14823 is still open.
Why replacing the host name does nothing
A step that runs sed -i 's|azure\.archive\.ubuntu\.com|archive.ubuntu.com|g' over /etc/apt/sources.list and sources.list.d changes nothing on these runners: the sources do not name the host, they name the mirror list (snapotter-hq/SnapOtter #1808). Edit /etc/apt/apt-mirrors.txt instead.
A step that moves on
The maintainers' answer in #15142, with your own package list:
- name: Install system packages
timeout-minutes: 6
run: |
pkgs="ffmpeg imagemagick"
sudo apt-get update
if ! timeout 90 sudo apt-get install --yes --download-only $pkgs; then
sudo sed -i '/azure\.archive\.ubuntu\.com/d' /etc/apt/apt-mirrors.txt
sudo apt-get install --yes --download-only $pkgs
fi
sudo apt-get install --yes $pkgs
The first download gets 90 seconds. If it does not finish, the Azure line comes out of the mirror list and the download is tried again from archive.ubuntu.com; the last line installs what was downloaded. timeout-minutes is the backstop for the whole step. Set the 90 from your own packages: a healthy mirror serves tens of MB/s, so most installs need far less.
For an apt-get update that stops, give each try a limit and try again:
for i in 1 2 3; do
timeout 120 sudo apt-get update && break
[ "$i" = 3 ] && exit 1
done
Or keep away from the Azure mirror
- Remove its line before
apt-get update, with the samesed, so every fetch goes toarchive.ubuntu.com. You give up the Azure mirror's speed on the days it is healthy; a maintainer of uv listed "disabling the Azure mirrors entirely" among the mitigations they had added (#14823). - Keep the
.debfiles in the Actions cache, keyed on the package list, so only a cache miss reaches a mirror. The reporter of #14823 does this, and checks each file against the SHA256 in the package index.
Telling it apart
- This one: the
Get:lines nameazure.archive.ubuntu.com, the lists are fast and the archives slow, there is noErr:, a re-run passes, andubuntu-24.04does not help. - A mirror that does not answer:
Could not connect to azure.archive.ubuntu.com:80 ... connection timed out. Here the 15-second timeout does apply, and apt goes on to the next mirror in the list. E: Unable to locate packageorhas no installation candidate: a package name that changed on Ubuntu 26.04, see apt on ubuntu-latest (Ubuntu 26.04).- Only "The operation was canceled." with no apt step in sight: the five reasons for it.
The same on Azure Pipelines
Microsoft-hosted agents for Azure Pipelines are built from the same repository, actions/runner-images. If an Ubuntu job's log shows Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist, the same step works there, with timeoutInMinutes on the step instead of timeout-minutes.
See failed jobs from IntelliJ IDEA
Pipeline Inbox for GitHub Actions Workflows, a plugin by Lesstep for JetBrains IDEs 2024.1 or later, lists the runs of all your repositories in an IDE tool window, with their jobs and steps, so the jobs that were cancelled in an apt step stand out, and your current branch's runs are in the status bar. Reading is free; re-running failed jobs, cancelling, running workflows and reviewing deployments are Pro (US$1.50 a month, with a 30-day trial).
Get it on JetBrains Marketplace
GitHub and GitHub Actions are trademarks of GitHub, Inc.; Ubuntu is a trademark of Canonical Ltd.; Azure and Azure Pipelines are trademarks of Microsoft Corporation; IntelliJ IDEA is a trademark of JetBrains s.r.o. This page is not affiliated with, sponsored or endorsed by GitHub, Inc., Canonical, Microsoft or JetBrains.
More: apt on Ubuntu 26.04: renamed and removed packages · "The job has exceeded the maximum execution time" · "The ubuntu-latest label will migrate to Ubuntu 26" · 日本語