Azure Pipelines: "No space left on device" on a hosted agent
A job on a Microsoft-hosted agent has the disk of a fresh virtual machine, with Visual Studio, SDKs and other tools already on it, and whatever space is left. Source code, packages, Docker images and build output all go into that space. When it runs out, the step that writes next fails with No space left on device, and Microsoft does not give hosted agents more disk.
When the failing build is a pull request's build validation, the pull request waits on it: Azure DevOps Pull Requests, a plugin for JetBrains IDEs, lists the pull request's branch policies with their state in IntelliJ IDEA or Rider, and with Pro runs a failed build policy again once the fix is pushed. Get it on JetBrains Marketplace
What it looks like
##[warning]Free disk space on / is lower than 5%; Currently used: 96.12%. The agent checks while the job runs and prints this when free space is 5% or less of the disk (the threshold is a constant in the agent's source); the job goes on until a write fails.System.IO.IOException: No space left on devicefrom a .NET tool,write /var/lib/docker/...: no space left on devicefrom Docker, orENOSPC: no space left on devicefrom Node.js. On Windows agents the same thing readsThere is not enough space on the disk.- If the disk fills so far that the agent can no longer write its own logs, the run may end with We stopped hearing from agent instead.
How much space a hosted agent has
- Microsoft's documentation of Microsoft-hosted agents lists Windows and Linux agents as virtual machines with 2 cores, 7 GB of RAM and 14 GB of SSD, and says every agent has at least 10 GB of free disk space for the pipeline.
- The same page says Microsoft cannot honor requests for more disk on hosted agents. The ways out it gives: make the pipeline fit, or move to self-hosted agents, scale set agents or Managed DevOps Pools.
Freeing space on the Ubuntu images
- Azure Pipelines uses the images from the runner-images repository, the same ones GitHub Actions runs, so the large preinstalled toolchains are in the same directories. Delete those your later steps do not use, at the start of the job:
- script: | df -h / sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup docker image prune --all --force df -h / displayName: Free disk space - Keep what the build needs: an Android build needs
/usr/local/lib/android, and a .NET build needs/usr/share/dotnetunless aUseDotNet@2step installs the SDK again. What each directory freed on a runner we measured is on the GitHub Actions version of this page. - The two
df -h /lines show in the log how much each run starts with and how much the cleanup won, so you can see if the job is close to the edge.
Using less
- Check out only what the build needs: a shallow fetch (
fetchDepth: 1on thecheckoutstep) skips the history, and Git LFS files are fetched only withlfs: true. - Remove what a step leaves behind once it is published: after
PublishPipelineArtifact@1uploads a folder, a later step that does not need it can delete it. - Split a job that builds several large images or targets into jobs, since each job gets a fresh agent with the full free space.
- Check what fills the disk with
du -sh $(Agent.BuildDirectory)/* | sort -hnear the end of a run that still succeeds.
When it does not fit
- On a self-hosted agent, a scale set agent or a Managed DevOps Pool, you choose the machine and its disk. Managed DevOps Pools keep Microsoft managing the machines while the size is yours; a self-hosted agent also keeps its work folder between runs, so add a cleanup there too.
Azure DevOps pull requests in IntelliJ IDEA and Rider
Azure DevOps Pull Requests, a plugin by Lesstep for JetBrains IDEs 2024.1 or later and Android Studio, reads the organization, project and repository from your git remote and lists the pull requests that wait for your review in an IDE tool window, on Azure DevOps Services and on Azure DevOps Server. It shows the branch policies of a pull request with their state, so a required build that has not passed is next to its pull request, and with Pro you run a failed or expired build policy again from there. The review inbox is free; voting, commenting, completing and creating are Pro, with a 30-day trial.
Get it on JetBrains Marketplace
Azure DevOps and Azure Pipelines are trademarks of Microsoft Corporation; Docker is a trademark of Docker, Inc.; IntelliJ IDEA and Rider are trademarks of JetBrains s.r.o. This page is not affiliated with or endorsed by Microsoft, Docker or JetBrains.
More: We stopped hearing from agent · GitHub Actions: No space left on device · Bitbucket Pipelines: No space left on device · 日本語