Azure Pipelines: "We stopped hearing from agent": an agent out of memory, CPU or disk, a machine that restarted, or a lost network
The agent running the job stopped reporting to Azure Pipelines, so the job failed where it was. The message names the three ways it happens: something ended the agent process, the agent got no CPU, or its network was blocked. On a Microsoft-hosted agent it is usually the first two, a step that used up the machine's memory or disk; turn on system diagnostics and read the memory, CPU and disk that the log records for each step (a Stack Overflow question on this message has been seen almost 10,000 times).
When a pull request's build validation loses its agent this way, the required build stays failed until a new run passes: Azure DevOps Pull Requests, a plugin for JetBrains IDEs, lists the pull request's branch policies with their state in your IDE, so you see which required build failed next to the pull request. Get it on JetBrains Marketplace
What it looks like
##[error]We stopped hearing from agent Hosted Agent. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error.- On a Microsoft-hosted agent the name is
Hosted AgentorAzure Pipelines 3; on a self-hosted agent it is the name you gave the agent. The step that was running when the agent went quiet is the last one in the log.
See what the agent used, step by step
- Queue the run with Run pipeline → Enable system diagnostics, or set the variable
system.debugtotrue. The log then records the agent's resources for each step (Configure verbose logs). - Search the log for
Agent environment resources. Microsoft's example line:##[debug]Agent environment resources - Disk: D:\ Available 12342.00 MB out of 14333.00 MB, Memory: Used 1907.00 MB out of 7167.00 MB, CPU: Usage 17.23%. Memory close to the total, or almost no disk left, in the steps before the loss is the answer (Microsoft's troubleshooting for this message). - On Azure DevOps Server, Microsoft gives a PowerShell step that writes the machine's resources to a file uploaded with the logs (Capture resource utilization details).
A Microsoft-hosted agent that ran out of memory, CPU or disk
- Windows and Linux agents are virtual machines with 2 CPU cores, 7 GB of memory and a 14 GB SSD with at least 10 GB free for the job. On Linux, the steps run in a cgroup with 6 GB of physical memory and 13 GB in total (Microsoft-hosted agents).
- Microsoft does not give larger hosted machines. Make the step use less (fewer parallel workers in the test runner or the build tool, fewer Docker images kept on disk), or run the job on a self-hosted agent, scale set agents or Managed DevOps Pools on a machine of the size you need.
- When disk runs out, the step itself may fail first with
No space left on device; Microsoft's FAQ for that message points to the same 10 GB.
A self-hosted agent: the machine, the agent process, the network
- The machine went to sleep, restarted or lost its network, or something ended the agent process (the agent service was stopped, the system ran out of memory and killed it, security software ended it).
- The agent's own logs are in the
_diagfolder of the agent's directory (the agent's troubleshooting notes).Agent_*.logshows when the agent connected and how each job ended;Worker_*.logwith the run's date and time is the worker's log of that run (Agent diagnostic logs). - The resource lines from
system.debugwork on any agent, so they show a self-hosted machine running short too.
A step that restarts the machine
- A job runs inside the agent on that machine, so a step that restarts the machine (installing updates, a reboot command) ends the agent with it, and Azure Pipelines stops hearing from it. An answer on a Stack Overflow question about this suggests doing the restart from outside the job, for example over SSH or PowerShell remoting.
Telling it apart
The job running on agent Hosted Agent ran longer than the maximum time of 60 minutes: the time limit of the free Microsoft-hosted job, not a lost agent. See the page on Microsoft-hosted parallel jobs.- The job never got an agent and waits or fails with
No agent found in pool ... which satisfies the specified demands: that page.
Azure DevOps pull requests in IntelliJ IDEA and Rider
Azure DevOps Pull Requests, a plugin by Lesstep for JetBrains IDEs 2024.1 or later and Android Studio, reads the organization, project and repository from your git remote and lists the pull requests that wait for your review in an IDE tool window, on Azure DevOps Services and on Azure DevOps Server. It shows the branch policies of a pull request with their state, so a required build that has not passed is next to its pull request, and when a required policy has not passed, its Complete dialog offers auto-complete first. The review inbox is free; voting, commenting, completing and creating are Pro, with a 30-day trial.
Get it on JetBrains Marketplace
Azure DevOps and Azure Pipelines are trademarks of Microsoft Corporation; GitHub is a trademark of GitHub, Inc.; IntelliJ IDEA and Rider are trademarks of JetBrains s.r.o. This page is not affiliated with or endorsed by Microsoft, GitHub, Inc. or JetBrains.
More: no agent in the pool meets the demands · no Microsoft-hosted parallel job yet · a resource authorization issue · 日本語