"TF400733: The request has been canceled: Request was blocked due to exceeding usage of resource"
Azure DevOps Services limits how much load one account, or one pipeline, can put on the shared service. Past the limit it first delays requests, then blocks them with HTTP 429 and this message, naming the resource and the namespace that ran out. It is about the account's recent usage, not about the request itself: once the usage drops, the delays stop within five minutes. From Microsoft's Rate and usage limits and Monitor usage.
The limit
- Load is counted in TSTUs (Azure DevOps throughput units), a blend of database, compute and storage use. One TSTU is the average load of a typical user over five minutes; normal activity stays at 10 or fewer per five minutes.
- The limit is 200 TSTUs in any sliding five-minute window, per account, and the same for each pipeline, counted separately (self-hosted agents count too: cloning and sending logs is load).
- Delays range from a few milliseconds to 30 seconds a request, and stop within five minutes once usage drops. When a request is delayed a lot, the account gets an email and a warning banner on the web; for the build service and other accounts without an email address, the Project Collection Administrators get the email.
- An identity that needs more can get a higher limit with the Basic + Test Plans access level (and only that one), charged while it is assigned.
- These limits are documented for Azure DevOps Services. The same wording also appears in public reports from the Visual Studio Marketplace when publishing (vsmarketplace #2256, VSID Concurrency) and from GitHub Actions' cache in 2020 (#68); this page is about Azure DevOps itself.
The warning before the block
Azure DevOps sends these response headers, all but X-RateLimit-Delay before requests start being delayed, so a client can slow down in time:
Retry-After: seconds to wait before the next request. It can come with HTTP 200.X-RateLimit-Resource: which service and threshold; meant for people to read, not for code.X-RateLimit-LimitandX-RateLimit-Remaining: TSTUs allowed, and left before delays start (0 once delayed or blocked).X-RateLimit-Reset: when the tracked usage would be back to 0 if all use stopped (Unix time).X-RateLimit-Delay: how long this request was delayed, in seconds.X-RateLimit-Cost, when present: the TSTUs this request used.
Finding what used it
Organization settings, Usage (Project Collection Administrators only) lists the requests of the last hour by default: by user, by pipeline, top user agents, top commands, with each request's TSTUs, delay and status (Normal, Delayed, Blocked). It centers on the 30 minutes before and after the first delayed request. The user agent and the command usually point at the script or tool.
Keeping a script or tool under it
- On 429, wait the
Retry-Afterseconds before the next request; on a 200 withRetry-After, do the same before the next one. - Watch
X-RateLimit-Remainingand spread bursts out. - Ask for less: one call for a list instead of one call per item,
$topand filters instead of reading everything, and longer intervals for polling. - Give a heavy automation its own identity, so its usage does not block the people who share the account.
In IntelliJ IDEA, Rider and the other JetBrains IDEs
Azure DevOps Pull Requests is a pull request plugin for JetBrains IDEs 2024.1 or later and Android Studio. Its list refresh, every 5 minutes while the tool window is open, reads the pull requests and the comment threads of up to 100 of them, 4 at a time. When Azure DevOps blocks a call, the plugin says so with Azure DevOps's own message, sends nothing more to that organization for the seconds Retry-After asks for, and does not read the rest of the discussions meanwhile; the refresh can be turned off in Settings | Tools | Azure DevOps Pull Requests. Reading is free; voting, commenting and completing are Pro, with a 30-day trial.
Get it on JetBrains Marketplace
Azure DevOps is a trademark of Microsoft Corporation; IntelliJ IDEA is a trademark of JetBrains s.r.o. This page is not affiliated with or endorsed by Microsoft or JetBrains.
More: "API resource location … is not registered on …" · TF400813 · global personal access tokens retired · 日本語