Bitbucket Pipelines: "There was an unexpected system error that may have left your step half-completed"
The full message, next to the step's log, goes on: "To solve this error, check your step progress and rerun it if you see fit." Your commands did not fail. The container the step ran in went away in the middle of the build. Bitbucket does not know how far the step got, so it does not try again by itself: you look, then rerun.
Why the container went away
Atlassian's article on this message (Pipeline build failed with "unexpected system error...") says it is seen most often with self-hosted runners, and lists these causes among others:
- a power outage,
- a hardware failure,
- a network link interruption,
- the runner's host machine running out of memory or disk space.
A step that ran out of its own memory limit ends with a different message, "Container 'Build' exceeded memory limit." This one is about the machine underneath.
Check how far the step got
Pipelines cannot know which of your commands finished, so before you rerun, read the step's log and see the last command that ran. A step that deploys, publishes or migrates a database may have done half of it. Rerunning it is safe only if each command can run twice; otherwise finish or undo the half by hand first.
Check the runner host
Atlassian suggests watching memory, disk space and the network on the runner's host machine while a build runs, and reading the runner's log as well as the build log. On a Linux host:
df -h # disk space left free -m # memory and swap journalctl -k | grep -i "out of memory" docker system df # space used by Docker
journalctl -k prints the kernel's messages; "Out of memory: Killed process" there means Linux ended a process to free memory. On a Docker runner, old images, containers and build cache fill the disk over time; docker system prune removes stopped containers, unused networks, dangling images and unused build cache. Fix what you find, then rerun. If the error keeps coming back and the cause is not clear, Atlassian asks for a support ticket with the pipeline's URL and the runner's logs.
Rerun only the failed steps
From Atlassian's View your pipeline: on a failed pipeline, Rerun offers two choices.
- Rerun failed steps runs only the failed steps again, in the same pipeline, and adds the new run to its log. The Runs dropdown in the log switches between the runs.
- Rerun entire pipeline starts a new pipeline, with its own log.
Artifacts are kept for 14 days, so a step that uses artifacts can be rerun only within that time.
See the reason and run again from IntelliJ IDEA
Pipeline Inbox for Bitbucket Pipelines, a plugin by Lesstep for JetBrains IDEs 2024.1 or later, lists the pipelines of all your Bitbucket Cloud repositories in an IDE tool window. When Bitbucket ends a step with an error and gives its reason, the plugin shows the reason, the command that was running and its last lines, so you can see how far the step got. With Pro, Run again starts the same pipeline on the same commit from the IDE, as a new pipeline, like Rerun entire pipeline. The list, steps and logs are free; running, re-running and stopping pipelines are Pro (US$1.50 a month, with a 30-day trial).
Get it on JetBrains Marketplace
Atlassian, Bitbucket and Bitbucket Pipelines are trademarks of Atlassian; Docker is a trademark of Docker, Inc.; Linux is the registered trademark of Linus Torvalds; IntelliJ IDEA is a trademark of JetBrains s.r.o. This page is not affiliated with or endorsed by Atlassian, Docker or JetBrains.
More: The cache is not used · Pipelines did not run after a push · "Exceeded build time limit of 120 minutes" · "Container 'Build' exceeded memory limit." · "Container 'docker' exceeded memory limit." · "command not found" · Submodules not cloned · "We couldn't clone the repository" · "No online runner available" · "No space left on device" · "Step exceeded processing limits" · The artifact is missing in the next step · The build passed before and fails now · Build setup takes too long · "You've run out of build minutes" · "Permission denied to update branch" · "You have reached your pull rate limit" (Docker Hub) · "There is an error in your bitbucket-pipelines.yml" · "This pipeline was paused" (deployments) · Bitbucket Pipelines plugins for IntelliJ IDEA, compared · 日本語