The Essential Cheatsheet to Troubleshooting ImagePullBackOff with Examples

Imagine you have a perfectly working Kubernetes cluster, and when everything seems on course, you get an “ImagePullBackOff” error. Although this is a popular issue in Kubernetes, understanding and troubleshooting the root cause can be a real headache. 

Kubernetes is an open-source container orchestration platform originally developed by Google. Within the last few years, it has gained immense popularity among developers, achieving a 92% market share compared to other container orchestration tools. 

However, despite its booming popularity, Kubernetes has its quirks and challenges. 

So, in this article, we will discuss what ImagePullBackOff is, why it happens, and several tips you can follow to troubleshoot it with ease.

What is ImagePullBackOff?

The demand for container-based application deployment over serverless is more than just a fleeting web development trend. The ImagePullBackOff error is a common issue in Kubernetes that happens when the kubelet agent struggles to fetch a container image from a registry.

When the Kubelet tries to start a pod, it first needs to pull the specified container image. If that pull fails, the kubelet will repeatedly retry pulling the image with an exponential backoff delay, leaving the pod stuck in an ImagePullBackOff status and unable to start correctly.

The error affects developers, DevOps engineers, or anyone operating a Kubernetes cluster trying to run pods that rely on container images stored in registries. Resolving it requires diagnosing the root cause of the failed image pull.

If you’re a beginner with Kubernetes, you can read our step-by-step guide to seamless Kubernetes deployment first. 

Why Does ImagePullBackOff Happen?

The ImagePullBackOff error occurs when the Kubernetes kubelet agent fails to pull the container image for a pod from a registry. There are several potential reasons this can happen:

• The image name or tag is typed incorrectly in the pod spec, so the image can’t be found.
• Authentication to access a private image registry fails due to missing or invalid credentials in the Kubernetes cluster.
• The network connection between the Kubernetes cluster and the registry is interrupted or too slow.
• The container registry has rate limiting in place, and the Kubernetes cluster has exceeded the limit.
• The image manifest is not found in the repository, meaning the image tag or digest requested has been deleted or moved.

In all these cases, the kubelet will retry pulling the image with exponential backoff delays of up to five minutes. But, since the root cause is still present, the pod remains stuck in the ImagePullBackOff status, unable to start the container.

Of course, when you can troubleshoot ImagePullBackOff and your Kubernetes containers are running smoothly, you’ll still need to prioritize container security.

The Essential Cheatsheet to Troubleshooting ImagePullBackOff – With Examples

As mentioned, there are multiple reasons behind the ImagePullBackOff error. Here are a few tips to troubleshoot and fix it with minimum effort to keep up the speed of development.

1. Check the image name and tag for typos

Typos in the image name or tag specified in the pod spec are among the most common causes of ImagePullBackOff. For example, a pod defined like the one below will fail with ImagePullBackOff if there is no image with the name “myimage” or a typo in the name or tag.

apiVersion: v1
kind: Pod
metadata:
 name: myapp
spec:
 containers:
 - name: myapp
  image: myimage/myimage:latest

Troubleshooting tip:

Always double-check that your pod spec’s image name and tag match the images in the container registry.

2. Check kubelet and container runtime logs

Kubelet and container runtime logs are literal treasures for developers trying to troubleshoot Kubernetes errors.

Kubelet logs:

The kubelet log located at /var/log/syslog on nodes contains detailed information on the kubelet operations and errors. Look through these logs for messages related to pulling container images. For example, you may see errors like:

Error response from daemon: pull access denied for myregistry/myimage, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

This message indicates the kubelet failed to convince the container registry to hand over the image. It’s a classic example of authentication and authorization failure during the image pull.

Container runtime logs:

The container runtime logs, such as Docker, also record errors during image retrieval. The Docker logs may contain more specific connectivity, authorization, or rate-limiting errors. For example:

toomanyrequests: Too Many Requests.

The above message indicates that there is a rate-limiting issue. It’s like the registry is telling you, “Slow down! You’re asking for images too quickly.” Carefully examining those logs can give you crucial information to pinpoint the root cause and save significant time on debugging.

3. Verify network connectivity to registries

Kubernetes nodes need network access to pull images from public or private registries. If your nodes can’t establish this connection, it could be due to some culprits like firewall rules, security group restrictions, or network policies blocking the path.

Troubleshooting tip:

First, try manually pulling the image from a node using docker or podman.

docker pull myregistry.example.com/myimage:1.0

If this fails with network errors like “timeout” or “connection refused”, then there are connectivity issues between the node and the registry. Once you verify there is a network error, try the below steps:

• Ensure ports 443 or 80 are open for the registry URL/IP in node security groups.
• Disable or configure network policies to allow egress access to the registry.
• Try tracing the route to the registry from a node with traceroute to check for firewall blocking.

You can also run a diagnostic pod that only attempts to access the registry URL:

kubectl run test --image=busybox --restart=Never --rm -it -- wget -O- http://myregistry.example.com

A successful connection will return the HTML of the registry’s homepage, while errors indicate network policy or firewall blocks from pods.

4. Check for insufficient disk space

Kubelet requires adequate disk space on nodes to perform tasks like pulling and extracting container images. If the available disk space is running low, it can lead to failures during image pulls.

You can check the free disk space on nodes with df -h command. This command must be executed on the node level, and you need relevant permission to access the nodes.

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G   5.5G   13G  31% /
tmpfs            1.9G     0  1.9G   0% /dev/shm
/dev/sdb         100G   50G   50G  50% /data

Troubleshooting tip:

• Free up space by removing old images.
• Add additional storage spaces.

5. Watch for registry rate limit errors

Public container registries like Docker Hub use a set of rules called rate limits to prevent abuse of their free tiers. Mainly, these limits apply to anonymous or unauthenticated users, restricting the number of images that can be pulled within a specific time. If a Kubernetes cluster exceeds this rate limit, the registry may reject image pull requests, causing the kubelet to report ImagePullBackOff errors.

Troubleshooting tips:

• Check kubelet and container runtime logs for explicit errors related to exceeding rate limits or being blocked by the registry. For example, Docker Hub enforces a limit of 100 image pulls for anonymous users and 200 for authenticated free accounts every six hours.
• If you have hit a rate limit, you can wait for the limit to reset, like a traffic light turning green again.
• Cache commonly used images in a local private registry to reduce external pulls.
• Consider importing required images into your private registry if you continually hit rate limits on public repos.

6. Using private container registry without authentication credentials

If your Kubernetes cluster pulls images from a private registry, you need to provide credentials to authenticate the cluster with the registry. If you don’t use proper credentials, image pulling will fail with the ImagePullBackOff error.

Troubleshooting tip:

Create a Secret with credentials for the private registry.

kubectl create secret docker-registry regcred \
  --docker-server=myprivateregistry.com \
  --docker-username=myuser \
  --docker-password=mypassword

Add the Secret to the Pod or ServiceAccount:

apiVersion: v1
kind: Pod
...
imagePullSecrets:
- name: regcred

Restart the Pod / Deployment

7. Restart the kubelet service

As a last resort, restarting the kubelet service on problematic nodes can clear failed image pull operations or connectivity issues.

sudo systemctl restart kubelet

After a restart, kubelet will initiate a fresh attempt to pull the required container images.

Ship Confidently by Safeguarding Your Kubernetes Clusters

ImagePullBackOff is a frustrating issue developers often face when working with Kubernetes clusters. We’ve just discovered seven ways to troubleshoot it, but ImagePullBackOff can still be a huge headache for you. That’s why you should take a proactive approach to securing your clusters and prevent these errors from happening. 

With Spectral’s automated scanning engine, you can detect misconfigurations, exposed secrets, and policy violations before they cause significant damage. Spectral integrates with CI/CD pipelines for rapid feedback on manifests, charts, and IaC before they deploy. Most importantly, developers gain the confidence to ship faster, knowing Spectral has their back and protects Kubernetes secrets.

Create your free account today to see how Spectral improves Kubernetes cluster security.