even in high-maturity Kubernetes clusters Pod pending is also ubiquitous.

If you randomly ask any DevOps engineer who uses Kubernetes to determine the most common mistakes that plague their nightmares, pod pending can be a very common problem (probably second only to CrashLoopBackOff).

Trying to push an update and seeing it get stuck can strain DevOps. Even if the solution is fairly simple, it’s important to find the cause of the pod hangs and understand the changes you need to apply (Kubernetes troubleshooting is rarely trivial).

In this article, we’ll shed light on the different scenarios that cause this problem, allowing DevOps teams to quickly find a solution and, most importantly, avoid it if possible.

What does Kubernetes Pod pending mean?

The lifecycle of a pod

in Kubernetes consists of several different phases:

when
• a pod is created, it starts with the Pending phase.

Once the pod

• is scheduled and the container has started, the pod enters the Running phase.

Most pods only take a few seconds to go from Pending to Running and spend most of their time in that state.

At this point, the pod has been accepted by the Kubernetes cluster. However, one or more containers are not ready for external service. This includes the time it takes for pods to wait for scheduling and the time it takes to download container images over the network.

When a pod fails to advance from the PendingtoRunning phase, the lifecycle stops and the pod is retained until the problem preventing it from moving forward is fixed.

If we list pods using kubectl, we will see output showing the Kubernetes pod hang:

 $ kubectl -n troubleshooting get podsNAME                                           READY   STATUS    RESTARTS   AGEstress-6d6cbc8b9d-s4sbh                        0/1     Pending   0          17s

Unless we fix the problem, the pod gets stuck and won’t run.

Common reasons to troubleshoot Kubernetes pod pending There are several reasons to

prevent pods

from running, but we’ll describe three main

issues:

• scheduling issues: pods can’t be scheduled on any node.
• Image issues: There was a problem downloading the container image.
• Dependency issues: Pods require a volume, secret, or configMap to run.

The first is the most common, the last one is rare. Let’s elaborate on each case.

Scheduling issues caused Kubernetes pod pending

After creating a pod, the first thing a Kubernetes cluster does is try to schedule the pod to run on one of the nodes. This process is usually very fast, and pods are quickly allocated to nodes that have enough resources to run it.

To place it, the pods in the cluster are assigned to nodes with more unrequested resources and continue their happy and wonderful life, which is full of SLO-compliant replies to requests.

However, if this process works every time, there are several factors that can cause the cluster to fail to allocate pods.

Let’s review the most common.

There are not enough resources in any node to allocate pods Kubernetes uses a dispatch request to decide if there are

podsin the fits node.

The actual use of the resources does not matter, only the resources that other pods have already requested.

effective requests When a pod has enough requestable resources to participate in that pod’s memory and CPU, it is dispatched to a node. And the node must not have reached the maximum number of pods it can run.

When no node meets all the requirements of a pod, it remains in the Kubernetes pod pending state until some resources are released.

Unschedulable nodes

may become unschedulable due to different issues (node stress) or human behavior (node lockdown). These nodes don’t schedule any pods until the state changes.

Tainting and tolerating

taints is a mechanism in Kubernetes that allows us to limit the pods that can be assigned to different nodes. When a node has taint, only pods that match tolerance can run in that node.

This mechanism allows for special uses of Kubernetes, such as using different types of nodes for different workloads (nodes with GPUs, with different CPU/memory ratios, etc.).

Even if we describe each cause separately, scheduling problems are often caused by a combination of these problems. Typically, you cannot schedule because some nodes are full and others are polluted, or a node may fail to schedule due to memory pressure.

In order to find out what the scheduling problem is, you need to look at the events generated by the scheduler about the pod, which will detail what is preventing node allocation. We can use kubectl describe to view events, for example:

 $ kubectl -n troubleshooting describe pod stress-6d6cbc8b9d-s4sbhName:           stress-6d6cbc8b9d-s4sbhNamespace:      troubleshootingPriority:       0Node:            Labels:         app=stress                pod-template-hash=6d6cbc8b9dAnnotations:    Status:         PendingIP:IPs:             Controlled By:  ReplicaSet/stress-6d6cbc8b9dContainers:  stress:    Image:      progrium/stress    Port:           Host Port:       Args:      --cpu      1      --vm      2      --vm-bytes      150M    Limits:      cpu:     300m      memory:  120000Mi    Requests:      cpu:        200m      memory:     100000Mi    Environment:       Mounts:      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-snrww (ro)Conditions:  Type           Status  PodScheduled   FalseVolumes:  kube-api-access-snrww:    Type:                    Projected (a volume that contains injected data from multiple sources)    TokenExpirationSeconds:  3607    ConfigMapName:           kube-root-ca.crt    ConfigMapOptional:       
     DownwardAPI:             true
QoS Class:                   BurstableNode-Selectors:              
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                             node.kubernetes.io/unreachable:NoExecute op=Exists for  300sEvents:  Type     Reason            Age                   From               Message  ----     ------            ----                  ----               -------
   Warning  FailedScheduling  4m17s (x41 over 34m)  default-scheduler  0/5 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 4 Insufficient memory. 

We can see the exact reason in the message in the output

:

0/5 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't 't tolerate, 4 Insufficient memory. 

• one of the nodes is contaminated.
• Four of the nodes do not have enough requestable memory.

To solve this problem, we have two options:

• reduce the resource request size in the pod definition.
• Increase the capacity of your cluster by adding more nodes or increasing the size of each node.

If you are updating a workload that is currently running, there is another important factor to consider: upgrade strategy.

Because of this strategy, Kubernetes can allow workloads to create more pods than usual during the update process, keeping old pods for a while when creating new pods. This means that the workload may request more resources than expected over a period of time. If the cluster does not have enough spare resources, the update will be blocked, leaving some pods pending until the process is unblocked (or the rollback timeout stops the update).

Due to image issues, once a pod is allocated in a

node, kubelet will try to start all the containers in the pod. To do this, it will try to download the image and run it.

There are several errors that prevent the image from being downloaded:

• wrong mirror name.
• Wrong image label.
• Wrong storage repository.
• Storage warehouses require authentication.

Kubernetes pods hang due to dependency issues

Before the pod starts, the kubelet will try to check all dependencies with other Kubernetes elements. If one of these dependencies cannot be satisfied, the pod will remain suspended until the dependency is satisfied.

In this case, kubectl

will display pods like this:

$ kubectl -n mysql get pods NAME READY STATUS RESTARTS AGE mysql-0 0/1 ContainerCreating 0 97s

in the event, We can see the following

:

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled    3m19s                default-scheduler  Successfully assigned mysql/mysql-0 to ip-172-20-38-115.eu-west-1.compute.internal
   Warning  FailedMount  76s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[kube-api-access-gxjf8 data config]: timed out waiting  for the condition

  Warning  FailedMount  71s (x9 over 3m19s)  kubelet            MountVolume.SetUp failed for volume "config" : configmap "mysql"  not foundThe

Message column will give you enough information to be able to pinpoint the missing element. Common causes are:

• ConfigMap or Secret has not been created, or the name provided is incorrect.
• The volume cannot be mounted in a node because it has not been freed by another node. This especially happens during the update of statefulset, which must have the same volume mounted as the old pod.

Conclusion

Understanding that the reason pods remain in this Pending phase is key to securely deploying and updating workloads in Kubernetes. Being able to locate issues quickly and speed up deployment will save you some headaches and reduce downtime.