Your OpenShift node fails to restart. Have you checked your PodDisruptionBudget?

Pam Andrejko, Barry Mosakowski

 

When running an OpenShift cluster, sometimes you need to update your nodes, for example if you upgrade to a new version of OCP,  or apply a change with an oc patch image.config.openshift.io/cluster command, or apply an ImageContentSourcePolicy.

Changes like this often require all the nodes in the cluster to be restarted.

The Problem 

Occasionally, one of the nodes fails to restart which is problematic, because the updates need to be pushed to every node, and obviously every node in the cluster needs to be Schedulable and Ready.

How do you know if you have this problem?

Run the command:

oc get machineconfigpool

Monitor the MACHINECOUNT column which shows the total number of nodes in your cluster and compare it to the values in the READYMACHINECOUNT column:

NAME     CONFIG                  MACHINECOUNT   READYMACHINECOUNT  
master   rendered-master-xxxx         3               0                 

worker   rendered-worker-yyyy         6               0                 

As the nodes are updated and restarted, you can see the values in the READYMACHINECOUNT column increment. This takes some time for each node to complete but sometimes you will notice it gets stuck and is not progressing.

In this case,  run the command:

oc get nodes 

to view which node is stuck. Its status will be “SchedulingDisabled”.  For example, in the following output, the node named compute-02.ps-cp4d.com has been stuck in this state for a long time:

NAME                                      STATUS                     
compute-00.ps-cp4d.com         Ready                      
compute-01.ps-cp4d.com         Ready                      
compute-02.ps-cp4d.com       Ready,SchedulingDisabled   
control-plane-00.ps-cp4d.com  Ready                      
control-plane-01.ps-cp4d.com  Ready                      
control-plane-02.ps-cp4d.com  Ready   

Why is this occurring?

One cause of this behavior can be the Kubernetes PodDisruptionBudget policy for one or more of the pods running on the node.  For High Availability reasons, a pod disruption budget is used to enforce a minimum number of instances of the pod be running at any given time on the cluster. When that condition is not met, the pod disruption budget prevents the node from restarting because the pod cannot be terminated and therefore the node cannot be drained.

How can I determine if the PodDisruptionBudget is causing my problem?

If you run the command: 

  oc adm drain <node> --delete-local-data --ignore-daemonsets=true  --disable-eviction=false

and it fails with an error similar to: 

error when evicting pod "<pod-name>" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Then you can be sure that the pod disruption budget is the cause of this node restart failure.

How to resolve the issue?

1. From your OCP console, navigate to Compute > Nodes and open the node whose status is SchedulingDisabled.
   
2. Click Pods and search for the pod listed in the error in the previous step.  
3. From the pod action menu on the right, click Delete Pod.  

Repeat these steps for each pod listed in the error message above.  

Wait a few minutes for the pod or pods to be deleted and redeployed on another node, then OpenShift can  reset the node as schedulable and mark it as Ready.  Problem solved!

 

 

 

 

2.