how to debug network latency issues in a Kubernetes cluster

Kubernetes that newbies must know Architecture

< how a href="https://mp.weixin.qq.com/s?__biz=MzIxMTE0ODU5NQ==&mid=2650248445&idx=1&sn=2cce59c7a9b521c897b4d864f2c8ac08&scene=21#wechat_redirect" > improved Flink K8s cluster resource utilization?

Kubernetes Node nodes are allowed to create up to 110 pods per default, and sometimes it may be necessary to control the number of pods running on node nodes due to system hardware issues.

That is: the maximum number of runnable pods of a node node needs to be adjusted.

Generally speaking, we only need to add the —max-pods parameter to the kubelet startup command, and then restart the kubelet service to take effect.

Restarting the kubelet does not affect existing running containers and does not cause the container to restart.

2Modify the limit on the number of pods launched

As you can see, the cluster currently has 3 node nodes, and the maximum number of pods that can run is the default value: 110.

Next, let’s modify the parameter value.

Modify

  1. log in to the Node node and view the kubelet startup file path
[root@VM-2-8-centos ~] # systemctl status kubelet

As you can see from the figure above, the startup file of the kubelet is /usr/lib/systemd/system/kubelet.service

As you can see, the Environment file references /etc/kubernetes/kubelet, so far, we only need to add the max-pods variable in /etc/kubernetes/kubelet, and then / usr/lib/systemd/system/ kubelet.service, add variables to the startup parameters.

  1. modification /etc/kubernetes/kubelet

strongly

recommends that when modifying the number of pods, if it is reduced (currently running 70 pods, 20 seen), it is strongly recommended to put it first The pods on the node are dispersed, and when the dispersion is finished, they are restarted, and finally the node is activated.

[root@VM-2-8-centos ~]# vim /etc/kubernetes/kubelet
# Add
MAX_PODS="--max-pods=61"
  1. modify /usr/lib/systemd/system/kubelet.service

Note: This variable must be added after the startup command to make the just defined -- max-pods=61 take effect.

[root@VM-2-8-centos ~]# systemctl  daemon-reload
[root@VM-2-8-centos ~]# systemctl restart kubelet

View the modification result

class=”rich_pages wxw-img” src=”https://mmbiz.qpic.cn/mmbiz_png/A1HKVXsfHNkJ3ic3WibhKsT7gevag0giba3fdkcX4KyHKaDqLLD6aQV1ic3uiaqsDAxt75jKN5ltZ7BVea4upTUNYVw/640?wx_fmt=png”>

you can see, The maximum number of schedulable pods for nodes that have been modified has been adjusted to 61. The configuration takes effect.

3 Create a new

monitoring view

Here, we can deploy Prometheus + Grafana to configure the monitoring view to better reflect the pod allocation rate of the nodes in the cluster.

PromQL:

sum(kubelet_running_pod_count{node=~"$node"}) by (cluster, node) / sum(kube_node_status_allocatable_ pods{node=~"$node"}) by (cluster, node)

effect display:

This article is reproduced from: “Nugget”, original text: https://tinyurl.com/35yuwtzp, the copyright belongs to the original author.