- A+
所属分类:Kubernetes
日常巡检发现kubernetes node节点无法连接
[root@k8s-M1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.0.203 Ready <none> 335d v1.11.3 192.168.0.204 NotReady <none> 1y v1.11.3 192.168.0.205 NotReady <none> 1y v1.11.3 192.168.0.227 Ready <none> 271d v1.11.3 192.168.0.228 Ready <none> 250d v1.11.3
kubelet作用
每个节点上都运行一个 kubelet 服务进程,默认监听 10250 端口,接收并执行 master 发来的指令,管理 Pod 及 Pod 中的容器。
每个 kubelet 进程会在 API Server 上注册节点自身信息,定期向 master 节点汇报节点的资源使用情况,并通过 cAdvisor 监控节点和容器的资源。
kube-proxy作用
kube-proxy服务进程,这个进程可以看做service的透明代理和负载均衡器。其核心功能是将某个service的访问请求转发到后端的某个Pod上。对每一个TCP类型的service,kube-proxy都会在本地Node上建立一个socketserver来负责接收请求,然后均匀发送到后端某个Pod端口上。这个过程默认采用Round Robin负载均衡算法。
排查思路
检查node节点ip、网络联通性;
检查node节点kubelet、kube-proxy服务是否启动
检查kubelet启动日志
systemctl status glusterd systemctl status kubelet systemctl status kube-proxy systemctl status etcd systemctl status flanneld systemctl status docker
通过journalctl检查kubelet启动日志,发现bootstrap客户端证书已过期
journalctl -f -u kubelet
16:34:04.079812 28847 bootstrap.go:195] Part of the existing bootstrap client certificate is expired: 2019-09-11 16:35:00 +0000 UTC 16:34:04.079828 28847 bootstrap.go:56] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file 16:34:04.080581 28847 certificate_store.go:131] Loading cert/key pair from "/opt/kubernetes/ssl/kubelet-client-current.pem"
kubelete启动的时候,会向master注册证书,master节点查看证书状态是Pending
[root@k8s-M1 ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-DBVeWLnbaj7xv8OMh1AkE6rVomW9HAsyFvB3oWqkLwg 2h kubelet-bootstrap Pending node-csr-t3LiqtZi9X7FdKciwE9UZ6tYCQqn9UelhF2oZy8V8_M 2h kubelet-bootstrap Pending
批准证书
[root@k8s-M1 ~]# kubectl certificate approve node-csr-DBVeWLnbaj7xv8OMh1AkE6rVomW9HAsyFvB3oWqkLwg [root@k8s-M1 ~]# kubectl certificate approve node-csr-t3LiqtZi9X7FdKciwE9UZ6tYCQqn9UelhF2oZy8V8_M
重新检查证书,状态是已批准
[root@k8s-M1 ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-DBVeWLnbaj7xv8OMh1AkE6rVomW9HAsyFvB3oWqkLwg 2h kubelet-bootstrap Approved,Issued node-csr-t3LiqtZi9X7FdKciwE9UZ6tYCQqn9UelhF2oZy8V8_M 2h kubelet-bootstrap Approved,Issued
node节点查看新生成的证书
[root@k8s-node03 ~]# ll /opt/kubernetes/ssl/ total 32 -rw------- 1 root root 1679 Sep 11 2018 ca-key.pem -rw-r--r-- 1 root root 1359 Sep 11 2018 ca.pem -rw------- 1 root root 1273 Sep 12 2018 kubelet-client-2018-09-12-00-53-43.pem -rw------- 1 root root 1273 Sep 17 16:38 kubelet-client-2019-09-17-16-38-54.pem lrwxrwxrwx 1 root root 58 Sep 17 16:38 kubelet-client-current.pem -> /opt/kubernetes/ssl/kubelet-client-2019-09-17-16-38-54.pem -rw-r--r-- 1 root root 2181 Sep 12 2018 kubelet.crt -rw------- 1 root root 1679 Sep 12 2018 kubelet.key -rw------- 1 root root 1679 Sep 11 2018 server-key.pem -rw-r--r-- 1 root root 1651 Sep 11 2018 server.pem
重新检查node节点状态为Ready
[root@k8s-M1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.0.203 Ready <none> 335d v1.11.3 192.168.0.204 Ready <none> 1y v1.11.3 192.168.0.205 Ready <none> 1y v1.11.3 192.168.0.227 Ready <none> 271d v1.11.3 192.168.0.228 Ready <none> 250d v1.11.3