侧边栏壁纸
博主头像
运维匠博主等级

生活百般滋味,人生需要笑对

  • 累计撰写 24 篇文章
  • 累计创建 2 个标签
  • 累计收到 0 条评论

目 录CONTENT

文章目录

Kubernetes集群证书过期后,使用kubeadm重新颁发证书

admin
2024-04-27 / 0 评论 / 0 点赞 / 19 阅读 / 16982 字
温馨提示:
本文最后更新于 2024-04-27,若内容或图片失效,请留言反馈。部分素材来自网络,若不小心影响到您的利益,请联系我们删除。

Kubernetes集群证书过期后,使用kubeadm重新颁发证书

默认情况下使用kubeadm部署的kubernetes集群的证书一年内便过期,如果不及时升级证书导致证书过期,Kubernetes控制节点便会不可用,所以及时更新Kubernetes证书避免因证书过期导致集群不可用问题。

检查证书是否过期

在控制平面执行

kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 05, 2024 16:17 UTC   <invalid>       ca                      no      
apiserver                  Mar 05, 2024 16:17 UTC   <invalid>       ca                      no      
apiserver-etcd-client      Mar 05, 2024 16:17 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   Mar 05, 2024 16:17 UTC   <invalid>       ca                      no      
controller-manager.conf    Mar 05, 2024 16:17 UTC   <invalid>       ca                      no      
etcd-healthcheck-client    Mar 05, 2024 16:17 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  Mar 05, 2024 16:17 UTC   <invalid>       etcd-ca                 no      
etcd-server                Mar 05, 2024 16:17 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         Mar 05, 2024 16:17 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             Mar 05, 2024 16:17 UTC   <invalid>       ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 03, 2033 16:17 UTC   8y              no      
etcd-ca                 Mar 03, 2033 16:17 UTC   8y              no      
front-proxy-ca          Mar 03, 2033 16:17 UTC   8y              no 

根据返回信息可以知证书在Mar 05, 2024 16:17 UTC已经到期。接下来使用kubeadm更新证书,在更新证书之前先进行备份

备份相关证书文件的目录

在控制平面执行
养成备份的好习惯

cp -r /etc/kubernetes/ /tmp/backup/ # 静态pods配置以及证书
cp -r /var/lib/kubelet/pki/ /tmp/backup.crr #证书pem存放目录

轮换证书

kubeadm certs renew all   # 执行证书更新命令
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

根据返回信息告诉我们必须要重启一下kube-apiserver,kube-controller-manager, kube-scheduler and etcd,才能使用新的证书

重启服务

mv mv /etc/kubernetes/manifests/ /etc/kubernetes/manifests.bak

执行该命令后kube-apiserver,kube-controller-manager, kube-scheduler and etcd会慢慢停掉。等pod 停止掉以后在还原文件目录,恢复服务

mv /etc/kubernetes/manifests.bak/ /etc/kubernetes/manifests
说明:使用 kubeadm 构建的集群通常会将 admin.conf 证书复制到 $HOME/.kube/config 中, 如使用 kubeadm 创建集群中所指示的那样。 在这样的系统中,为了在更新 admin.conf 后更新 $HOME/.kube/config 的内容, 你必须运行以下命令:
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

检查新证书

kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Apr 27, 2025 02:29 UTC   364d            ca                      no      
apiserver                  Apr 27, 2025 02:29 UTC   364d            ca                      no      
apiserver-etcd-client      Apr 27, 2025 02:29 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Apr 27, 2025 02:29 UTC   364d            ca                      no      
controller-manager.conf    Apr 27, 2025 02:29 UTC   364d            ca                      no      
etcd-healthcheck-client    Apr 27, 2025 02:29 UTC   364d            etcd-ca                 no      
etcd-peer                  Apr 27, 2025 02:29 UTC   364d            etcd-ca                 no      
etcd-server                Apr 27, 2025 02:29 UTC   364d            etcd-ca                 no      
front-proxy-client         Apr 27, 2025 02:29 UTC   364d            front-proxy-ca          no      
scheduler.conf             Apr 27, 2025 02:29 UTC   364d            ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 03, 2033 16:17 UTC   8y              no      
etcd-ca                 Mar 03, 2033 16:17 UTC   8y              no      
front-proxy-ca          Mar 03, 2033 16:17 UTC   8y              no

更新完证书后,发现集群还是未能正常运行,查看kubelet日志,发现如下错误

Apr 27 10:59:04 master01 kubelet[10798]: E0427 10:59:04.395494   10798 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2024-03-05 16:17:31 +0000 UTC
Apr 27 10:59:04 master01 kubelet[10798]: E0427 10:59:04.395591   10798 run.go:74] "command failed" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

究其原因是因为Kubelet的证数没有更新。这种情况发生在手动执行了更新证数到期时间后导致的,kubeadm更新证数并不会更新到Kubelet的证数(实际上是客户端证书轮换失败)。
于是当kublet被重启后,就发生了证数不一致的问题。

Kubelet 客户端证书轮换失败

来源于kublet的文章Kubelet 客户端证书轮换失败原文如下:

默认情况下,kubeadm 使用 /etc/kubernetes/kubelet.conf 中指定的 /var/lib/kubelet/pki/kubelet-client-current.pem 符号链接来配置 kubelet 自动轮换客户端证书。如果此轮换过程失败,你可能会在 kube-apiserver 日志中看到诸如 x509: certificate has expired or is not yet valid 之类的错误。要解决此问题,你必须执行以下步骤:

1. 从故障节点备份和删除 /etc/kubernetes/kubelet.conf 和 /var/lib/kubelet/pki/kubelet-client*。

2. 在集群中具有 /etc/kubernetes/pki/ca.key 的、正常工作的控制平面节点上 执行 kubeadm kubeconfig user --org system:nodes --client-name system:node:$NODE > kubelet.conf。 $NODE 必须设置为集群中现有故障节点的名称。 手动修改生成的 kubelet.conf 以调整集群名称和服务器端点, 或传递 kubeconfig user --config (请参阅为其他用户生成 kubeconfig 文件)。 如果你的集群没有 ca.key,你必须在外部对 kubelet.conf 中的嵌入式证书进行签名。

3. 将得到的 kubelet.conf 文件复制到故障节点上,作为 /etc/kubernetes/kubelet.conf。

4. 在故障节点上重启 kubelet(systemctl restart kubelet),等待 /var/lib/kubelet/pki/kubelet-client-current.pem 重新创建。

5. 手动编辑 kubelet.conf 指向轮换的 kubelet 客户端证书,方法是将 client-certificate-data 和 client-key-data 替换为:

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

6. 重新启动 kubelet。

7. 确保节点状况变为 Ready

解决方法是复制/etc/kubernetes/admin.conf特定键的内容client-certificate-data并将client-key-data这些新字符串粘贴到/etc/kubernetes/kubelet.conf相同键下的文件中。然后重启一下kubelet
kubelet.conf原文如下

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1ETXdOakUyTVRjeU9Gb1hEVE16TURNd016RTJNVGN5T0Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS29iClA4OUdyWlFYR29mejRDMmY2SDF0QlB6dlNxWXM0MVpQdHl3eWlaaG5vNXdKdTNqSTZyRHlZdURSN200TmVSaFUKTjd4VUgvZmdHaTZDR2tsQ0Vnenp4ZWRRb2YrdWdVWlhISEJGS3dkZXNySmxUbk1SZkdHcld1OHovR2xsOTNuYgpzb2IrQ0lRTGRGSmZSMjBzUVJ3SDUrRHgrTnFkRDhTbFhROWNpZUZUaEhpdFdOblBsUlo1WkJsTmh3VXRheUhMCnFKT0dmQUE0VFBIQ01HVmxUVnUvcFVRcHBndWgycjRuSEgvdTlKcGV3M1BRS2ZHNlNENUZOcFNyWmVPWEQ0REwKUk9SbFVVL1FIOXhHZ1VMZlNVMjZLQU9wM0duYzNhRW1pQjRaRElsbHdBNS9XcGVPNm5remNKV2tqdGErOUFaTQpDS2hzeS9QRnRIOG52WDJhODdFQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZCOFAvOGh3ZldDR2xjZk5TU3FLQkFtSHBkRTNNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBR2xnelZCUWlyUm9DNFZXNHpRVQpEQldBMFRzNVRzaEdUdkpKekhCY1RSRm43aVhnakNhblBBWnNwWkw0NEpGZlQ1VUZOOFZFUUpCaHNDOE90UDBKCmhDdUlwVnk5MERUMkt2Tjc3Q0dHYVB4cUducXl2bjV0TFJ2ZU9RZWowa1NrTGZlYnVwZExLeUMyeGlOZnduUW8KUzlPMmNlU3E1cE9DQndWRzVGM3FFeTRoVkhUVHJuWjA0Q3dQOHNDdVJtQzE3dkRPRkpDTStlWG1iclFWYTVVMgpqSk1QOERhc2RYazFReCtSdzRDeTh5bUo0eXB4SjBHZmtFNVJnWVk3aVZ3dTFwdkFXWk4zdDJ0TDl5eDN2aU9sCkkrdm1PYTBJdjhNWTFsT3hHNm5nRDZBamNJWU1uSDVPTng1ZDBoa3p6YjRwbVl4eVUvWC9wRnM3WnhLaUhvSisKMUNnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://master01:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:master01
  name: system:node:master01@kubernetes
current-context: system:node:master01@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:master01
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pe
  

替换后如下:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1ETXdOakUyTVRjeU9Gb1hEVE16TURNd016RTJNVGN5T0Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS29iClA4OUdyWlFYR29mejRDMmY2SDF0QlB6dlNxWXM0MVpQdHl3eWlaaG5vNXdKdTNqSTZyRHlZdURSN200TmVSaFUKTjd4VUgvZmdHaTZDR2tsQ0Vnenp4ZWRRb2YrdWdVWlhISEJGS3dkZXNySmxUbk1SZkdHcld1OHovR2xsOTNuYgpzb2IrQ0lRTGRGSmZSMjBzUVJ3SDUrRHgrTnFkRDhTbFhROWNpZUZUaEhpdFdOblBsUlo1WkJsTmh3VXRheUhMCnFKT0dmQUE0VFBIQ01HVmxUVnUvcFVRcHBndWgycjRuSEgvdTlKcGV3M1BRS2ZHNlNENUZOcFNyWmVPWEQ0REwKUk9SbFVVL1FIOXhHZ1VMZlNVMjZLQU9wM0duYzNhRW1pQjRaRElsbHdBNS9XcGVPNm5remNKV2tqdGErOUFaTQpDS2hzeS9QRnRIOG52WDJhODdFQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZCOFAvOGh3ZldDR2xjZk5TU3FLQkFtSHBkRTNNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBR2xnelZCUWlyUm9DNFZXNHpRVQpEQldBMFRzNVRzaEdUdkpKekhCY1RSRm43aVhnakNhblBBWnNwWkw0NEpGZlQ1VUZOOFZFUUpCaHNDOE90UDBKCmhDdUlwVnk5MERUMkt2Tjc3Q0dHYVB4cUducXl2bjV0TFJ2ZU9RZWowa1NrTGZlYnVwZExLeUMyeGlOZnduUW8KUzlPMmNlU3E1cE9DQndWRzVGM3FFeTRoVkhUVHJuWjA0Q3dQOHNDdVJtQzE3dkRPRkpDTStlWG1iclFWYTVVMgpqSk1QOERhc2RYazFReCtSdzRDeTh5bUo0eXB4SjBHZmtFNVJnWVk3aVZ3dTFwdkFXWk4zdDJ0TDl5eDN2aU9sCkkrdm1PYTBJdjhNWTFsT3hHNm5nRDZBamNJWU1uSDVPTng1ZDBoa3p6YjRwbVl4eVUvWC9wRnM3WnhLaUhvSisKMUNnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://master01:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:master01
  name: system:node:master01@kubernetes
current-context: system:node:master01@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:master01
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZ0F3SUJBZ0lJU25jKzA1bUJOMUF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TXpBek1EWXhOakUzTWpoYUZ3MHlOVEEwTWpjd01qSTVNVGRhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTY3aVJ4d3VXQUp2MEFZdjAKOXBHM1BWbXc5V1Q5NGtaV09RRVJ6T0VscGR2cENwc0JlMXFneVJrZnpCL2s2UlcrUWE2S3k2aWVpZGpjL2xyRApoQ0dRSGlkbWFmWG9wTXlldWZIeTAwaW84OGJRNHE1NzEzOTh4MllvTmlqMmhMS1p1QkpnQjB2Z3ZEc1JOcjVYClFhWExLeGk3NXpaYlU4SFZpOUpRWFl2SHZoVDJkcktOazNqZlRIUVJkV0NWNjFhR3NGMUsxeFI1bU9pUkgweFEKZDE3OGtOeThQclZiaXMwVFdONUo2ZDlaMURjNERyRStWVXBDcDlnSzJvOW1UeDMwbDlZRm9YSTFOV2RNUTR5QgpYbFFqa2JacjVYSkdIM01wdFE1c3FtV1FGOTNNajlTekpKL3NFN0tmR0dJejM4T3h0UVRBczBRcTBwWWp0RkxhClM0YWVOUUlEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JRZkQvL0ljSDFnaHBYSHpVa3FpZ1FKaDZYUgpOekFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBZnAycVJJN0FLMXd0YWQwZEg5TW5hNnNSMnljUjVlNGQ5dGFvCmpPSGk0OU5SN2ZnNWx2S0NrdTE2UUFNNjRhSFYwRzdKaEd3andQVlVpQWF3T29lQmJrSlBSTldCd25zZGg1SVAKeHQyZ21NRlpCajhua1c1N1A5RUxZWnJBRnIxZFcyaVphb3hFU2xsUG05WGViUVVjd0VWWlVDVUplRDZrT1YxTgorQmlmd3FLN1lUdGhCdHBwemRKd2pOTWpxOFRWWXQ3NkNscC9IV3NKc0N0aElnRXRiQmc0U3k3RmlUb2pmQmV0Ck8xdmptcUhpcGVkQkFDUjJTMXN3TG9oZGtKQWFxVkNrY0ZTbzFvQjVxMi9Ga1U5MEtOS0QwMlRpR25pV0tocGQKRzhXUFF0Vks2UmRucU1aa2xqYnh6RG9TZUt0d0k0T2dJMkNablRiOCt6eGtvc0llL0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBNjdpUnh3dVdBSnYwQVl2MDlwRzNQVm13OVdUOTRrWldPUUVSek9FbHBkdnBDcHNCCmUxcWd5UmtmekIvazZSVytRYTZLeTZpZWlkamMvbHJEaENHUUhpZG1hZlhvcE15ZXVmSHkwMGlvODhiUTRxNTcKMTM5OHgyWW9OaWoyaExLWnVCSmdCMHZndkRzUk5yNVhRYVhMS3hpNzV6WmJVOEhWaTlKUVhZdkh2aFQyZHJLTgprM2pmVEhRUmRXQ1Y2MWFHc0YxSzF4UjVtT2lSSDB4UWQxNzhrTnk4UHJWYmlzMFRXTjVKNmQ5WjFEYzREckUrClZVcENwOWdLMm85bVR4MzBsOVlGb1hJMU5XZE1RNHlCWGxRamtiWnI1WEpHSDNNcHRRNXNxbVdRRjkzTWo5U3oKSkovc0U3S2ZHR0l6MzhPeHRRVEFzMFFxMHBZanRGTGFTNGFlTlFJREFRQUJBb0lCQUc0Sy9Tc2lFb1g0U0VKTQpsekJndUYyUXVKYm03Y3Nyc09idHcrU1VteUhCOXhvM1lNcTRkV1ZNTUZiMzhNS0xud1ZFdVpENENBTXNWWWI4CjBsZWwzNFRrT2VCdnA0ci81MzNCSU81WDlsL1B5Z1o1RkdGM0o4Wml2NVVCTEl5b1lERFppekQ4MEU2dmVJckkKOHkwM2ZCQ0RmSDBsR0IrUzF1RnVib3d0VERORTZxWXR4K3JZOGpaT240M1RDTFN0ME1UUjZzeGkvNmQ1djhQZQpGd0h0ZzZickhLWXpHM25JM2dhL2R1Rk1SL091eFJTb2NRMWlzTGNnOHhNOG5BYWpyVnEvYVduT0FJSDZ2aEY5ClRsL3VZQzJJblhSTkRnOWdVekk2eElRQ3Zoako4bC9TTXlDblJQcURXTlhlMi9LbmdNb2lVL2htVThQbS9QbVYKeCtxTVNLVUNnWUVBK0hRMDZUcU9VSTdXbmdJYWFubHhSRUpCV0VGdG1rSnNUaXAvbFZNTUpqbFlERE8wTEI1dgpEYUJZRGJGaUtaY1E0azJORURDKzh2OVVNSHNnUCtjWXkrQmxKeEhPcUN5WDRRMlZYaWQrejBBNXRoWDVCdmN1Ck54b3drY0xxTS9PcTdBeVV4aTFETXNZSnZkTm10Z0cvOE1USjhNMGZSc2JNamQ2NE8yMytZRmNDZ1lFQTh1RmMKV08xSGh1eG1BR0wrbjhtUWlUaWdwQVZNcS9WSldqWmdTRTdNNTNVNWQvSUoxTnNPZFdEYTd0ZFRYYjJ4RjkyZwp2VTloQTJIN0dFZ084STltUjM4aFlMOVNNMXlCelhWcGZ1VENnSGxHbVRQOG1FWlFPTngzbVhiOUdHTGpCSEcwCjlldXJialU3aWwweEdEbHc5Z1FHcGlQQ1RoRTBCUGNUVjBlWWJsTUNnWUJSam92ajE0WHA2Y21IbXJjakpuUkkKN05yWVpybmN2bHNoR1pCQjZ6Mzh1aXd0N1RjQ21JYnAyUzJ1YVFFSkYvN0IycVVvb0dlK0NwU1BTWXVmam5PMApVUmZEU0hRbS9tUm95dld5TDVMUDQ0eE9PazVCdkJnZW5Hay9IOVdUY2pRMFFqa0FDTkdiTkNqMGVCV3FQMDdMCmNUMjgzd2lhU2gzUm9EK251b2FiQndLQmdRREhLL1dkMng4UHp1WXFwUU9uSnhVRzRsQWhrVkw1cE5FOVVHQU8KaThoT2VudmhGN1BtUUtrUHFwaCt5a2RNRlJXWHU2Qm4vaU5STWRWR2FUa1BIcGxDUWtldDNkYmIxWjJBVGl0dgo2c3B4cm9JdGVrdEcyRXBMQnNoeEtmbU5PUEp2bjR0Rks3ak9USXNDQVVBVVhHNTg0SHBQWGYybUpVZGlWdmNlCldORHdqd0tCZ0ZlUGZ2dUxQbkZLU2diQ1o5bmFZYmhibHFybDlNSG5YQlVmdnBqMGM0bVpjYjEyZWtFbzNKZS8KSTNIQldDTmx1UW9oUkZxcUh2ZWMwL3p0ZmVUTE50b3lEdVNjajdsYk5EM1kxYS9TS3J4eUI1Wk9Fb1A3Rnc4QwpJZjg2Y3dsVEw2MXRvNGZZbC9uRGdveDc2TWtwUkZPSmpiWGpvRUU2cTdOUkc3V2pBMjZJCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

重启kubelet

systemctl start kubelet

评论区