Kubernetes and etcd certs renewal

Let's walk through how to renew Kubernetes component certificates.

Step 1: Backing Up Important Files

Before you do anything, taking backups is always a good idea, especially when dealing with something as crucial as certificates.

# Backup your Kubernetes configuration
cp -r /etc/kubernetes/ /tmp/k8s-certs-2023/
cp -r /root/.kube/config /root/.kube/config_2023

Step 2: Check Certificate Expiration

Now, let’s check which certificates have expired:

kubeadm certs check-expiration

If you find any expired certificates, proceed with renewing them.

Step 3: Renew the Kubernetes Certificates

You can renew all of them in one go with:

kubeadm certs renew all

Once renewed, copy the admin.conf back to the .kube directory:

# Update kube config with the renewed certificates
cp /etc/kubernetes/admin.conf /root/.kube/config

Don’t forget about the Kubelet certificate:

# Regenerate kubelet certificate
rm /etc/kubernetes/kubelet.conf
kubeadm init phase kubeconfig kubelet

Restart the Kubelet service to apply the new certificates:

# Restart kubelet to apply changes
systemctl restart kubelet
systemctl status kubelet
journalctl -u kubelet -r -l | more  # Check logs if needed

You may also need to delete the existing pods in the kube-system namespace so they pick up the new configurations:

kubectl delete pod -n kube-system --all

Step 4: ETCD Certificate Renewal

What about ETCD? If you're running ETCD separately for production workloads, you'll also need to renew its certificates.

First, check the expiration of the ETCD client certificate:

 # Check ETCD certificate expiry
 openssl x509 -enddate -noout -in /etc/kubernetes/pki/apiserver-etcd-client.crt

Next, copy the apiserver-etcd-client.key and apiserver-etcd-client.crt from one of the ETCD nodes to the Master’s /etc/kubernetes directory. If you’re unsure, check the mount path in /etc/kubernetes/manifests/kube-apiserver.yaml.

Renew the certificates for the API server and related components:

 kubeadm certs renew admin.conf
 kubeadm certs renew apiserver
 kubeadm certs renew apiserver-kubelet-client
 kubeadm certs renew controller-manager.conf
 kubeadm certs renew front-proxy-client
 kubeadm certs renew scheduler.conf

Finally, restart the Kubelet service once more:
```
 systemctl restart kubelet
```

Step 5: Check the Health of the ETCD Cluster

Once the certificates have been renewed, verify the health of your ETCD cluster using etcdctl:

# Check ETCD health
ETCDCTL_API=3 etcdctl --endpoints=https://IP:PORT \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  endpoint health --cluster -w table

If you're using containerized ETCD, here's the workaround to check health:

# For containerized ETCD, run this command to check health
ctr run --rm -t --net-host --mount type=bind,src=/etc/kubernetes/pki/etcd,dst=/etc/kubernetes/pki/etcd,options=rbind:rw registry.k8s.io/etcd:3.5.6-0 test1 etcdctl \
  --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --endpoints https://IP:Port \
  endpoint health --cluster -w table

And that's how you renew your ETCD and Kubernetes certificates! Hopefully, this guide saves your next chill Friday. Happy clustering!

See you in the next blog post :)

Step-by-Step Guide to Kubernetes Certificate Renewal

Hey there!

Table of contents

No headings in the article.