Containerd, kubelet is failing on Talos 1.7.2

[Rammurthy5]

I have talos 1.7.2 running and one of the worker nodes this morning went to NotReady state. Dmesg shows the below. Could i ask how to get this up & running again please?

x.x.xx: user: warning: [2024-06-18T09:45:03.062320046Z]: [talos] service[cri](Running): Process Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"]) started with PID 193001 
x.x.xx: user: warning: [2024-06-18T09:45:03.231232046Z]: [talos] service[cri](Waiting): Error running Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"]), going to restart forever: exit status 1
x.x.xx: user: warning: [2024-06-18T09:45:06.482470046Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "stat /etc/kubernetes/kubeconfig-kubelet: input/output error"}
x.x.xx: user: warning: [2024-06-18T09:45:07.740285046Z]: [talos] service[kubelet](Waiting): Error running Containerd(kubelet), going to restart forever: failed to create task: "kubelet": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: connection refused": unavailable
x.x.xx: user: warning: [2024-06-18T09:45:08.234730046Z]: [talos] service[cri](Running): Process Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"]) started with PID 193020
x.x.xx: user: warning: [2024-06-18T09:45:08.383217046Z]: [talos] service[cri](Waiting): Error running Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"]), going to restart forever: exit status 1

containerd logs says nothing suspicious, last few lines from the logs.

"level":"info","msg":"loading plugin \"io.containerd.grpc.v1.transfer\"...","time":"2024-06-07T10:23:24.784630141Z","type":"io.containerd.grpc.v1"}
 {"level":"info","msg":"loading plugin \"io.containerd.grpc.v1.version\"...","time":"2024-06-07T10:23:24.784640897Z","type":"io.containerd.grpc.v1"}
 {"level":"info","msg":"loading plugin \"io.containerd.internal.v1.restart\"...","time":"2024-06-07T10:23:24.784659780Z","type":"io.containerd.internal.v1"}
{"level":"info","msg":"loading plugin \"io.containerd.grpc.v1.healthcheck\"...","time":"2024-06-07T10:23:24.784706794Z","type":"io.containerd.grpc.v1"}
{"address":"/system/run/containerd/containerd.sock.ttrpc","level":"info","msg":"serving...","time":"2024-06-07T10:23:24.784907960Z"}
{"address":"/system/run/containerd/containerd.sock","level":"info","msg":"serving...","time":"2024-06-07T10:23:24.784953567Z"}
{"level":"info","msg":"containerd successfully booted in 0.034933s","time":"2024-06-07T10:23:24.784979123Z"}
 time="2024-06-07T10:29:29.591691354Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-06-07T10:29:29.591808503Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
 time="2024-06-07T10:29:29.591828496Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-06-07T10:29:29.592510124Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1

kubelet logs says some errors

{"ts":1718687682596.5786,"caller":"machine/info.go:104","msg":"Failed to get disk map: open /sys/block/nvme1c1n1/dev: no such file or directory\n"}
{"ts":1718687742670.2568,"caller":"iptables/iptables.go:575","msg":"Could not set up iptables canary","table":"mangle","chain":"KUBE-KUBELET-CANARY","err":"error creating chain \"KUBE-KUBELET-CANARY\": executable file not found in $PATH: "}

[smira]

Most probably your machine doesn't have enough resources (RAM/disk), and it leads to spurious failures like that. Talos reports on these issues early in the boot process.

[smira]

It might be running of resources.

[Rammurthy5]

would you advise to simply disown that worker and replace it with a higher configuration new worker node?

[smira]

I can't give such advice without enough information, but you can inspect the resource usage and figure out whether my guess about resource starvation is correct or not.

[Rammurthy5]

I see the below when i checked talosctl memory for the respective node. seem to be fine on the resources perspective if i am not wrong.

t --talosconfig talosconfig -n x.x.x.x memory
NODE          TOTAL   USED    FREE   SHARED   BUFFERS   CACHE   AVAILABLE
x.x.x.x   15406   10256   144    248      1         5004    5103

t --talosconfig talosconfig -n x.x.x.x dashboard
ip-x.x.x.x (v1.7.2): uptime 352h50m50s, 8x3.4GHz, 15 GiB RAM, PROCS 76, CPU 26.9%, RAM 66.6%                                                                                                            
                                                                                                                                                                                                            
 UUID     ec25efb8-7bdb-defa-3812-c9314fb699b7                               KUBERNETES v1.28.3                                 HOST         ip-x.x.x.x                                                 
 CLUSTER  n/a                                                                KUBELET    Unhealthy                               IP           x.x.x.x/20
 STAGE    Running                                                                                                               GW           n/a                                                            
 READY    False                                                                                                                 CONNECTIVITY OK                                                             
 TYPE     worker                                                                                                                DNS          x.x.x.x                                                       
 MACHINES 11                                                                                                                    NTP          time.cloudflare.com   

t --talosconfig talosconfig -n x.x.x.x stats
NODE          NAMESPACE   ID     MEMORY(MB)   CPU
x.x.x.x   system      apid   25.80        101847568000

[smira]

Make sure you also have enough disk space available.

You can troubleshoot the issue yourself with the tools and APIs provided by Talos.

E.g. if cri is failing and restarting, do talosctl logs cri to figure out why it fails.