Ingress Firewall when using Kubespan, doesn't allow pod traffic to host node via public ip

[benosman]

Hello,

I have a cluster that comprises of 3 VMs at a non-cloud based provider. They don't have a private network, the only interface on each node is a public one. They also don't have a firewall.

Therefore, I am using Kubespan to ensure all network traffic is encrypted between nodes and pods, and using Ingress Firewall to make sure I limit access.

I turned the default rule to block, and added NetworkRuleConfig entries for talos api and kube api and of course Kubespan. The cluster works and I am able to install cilium as a CNI and that also works over Kubespan, allowing pod to pod communication.

The issue i've found when trying to access node metrics from prometheus, requests from inside pod to the host node of that pod timeout when using the public ip for the node.

Example

Pod 1 on Node 1 -> can access Node 2 and Node 3, times out to Node 1
Pod 2 on Node 2 -> can access Node 1 and Node 3 times out to Node 2

If i understand how the Ingress firewall works, connections from kubespan are whitelisted, but when connecting to the current node, that does not need to traverse kubespan, so it is blocked as there is no firewall rule allowing it.

I can confirm it works when I add a rule allowing all requests from the CNI pod subnet of the current node.

I didn't see anything in the documentation, so i wondered if it was an oversight, bug, or an issue when using cilium. Unfortunately I can't test flannel right now to see if it has the same issue

[TimJones]

Just to confirm, do you whitelist the entire cluster pod cidr in the NetworkRuleConfig, or only the pod cidr allocated to the node?

[benosman]

the pod cidr allocated to the node

[smira]

This is right now how the things work, so you might want to add an explicit rule to allow traffic from pod CIDRs to the host network pod, as Talos Linux firewall won't whitelist it by default.