Kubernetes server and agent support

th3wyatt · October 22, 2020, 4:41pm

I currently have a requirement to run a k3s worker on vyos and i’m having some networking issues. The pods are not able to talk to the server API. I’ve been working with the k3s devs to try to troubleshoot the issue to see why they can’t communicate, but have hit a dead-end. What would be blocking the pods from being able to communicate over the flannel network?

github.com/k3s-io/k3s

Pods unable to connect to server API

opened 07:38PM - 19 Oct 20 UTC

closed 01:23AM - 14 Aug 21 UTC

th3wyatt

status/stale

**Environmental Info:** K3s Version: k3s version v1.18.9+k3s1 (630bebf9) … Node(s) CPU architecture, OS, and Version: Linux vsdr-kube 5.4.68-amd64-vyos #1 SMP Tue Oct 13 19:30:42 UTC 2020 x86_64 GNU/Linux Cluster Configuration: single node cluster **Describe the bug:** The pods are unable to connect to the API on the server. **Steps To Reproduce:** - Installed K3s: problem starts right after initial install on a fresh system **Additional context / logs:** ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-7944c66d8d-6p472 0/1 Running 0 17m kube-system metrics-server-7566d596c8-wtkhp 0/1 CrashLoopBackOff 7 17m kube-system local-path-provisioner-6d59f47c7-qwsxh 0/1 CrashLoopBackOff 7 17m kube-system helm-install-traefik-r6sg7 0/1 CrashLoopBackOff 6 17m ``` [describeNode.txt](https://github.com/rancher/k3s/files/5404250/describeNode.txt) [describePods.txt](https://github.com/rancher/k3s/files/5404252/describePods.txt) [iptables.txt](https://github.com/rancher/k3s/files/5404257/iptables.txt) [corednsLogs.txt](https://github.com/rancher/k3s/files/5404267/corednsLogs.txt)

rob · October 24, 2020, 3:57pm

Hi,

you try unsupported stuff

can you post all steps to reproduce this? maybe i can help you.

th3wyatt · October 26, 2020, 3:05pm

Install VyOS to disk
create directory /var/lib/rancher
Create ext4 filesystem and mount to /var/lib/rancher
convert to iptables-legacy
install k3s as described on their gitub using the curl command
sudo kubectl get pods -A (to see the pods)
sudo kubectl logs -n kube-system (name of a pod) here you will see the communication errors

The linked github issue shows all the commands and outputs on the system I was using.

Thanks for taking a look!

rob · October 26, 2020, 7:32pm

the nat rules for the kube nics and networkes don’t work.
“iptables -t nat -L”
I don’t know what exactly is happening. Either the chains don’t fit or switching to the iptables-legacy destroys something.

when you do a tcpdump on the kube nic (cni0 on my host) and on the outgoing interface of vyos you see the plain an unnatted traffic of the pods.

Viacheslav · October 28, 2020, 1:05pm

Can you describe this step in more detail? What do you change?

th3wyatt · October 28, 2020, 3:45pm

I use update-alternatives to switch to iptables-legacy then reboot. I am looking at cni plugins that support nftables since it appears to a natting issue. Converting to iptables-legacy has some pretty rough side effects with vyos.

Viacheslav · October 28, 2020, 3:54pm

Firewall rules in VyOS have own HOOKS, maybe this conflicts with your new iptables rules.
It shows you all the firewall rules.

$ sudo iptables-save

th3wyatt · October 28, 2020, 8:09pm

# Generated by xtables-save v1.8.2 on Wed Oct 28 20:07:31 2020
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:VYATTA_CT_IGNORE - [0:0]
:VYATTA_CT_TIMEOUT - [0:0]
:VYATTA_CT_HELPER - [0:0]
:VYATTA_CT_PREROUTING_HOOK - [0:0]
:VYATTA_CT_OUTPUT_HOOK - [0:0]
:NAT_CONNTRACK - [0:0]
-A PREROUTING -j VYATTA_CT_IGNORE
-A PREROUTING -j VYATTA_CT_HELPER
-A PREROUTING -j VYATTA_CT_TIMEOUT
-A PREROUTING -j VYATTA_CT_PREROUTING_HOOK
-A PREROUTING -j NAT_CONNTRACK
-A PREROUTING -j NOTRACK
-A OUTPUT -j VYATTA_CT_IGNORE
-A OUTPUT -j VYATTA_CT_HELPER
-A OUTPUT -j VYATTA_CT_TIMEOUT
-A OUTPUT -j VYATTA_CT_OUTPUT_HOOK
-A OUTPUT -j NAT_CONNTRACK
-A OUTPUT -j NOTRACK
-A VYATTA_CT_IGNORE -j RETURN
-A VYATTA_CT_TIMEOUT -j RETURN
-A VYATTA_CT_HELPER -p tcp -m tcp --dport 1536 -j CT --helper tns
-A VYATTA_CT_HELPER -p tcp -m tcp --dport 1525 -j CT --helper tns
-A VYATTA_CT_HELPER -p tcp -m tcp --dport 1521 -j CT --helper tns
-A VYATTA_CT_HELPER -p udp -m udp --dport 111 -j CT --helper rpc
-A VYATTA_CT_HELPER -p tcp -m tcp --dport 111 -j CT --helper rpc
-A VYATTA_CT_HELPER -j RETURN
-A VYATTA_CT_PREROUTING_HOOK -j RETURN
-A VYATTA_CT_OUTPUT_HOOK -j RETURN
-A NAT_CONNTRACK -j ACCEPT
COMMIT
# Completed on Wed Oct 28 20:07:31 2020
# Generated by xtables-save v1.8.2 on Wed Oct 28 20:07:31 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:VYATTA_PRE_FW_IN_HOOK - [0:0]
:VYATTA_PRE_FW_FWD_HOOK - [0:0]
:VYATTA_PRE_FW_OUT_HOOK - [0:0]
:VYATTA_POST_FW_IN_HOOK - [0:0]
:VYATTA_POST_FW_FWD_HOOK - [0:0]
:VYATTA_POST_FW_OUT_HOOK - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-FORWARD - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -j VYATTA_PRE_FW_IN_HOOK
-A INPUT -j VYATTA_POST_FW_IN_HOOK
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -j VYATTA_PRE_FW_FWD_HOOK
-A FORWARD -j VYATTA_POST_FW_FWD_HOOK
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -j VYATTA_PRE_FW_OUT_HOOK
-A OUTPUT -j VYATTA_POST_FW_OUT_HOOK
-A VYATTA_PRE_FW_IN_HOOK -j RETURN
-A VYATTA_PRE_FW_FWD_HOOK -j RETURN
-A VYATTA_PRE_FW_OUT_HOOK -j RETURN
-A VYATTA_POST_FW_IN_HOOK -j ACCEPT
-A VYATTA_POST_FW_FWD_HOOK -j ACCEPT
-A VYATTA_POST_FW_OUT_HOOK -j ACCEPT
-A KUBE-FIREWALL -m mark --mark 0x8000/0x8000 -m comment --comment "kubernetes firewall for dropping marked packets" -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -m comment --comment "block incoming localnet connections" -j DROP
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m tcp --dport 9153 -m comment --comment "kube-system/kube-dns:metrics has no endpoints" -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m udp --dport 53 -m comment --comment "kube-system/kube-dns:dns has no endpoints" -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m tcp --dport 53 -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.170.47/32 -p tcp -m tcp --dport 443 -m comment --comment "kube-system/metrics-server has no endpoints" -j REJECT --reject-with icmp-port-unreachable
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m mark --mark 0x4000/0x4000 -m comment --comment "kubernetes forwarding rules" -j ACCEPT
-A KUBE-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "kubernetes forwarding conntrack pod source rule" -j ACCEPT
-A KUBE-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "kubernetes forwarding conntrack pod destination rule" -j ACCEPT
COMMIT
# Completed on Wed Oct 28 20:07:31 2020
# Generated by xtables-save v1.8.2 on Wed Oct 28 20:07:31 2020
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
COMMIT