I’m new to VyOS, and not much used to speak about routing techniques and stuff. But I started testing and using VyOS, with the main purpose of building a managed Kubernetes service using VyOS as router, firewall, NAT and load-balancer, over VMware ESX.
I chose to try VyOS because I needed an easily configurable, terraformable routing OS. Of course, in the VMware environment we can have NSX-T but I can’t use it yet for two reasons : it’s not available yet for us, and the provider will charge us 30% more if we enable it…
This is why I started looking, and found VyOS, which I really think is THE tool we can use to achieve what we want.
And actually, it already is… here’s what I managed to do so far :
Build a terraformable, cloud-init enabled VyOS and deploy it using Terraform with a minimal configuration
Install kube-vip on a Kubernetes cluster behind VyOS, and use BGP and a private virtual IP to automatically load balance my services.
(In progress) Create an automated tool who listens to the Kubernetes state, automatically opens (or closes) a port on the VyOS firewall, and uses NAT to redirect traffic to this virtual IP.
Here’s the actual configuration :
VyOS with eth0 on WAN (with one or multiple possible public IPs) and eth1 on LAN
K8S cluster behind this eth1
virtual, private IP on the same IP range as the local machines, advertised with BGP thanks to kube-vip
But I’m wondering if I could do more…
I’m starting to wonder if I could advertise public IPs with BGP from inside the private network, and have VyOS take them into account and route traffic through eth1 using these public IPs…
Of course, eth0 would also have to have this IP set, in order to receive traffic from the WAN.
For instance :
for now :
VyOS has, say, public IP 8.8.8.8 set on eth0
My cluster is behind eth1, on the 192.168.1.0 network
BGP can advertise to VyOS on this private range, from nodes based on this private network
This situation is perfectly acceptable, actually, I’m just wondering what could be the next steps
What I would like to achieve :
VyOS has public IP 8.8.8.8 set on eth0 but can also be configured to use 8.8.8.9 as secondary IP
My cluster, still behind eth1, on the 192.168.1.0 network
BGP can advertise to VyOS, but it advertises 8.8.8.9 instead of an IP from the private range
VyOS takes it into account and redirects traffic it gets from eth0 to the routes advertised whenever it receives a packet directed to 8.8.8.9
I haven’t started yet, for two reasons :
I don’t clearly know what I’m talking about, I’m not a network guy and behind eth0 is a real provider network
Even if I was right about the concepts, I don’t know if it’s a good idea or not.
Thanks for sharing this interesting project, I have never seen this kube-vip before, however it looks interesting, It seems to me that some customers have used other projects to add network functionality to Kubernets ( Calico for example.)
But anyway , technically it is possible to do it with BGP , but I not sure that it a good practice , what I mean , I don’t know if the public prefixes ( 8.8.8.8.8/ 8.8.8.8.9) are announced to the internet or just live in a private environment (K8S-cluster) . In the end , you could change some attributes and announce those prefixes with more preference for K8S or eth0 .
although , I think it can be done as well with NAT , manipulate some rules .
Thanks @fernando for your kind answer
I’ll give some details and also explain a few things.
When setting up a Kubernetes cluster you have to deal with multiple network concepts :
pod networking (container to container, no matter which machine they run on)
service networking (a fixed IP that are used to load balance traffic to the pods). This concept exists because pods (containers) are not supposed to live long, and other pods that need to access them should not have to look for the pods’ IPs.
node networking (machine to machine, which is for my example 192.168.1.0
(load balanced or not) service exposure to the outside world
Container networking is a separate network, isolated from the outside world, and that’s where Calico is used. Calico can also be used to manage the node networking, and directly expose containers to the main network with BGP, but that’s not what I’m aiming for.
kube-vip is helping us managing the last one, giving us the ability to expose services to the outside world (public or private network) through a virtual IP, and advertise it with either ARP or BGP.
If it’s still unclear ask me anything I’ll try my best to answer
Now to get back to what you asked, yes, the public prefixes are announced to the Internet.
This is what the network looks like from VyOS point of view.
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:50:56:bb:9e:f2 brd ff:ff:ff:ff:ff:ff
inet 213.32.15.194/28 brd 213.32.15.207 scope global eth0
valid_lft forever preferred_lft forever
inet 213.32.15.195/28 brd 213.32.15.207 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:febb:9ef2/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:50:56:bb:41:41 brd ff:ff:ff:ff:ff:ff
inet 192.168.199.252/24 brd 192.168.199.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:febb:4141/64 scope link
valid_lft forever preferred_lft forever
Anything exposed by kube-vip via BGP to the VyOS router comes from the private network.
I got the idea ,but there are a few limitations when it comes to the Internet. On Internet, it’s important to announce a /24 prefix. Using a /32 prefix is not possible announce thought internet (they must be 24/). In order to address this, it might be necessary to establish an IBGP session from VyOS to Kube-vip. Additionally, you can manipulate route-selection attributes to prioritize the K8S prefix over the VyOS prefix.
On the other hand, if you require a prefix smaller than /24, NAT could be considered as a potential solution for this scenario.
:
