Vyos on KVM, best practice to pass all traffic through the vyos VM

General questions
1

Hi guys

I would like to know what is best practice to have Vyos on a KVM host especially if I want all traffic, including traffic to the host to be routed via the Vyos VM.

Thanks
Kitty

2

I would suggest to use Debian stable, newer kernel from testing, with virt-manager and/or cockpit web interface for KVM.
https://cockpit-project.org/guide/latest/feature-virtualmachines
WIPE OUT network manager and initialize bridges manually.

iface br2_DMZ inet manual
   bridge_ports enp18s0
   bridge_stp off
   bridge_waitpot 0
   bridge_fd 0.0
auto br2_INT
iface br2_INT inet manual
   bridge_ports enp19s0
   bridge_stp off
   bridge_waitpot 0
   bridge_fd 0.0

I’m not sure if routing traffic for host OS through VyOS running under KVM is a good idea. If virtual machine is down for whatever reason, the whole system becomes completely unmanageable via ssh or web.

BTW, please note not all combinations of hardware and software work as expected. For example, I have Jetway fanless PC with 10 Ethernet interfaces, and KVM/VyOS 1.1.8 completely freeze it at random fashion. I tried 3 variants of host OS - SuSE 42.3, SuSE Leap 15.0 and Debian 9. Debian crashes less frequently but still crashes. May be its hardware bug or hidden defect. Currently testing this config on ordinary PC with4-port Ethernet card.

1 Like
3

thanks for replying > I was thinking of using Centos 7, any results on using this ?

4

I use CentOS for oVirt/KVM, it works, yet CentOS is not in my taste, prefer SuSE and Debian.

5

My challenge is I want to place kvm on a dedicated box at the DC and the host only offers one connection to their switch, albeit I get a couple of IP addresses. The servers are Supermicros.You have
made me aware of an important issue, which is if my Vyos VM is down I will not have access to the Host.
I guess I will have to use some other method to protect my Host unless there is some common practice.

Thanks
Kitty

6

Plain firewall, non-standard ssh port and fail2ban against brute force login/password guess attacks.

7

Thank you, will start testing this.

8

I was wondering if Vyos allow for the creation of virtual inferfaces. I would have a bridge for the Vyos
and my host using the physical NiC and then have virtual interfaces for which my VMs can connect to for dhcp etc.

Thanks
Kitty

9

I don’t think I completely understood your question, but behind the scenes I think you are already using “virtual interfaces” in the form of tap network devices that you can add to bridges on your host (or directly use as a network interface), and which appear inside the KVM as a “normal” interface.

Can you describe a little better the “architecture” you are trying to achieve?

10

I may have a setup that comes pretty close to what you want (if I understand you correctly):

I’m on Debian testing (had issues with the KVM Q35 chipset on stable but you can do it also with stable and the older i440 chipset).
I have two physical interfaces WAN (br0), LAN (br1) - obviously configured in bridge mode.

VyOS is the central router/firewall all traffic to WAN goes via VyOS.
On LAN (br1) VyOS is also the DHCP and DNS forwading server for virtual (running on the same host) and physical LAN clients. VM to VM throughput is very good.

I can accces the host with VyOS not running (fix ip) and it can bypass VyOS for direct internet access if needed.

11

Can you please describe those issues? I may have same problem.

12

I had a bunch of these in the VM dmesg log:

[ ...] Uhhuh. NMI received for unknown reason x on CPU y.
[ ...] Do you have a strange power saving mode enabled?
[ ...] Dazed and confused, but trying to continue

Which seemed to have caused also some erratic behavior.
Solution is to either fallback to i440 (no issues there) or upgrade either qemu/kvm (not sure anymore which one). As I wanted to switch to a newer kernel and tools anyway I chose to upgrade to testing and as far as I can tell the errors are gone.
But this is my home net, not a super critical piece of infrastructure.

13

I think the better way to explain it is a bridge between the host and Vyos ( to access the core switch ) and an internal network between Vyos and all other VMs, taking into to account I only have one physical NIC. Vyos will control access to the VMs but I will have to secure the host outside of Vyos.

thanks
Kitty

14

That shouldn’t be a problem, assuming all VMs incl. VyOS are on the same host:

  1. Create 1 bridge (with the physical NIC) for the outside communication, expose it as a NIC to the VyOS VM
  2. Create a KVM internal virtual network, expose it as NICs to the VMs and VyOS VM
  3. Set up routing and firewall rules in VyOS
16

Thank you, I am going to try this out today.