VyOS inside KVM/QEMU for home network


#1

NFV in a Personal LAB

Replacing a Cisco Router with PC, KVM and VyOS

First, make sure the lab PC can support virtualization
Verified BIOS set to allow virtualization

Configure PC boot order and test hardware
Set boot order to USB first, CD second, hard disk third
Connected Gigabit Ethernet interface
Booted off Ubuntu 16.10 CD to make sure hardware is good
Powered down

Install new hardware and test again
I need 2 ports, and will eventually need more, as the next project is a switch. So:
Installed 4 port Gigabit Ethernet card in PC
Booted off Ubuntu 16.10 CD again to verify hardware. Did an ifconfig-a and saw original Gigabit Ethernet interface and four additional interfaces

Create Ubuntu Server boot drive and install
Downloaded Ubuntu 16.10 Yakkety server image
Burned ISO to USB thumb drive
Installed Ubuntu 16.10 Server ISO
Selected Standard System Utilities, Virtual Machine Host and OpenSSH Server

Set up connectivity
Since this is a headless server, I installed Virtual Machine Manager on my personal laptop from Ubuntu software store

Server’s IP is 10.0.0.5 via DHCP

Tried connecting via virt-manager.Failed on SSH password (ask-pass). Ran following on my personal laptop per https://help.ubuntu.com/community/KVM/VirtManager:
$ sudo apt-get install virt-manager ssh-askpass-gnome ssh-askpass
–no-install-recommends

Opened up virt-manager on my laptop again, and successfully connected to my server using virt-manager. Used SSH connection and my userid (not root).

First part of my goal is to create a Network Function Virtualization host in my KVM/QEMU server. I currently use a Cisco router for my WAN connection. The goal is to move away from the Cisco and use VyOS running inside KVM.

At this point, we just want to make sure that the KVM Server sees my interfaces. Doing an “ifconfig -a” shows my physical interfaces, virtual interfaces and bridge.

My Interfaces:
This is my primary interface, connected to my LAN. This is how I get in to the KVM Server.
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.5 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 2601:c3:c202:6960:1a03:73ff:fed4:3b9b prefixlen 64 scopeid 0x0
inet6 fe80::1a03:73ff:fed4:3b9b prefixlen 64 scopeid 0x20
ether 18:03:73:d4:3b:9b txqueuelen 1000 (Ethernet)
RX packets 88478 bytes 9347930 (9.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 68742 bytes 21219531 (21.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 20 memory 0xe1c00000-e1c20000

I want to use this for my WAN interface on my VyOS router
enp5s0f0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:ad txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19 memory 0xe1ba0000-e1bc0000

I want to use this for my LAN interface on my VyOS router
enp5s0f1: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:ac txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18 memory 0xe1b40000-e1b60000

When I install ovswitch in a different virtual machine, this will be a physical switch interface
enp6s0f0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:af txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17 memory 0xe1aa0000-e1ac0000

This will also be a switch interface, controlled by ovswitch
enp6s0f1: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:ae txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xe1a40000-e1a60000

This is my loopback
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1 (Local Loopback)
RX packets 120010 bytes 18056633 (18.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 120010 bytes 18056633 (18.0 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

This allows virtual hosts inside my KVM to communicate with the network. This is it’s gateway out
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:58:66:e9 txqueuelen 1000 (Ethernet)
RX packets 21 bytes 2160 (2.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15 bytes 990 (990.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0-nic: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 52:54:00:58:66:e9 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fc54:ff:fecb:57e5 prefixlen 64 scopeid 0x20
ether fe:54:00:cb:57:e5 txqueuelen 1000 (Ethernet)
RX packets 7 bytes 818 (818.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6656 bytes 346566 (346.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Download VyOS
Verify the SHA1 sums: sha1sum vyos-1.1.7-amd64.iso
Compare to http://packages.vyos.net/iso/release/1.1.7/sha1sums

When installing, leave Gigabit Ethernet0 as main link to server. Use 1st of the 4 new interfaces for the VyOS WAN Interface.

Create a new image:
Tried copying the iso to the server:
$ sftp bruce@10.0.0.5
bruce@10.0.0.5’s password:
Connected to 10.0.0.5.
sftp> cd /var/lib/libvirt/images
sftp> ls
remote readdir("/var/lib/libvirt/images"): Permission denied
sftp> put vyos-1.1.7-amd64.iso
Uploading vyos-1.1.7-amd64.iso to /var/lib/libvirt/images/vyos-1.1.7-amd64.iso
remote open("/var/lib/libvirt/images/vyos-1.1.7-amd64.iso"): Permission denied
sftp> quit

This tells me I can’t write to libvirt. Doing an ls -all shows me /var/lib/libvirt/images is owned by root. Entered $sudo chown bruce images
Then was able to put the ISO: (note – there may be better ways of doing this, other than changing the owner of the directory)

sftp> put vyos-1.1.7-amd64.iso

I opened a virt-manager session with my virtual server and got error messages stating: “could not grab keyboard”
Followed instructions at this site to set up my ssh keys between my laptop and my KVM server:

Then I powered it up!

And failure. We are not set to boot from CD! Went in to Information on the VyOS image and clicked Boot Options and set it to boot from CD. Then went in to IDE CDROM1 and set the location of the ISO.

Just like a physical server needs to have a boot image, and can boot from CD or USB on an install, we have the same requirement on the virtual server. So, I set the virtual server to boot from CD and pointed the boot source to my ISO. Success! We have VyOS bootup screen!

Followed VyOS installation instructions found here:
https://wiki.vyos.net/wiki/Installation

You need to install the image, otherwise VyOS will run off the live CD and will not save or boot up with configuration changes.

Next up: configure bridges and Ethernet interfaces on the KVM Server. Followed guide from here, substituting for my needs:

First we need to shut the virtual machine down and add interfaces through virt-manager. I added the NIC with Bridge brwan. Then hit right click, add hardware and added a second NIC using brlan. (Note: I selected virtio, and it worked. I also tested E1000).

Setting up the network interfaces:
On the KVM Virtual Server, edit interfaces file, adding the following lines (remember, we got the interface names by doing an ifconfig -a):
$ sudo vi /etc/network/interfaces

The LAN port

auto enp5s0f1
iface enp5s0f1 inet manual

The WAN port

auto enp5s0f0
iface enp5s0f0 inet manual

The LAN-side switch

auto brlan
iface brlan inet manual
bridge_ports enp5s0f1
bridge_stp off
bridge_fd 0
bridge_maxwait 0

The WAN-side switch

auto brwan
iface brwan inet manual
bridge_ports enp5s0f0
bridge_stp off
bridge_fd 0
bridge_maxwait 0

I had 2 small desktop switches. I connected my WAN interface to one switch and my LAN interface to the other. (using a higher end switch that supports vlans would also work).

Configure the VyOS Router
What we want to create is this:

LAN—Bridge—VyOS Router----Bridge----WAN

Followed the VyOS Quick Start Guide:
Enter configuration mode:
vyos@vyos$ configure
vyos@vyos#
Configure network interfaces:
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description ‘OUTSIDE’

set interfaces ethernet eth1 address ‘10.0.5.1/24’
set interfaces ethernet eth1 description ‘INSIDE’
Enable SSH for remote management:
set service ssh port ‘22’
Did a commit then a save then a exit.
After doing an exit, did a “show interfaces”

I grabbed an IP via DHCP from my “real” router, meaning my VyOS WAN Connection is talking to my network! I tried pinging, and from VyOS console I can ping my VyOS eth0 interface, my KVM host IP, my KVM gateway IP. I can also ping outside my KVM server and can ping my hardware router and hardware switch. A traceroute shows I have connectivity! I can also ping from network devices outside of KVM in to my VyOS router. Looks like the WAN side is good!

Do a “commit”, “save” and “exit”.

Finished configuring router using the VyOS guideline:
Open up a terminal session and ssh to VyOS: ssh vyos@10.0.1.111
Configure Source NAT for our “Inside” network. Once again, straight from the VyOS guide.

set nat source rule 100 outbound-interface ‘eth0’
set nat source rule 100 source address ‘10.0.5.0/24’
set nat source rule 100 translation address masquerade

Configure a DHCP Server:

set service dhcp-server disabled ‘false’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 default-router ‘10.0.5.1’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 dns-server ‘10.0.5.1’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 domain-name ‘internal-network’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 lease ‘86400’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 start 10.0.5.9 stop ‘10.0.5.254’

And a DNS forwarder:

set service dns forwarding cache-size ‘0’
set service dns forwarding listen-on ‘eth1’
set service dns forwarding name-server ‘8.8.8.8’
set service dns forwarding name-server ‘8.8.4.4’

Add a set of firewall policies for our “Outside” interface:

set firewall name OUTSIDE-IN default-action ‘drop’
set firewall name OUTSIDE-IN rule 10 action ‘accept’
set firewall name OUTSIDE-IN rule 10 state established ‘enable’
set firewall name OUTSIDE-IN rule 10 state related ‘enable’

set firewall name OUTSIDE-LOCAL default-action ‘drop’
set firewall name OUTSIDE-LOCAL rule 10 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 10 state established ‘enable’
set firewall name OUTSIDE-LOCAL rule 10 state related ‘enable’
set firewall name OUTSIDE-LOCAL rule 20 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 20 icmp type-name ‘echo-request’
set firewall name OUTSIDE-LOCAL rule 20 protocol ‘icmp’
set firewall name OUTSIDE-LOCAL rule 20 state new ‘enable’
set firewall name OUTSIDE-LOCAL rule 30 action ‘drop’
set firewall name OUTSIDE-LOCAL rule 30 destination port ‘22’
set firewall name OUTSIDE-LOCAL rule 30 protocol ‘tcp’
set firewall name OUTSIDE-LOCAL rule 30 recent count ‘4’
set firewall name OUTSIDE-LOCAL rule 30 recent time ‘60’
set firewall name OUTSIDE-LOCAL rule 30 state new ‘enable’
set firewall name OUTSIDE-LOCAL rule 31 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 31 destination port ‘22’
set firewall name OUTSIDE-LOCAL rule 31 protocol ‘tcp’
set firewall name OUTSIDE-LOCAL rule 31 state new ‘enable’
set firewall name OUTSIDE-LOCAL rule 32 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 32 destination port ‘161’
set firewall name OUTSIDE-LOCAL rule 32 protocol ‘udp’

Note: originally firewall rules did not allow snmp. Added rule 32 to enable snmp management.

Apply the firewall policies:
set interfaces ethernet eth0 firewall in name ‘OUTSIDE-IN’
set interfaces ethernet eth0 firewall local name ‘OUTSIDE-LOCAL’

Commit changes, save the configuration, and exit configuration mode:

vyos@vyos# commit
vyos@vyos# save
Saving configuration to ‘/config/config.boot’…
Done
vyos@vyos# exit
vyos@vyos$

Communicating through VyOS:
I connected an Ethernet cable between my laptop and a switch that VyOS LAN Interface is connected to. Then I disabled my wireless interface. The laptop can communicate with my home servers (Cacti, nfsen/nfdump and a NAS server). It can also communicate outside my network. For a home user, I am not noticing any major differences between this connection and my wireless connection. My Cisco router did not have a route back to my VyOS LAN interface, so I entered a static route on the Cisco.

How about measurements?
SNMP
VyOS supports snmp. I had to open up the firewall (rule 32 noted above) to allow udp port 161, then configure snmp using the following commands:
set service snmp community (snip)
set service snmp community (snip) client 10.0.1.51
set service snmp community (snip) authorization ro
set service snmp contact “Bubba”
set service snmp location “Bubbatown”

I then started polling VyOS from Cacti (an snmp platform), getting CPU, Memory, Load and Ethernet Interface statistics. I also used the Cacti Advanced Ping tool to give me basic latency between my Cacti server running on CentOS7 in a VMWare esxi 6.0 hypervisor and VyOS router.

ToDos: I am using a generic Linux SNMP template in Cacti. I found no VyOS or Vyatta templates at Cacti site. This is okay for a home router, but I would expect to see snmp collectors for errors, discards and QoS (if configured). More information is available when walking the SNMP MIB, so it could be made available by creating a template in Cacti (or any other preferred snmp management system). I haven’t decided if I want to do this.

Netflow
I am running an OpenSource Netflow server called nfsen/nfdump. So, configure netflow on VyOS:
(I copied this directly based on Google search, just substituted in my Netflow server IP and port).
vyos@vyos# set system flow-accounting interface eth0

vyos@vyos# set system flow-accounting interface eth1
[vyos@vyos# set system flow-accounting netflow server 10.0.1.53 port 9996
vyos@vyos# set system flow-accounting netflow version
Possible completions:
1 NetFlow version 1
5 NetFlow version 5 (default)
9 NetFlow version 9

vyos@vyos# set system flow-accounting netflow version 5

On the Netflow server, configure iptables to allow port 9996 in:
Edited IP tables on nfsen server to allow port 9996:
sudo iptables -I INPUT -p udp -m state --state NEW -m udp --dport 9996 -j ACCEPT

On the Netflow server, add the new source:
Do an “nfsen stop”
Edit nfsen.conf on my CentOS7 nfsen/nfdump server. BASEDIR is /data
vi /data/nfsen/etc/nfsen.conf
Do an “nfsen reconfig”
Then “nfsen start”

nfsen/nfdump is showing zero packets, although TCPDUMP shows I am getting traffic.

Modified my VyOS Netflow configuration:
flow-accounting {
interface eth1
interface eth0
netflow {
engine-id 100
server 10.0.1.53 {
port 9996
}
timeout {
expiry-interval 60
flow-generic 3600
icmp 300
max-active-life 604800
tcp-fin 300
tcp-generic 3600
tcp-rst 120
udp 300
}
version 5
}
}

Still no luck with Netflow. I can see the packets arriving at my Netflow server by doing a TCPDUMP. VyOS and nfsen/nfdump are configured for Netflow Version 5. Setting this one aside for now.Verified flow-accounting on VyOS. VyOS looks good!

Update: I updated my IPTABLES rules, but did not save them before rebooting my nfsen/nfdump server. I was receiving packets on port 9996, but not accepting them. So – update IPTABLES and save it!
sudo iptables -I INPUT -p udp -m state --state NEW -m udp --dport 9996 -j ACCEPT
service iptables save

For now I have my personal laptop and my work laptop connected to a low-end switch connected to my PC running VyOS in KVM. I can view my utilization using snmp. I can view communicating pairs using Netflow. My next goal is to learn a little more about Software Defined Networks and see if I can figure out putting up another virtual server, run ovswitch on the new server and connect my un-used Gigabit interface cards to ovswitch. I’m considering using OpenDaylight for the SDN Controller. (if anyone has experience with ovswitch and how I would add my two un-used interfaces and the VyOS LAN interface to ovswitch, let me know. I’m trying to figure out the hardware and bridge mapping. I think I’d also need to pick up a few cross-over cables to connect my laptop directly to the PC running KVM, VyOS and hopefully ovswitch.


#2

I will strongly advise to grab wiki.vyos.net account,
and make it available there too :slight_smile:
Nice writeup!


#3

Thanks! Appreciate it. I will clean up the documentation and post it on the wiki. I’m trying to sort through phases 2 and 3 of the project.

I installed a second virtual server in KVM and installed ovswitch. I set up a bridge (br0) in ovswitch, moved my IP config to br0 and popped ens3 (my LAN interface) into the bridge group. So now VyOS is talking through ovswitch. I installed Open Daylight SDN controller in a 3rd VM under KVM. From ODL I can see ovswitch, I can see my layer 2 flow. I can see my VyOS router mac and IP, my ovswitch mac and links and my laptop connected via NIC through the LAN interface. I can also see a laptop connected directly to ovswitch using a dedicated NIC on my KVM server and a crossover cable. Layer 3 isn’t working for this host yet, but I can see it at layer 2.

So:
the goal is to have a PC with multiple ethernet NICS (at least 5, but 9 would be better).
Use VyOS for layer 3 routing
Use ovswitch for layer 2 connectivity
Use Open Daylight to configure ovswitch (for now, I’m just using ovswitch’s default layer 2 forwarding rule: ovs-ofctl add-flow br0 action=NORMAL

As soon as I can get my laptop connected directly to the KVM Server talking over layer 3, I’ll do a write up. (I’m thinking it may mean vlans, but hoping it might be something much simpler)