NFV in a Personal LAB
Replacing a Cisco Router with PC, KVM and VyOS
First, make sure the lab PC can support virtualization
Verified BIOS set to allow virtualization
Configure PC boot order and test hardware
Set boot order to USB first, CD second, hard disk third
Connected Gigabit Ethernet interface
Booted off Ubuntu 16.10 CD to make sure hardware is good
Powered down
Install new hardware and test again
I need 2 ports, and will eventually need more, as the next project is a switch. So:
Installed 4 port Gigabit Ethernet card in PC
Booted off Ubuntu 16.10 CD again to verify hardware. Did an ifconfig-a and saw original Gigabit Ethernet interface and four additional interfaces
Create Ubuntu Server boot drive and install
Downloaded Ubuntu 16.10 Yakkety server image
Burned ISO to USB thumb drive
Installed Ubuntu 16.10 Server ISO
Selected Standard System Utilities, Virtual Machine Host and OpenSSH Server
Set up connectivity
Since this is a headless server, I installed Virtual Machine Manager on my personal laptop from Ubuntu software store
Server’s IP is 10.0.0.5 via DHCP
Tried connecting via virt-manager.Failed on SSH password (ask-pass). Ran following on my personal laptop per https://help.ubuntu.com/community/KVM/VirtManager:
$ sudo apt-get install virt-manager ssh-askpass-gnome ssh-askpass
–no-install-recommends
Opened up virt-manager on my laptop again, and successfully connected to my server using virt-manager. Used SSH connection and my userid (not root).
First part of my goal is to create a Network Function Virtualization host in my KVM/QEMU server. I currently use a Cisco router for my WAN connection. The goal is to move away from the Cisco and use VyOS running inside KVM.
At this point, we just want to make sure that the KVM Server sees my interfaces. Doing an “ifconfig -a” shows my physical interfaces, virtual interfaces and bridge.
My Interfaces:
This is my primary interface, connected to my LAN. This is how I get in to the KVM Server.
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.5 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 2601:c3:c202:6960:1a03:73ff:fed4:3b9b prefixlen 64 scopeid 0x0
inet6 fe80::1a03:73ff:fed4:3b9b prefixlen 64 scopeid 0x20
ether 18:03:73:d4:3b:9b txqueuelen 1000 (Ethernet)
RX packets 88478 bytes 9347930 (9.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 68742 bytes 21219531 (21.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 20 memory 0xe1c00000-e1c20000
I want to use this for my WAN interface on my VyOS router
enp5s0f0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:ad txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19 memory 0xe1ba0000-e1bc0000
I want to use this for my LAN interface on my VyOS router
enp5s0f1: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:ac txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18 memory 0xe1b40000-e1b60000
When I install ovswitch in a different virtual machine, this will be a physical switch interface
enp6s0f0: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:af txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17 memory 0xe1aa0000-e1ac0000
This will also be a switch interface, controlled by ovswitch
enp6s0f1: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 00:15:17:cc:59:ae txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xe1a40000-e1a60000
This is my loopback
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1 (Local Loopback)
RX packets 120010 bytes 18056633 (18.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 120010 bytes 18056633 (18.0 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
This allows virtual hosts inside my KVM to communicate with the network. This is it’s gateway out
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:58:66:e9 txqueuelen 1000 (Ethernet)
RX packets 21 bytes 2160 (2.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 15 bytes 990 (990.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0-nic: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 52:54:00:58:66:e9 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fc54:ff:fecb:57e5 prefixlen 64 scopeid 0x20
ether fe:54:00:cb:57:e5 txqueuelen 1000 (Ethernet)
RX packets 7 bytes 818 (818.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6656 bytes 346566 (346.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Download VyOS
Verify the SHA1 sums: sha1sum vyos-1.1.7-amd64.iso
Compare to http://packages.vyos.net/iso/release/1.1.7/sha1sums
When installing, leave Gigabit Ethernet0 as main link to server. Use 1st of the 4 new interfaces for the VyOS WAN Interface.
Create a new image:
Tried copying the iso to the server:
$ sftp [email protected]
[email protected]’s password:
Connected to 10.0.0.5.
sftp> cd /var/lib/libvirt/images
sftp> ls
remote readdir(“/var/lib/libvirt/images”): Permission denied
sftp> put vyos-1.1.7-amd64.iso
Uploading vyos-1.1.7-amd64.iso to /var/lib/libvirt/images/vyos-1.1.7-amd64.iso
remote open(“/var/lib/libvirt/images/vyos-1.1.7-amd64.iso”): Permission denied
sftp> quit
This tells me I can’t write to libvirt. Doing an ls -all shows me /var/lib/libvirt/images is owned by root. Entered $sudo chown bruce images
Then was able to put the ISO: (note – there may be better ways of doing this, other than changing the owner of the directory)
sftp> put vyos-1.1.7-amd64.iso
I opened a virt-manager session with my virtual server and got error messages stating: “could not grab keyboard”
Followed instructions at this site to set up my ssh keys between my laptop and my KVM server:
Then I powered it up!
And failure. We are not set to boot from CD! Went in to Information on the VyOS image and clicked Boot Options and set it to boot from CD. Then went in to IDE CDROM1 and set the location of the ISO.
Just like a physical server needs to have a boot image, and can boot from CD or USB on an install, we have the same requirement on the virtual server. So, I set the virtual server to boot from CD and pointed the boot source to my ISO. Success! We have VyOS bootup screen!
Followed VyOS installation instructions found here:
https://wiki.vyos.net/wiki/Installation
You need to install the image, otherwise VyOS will run off the live CD and will not save or boot up with configuration changes.
Next up: configure bridges and Ethernet interfaces on the KVM Server. Followed guide from here, substituting for my needs:
https://community.sinefa.com/hc/en-us/articles/227541808
First we need to shut the virtual machine down and add interfaces through virt-manager. I added the NIC with Bridge brwan. Then hit right click, add hardware and added a second NIC using brlan. (Note: I selected virtio, and it worked. I also tested E1000).
Setting up the network interfaces:
On the KVM Virtual Server, edit interfaces file, adding the following lines (remember, we got the interface names by doing an ifconfig -a):
$ sudo vi /etc/network/interfaces
The LAN port
auto enp5s0f1
iface enp5s0f1 inet manual
The WAN port
auto enp5s0f0
iface enp5s0f0 inet manual
The LAN-side switch
auto brlan
iface brlan inet manual
bridge_ports enp5s0f1
bridge_stp off
bridge_fd 0
bridge_maxwait 0
The WAN-side switch
auto brwan
iface brwan inet manual
bridge_ports enp5s0f0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
I had 2 small desktop switches. I connected my WAN interface to one switch and my LAN interface to the other. (using a higher end switch that supports vlans would also work).
Configure the VyOS Router
What we want to create is this:
LAN—Bridge—VyOS Router----Bridge----WAN
Followed the VyOS Quick Start Guide:
Enter configuration mode:
vyos@vyos$ configure
vyos@vyos#
Configure network interfaces:
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description ‘OUTSIDE’
set interfaces ethernet eth1 address ‘10.0.5.1/24’
set interfaces ethernet eth1 description ‘INSIDE’
Enable SSH for remote management:
set service ssh port ‘22’
Did a commit then a save then a exit.
After doing an exit, did a “show interfaces”
I grabbed an IP via DHCP from my “real” router, meaning my VyOS WAN Connection is talking to my network! I tried pinging, and from VyOS console I can ping my VyOS eth0 interface, my KVM host IP, my KVM gateway IP. I can also ping outside my KVM server and can ping my hardware router and hardware switch. A traceroute shows I have connectivity! I can also ping from network devices outside of KVM in to my VyOS router. Looks like the WAN side is good!
Do a “commit”, “save” and “exit”.
Finished configuring router using the VyOS guideline:
Open up a terminal session and ssh to VyOS: ssh [email protected]
Configure Source NAT for our “Inside” network. Once again, straight from the VyOS guide.
set nat source rule 100 outbound-interface ‘eth0’
set nat source rule 100 source address ‘10.0.5.0/24’
set nat source rule 100 translation address masquerade
Configure a DHCP Server:
set service dhcp-server disabled ‘false’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 default-router ‘10.0.5.1’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 dns-server ‘10.0.5.1’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 domain-name ‘internal-network’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 lease ‘86400’
set service dhcp-server shared-network-name LAN subnet 10.0.5.0/24 start 10.0.5.9 stop ‘10.0.5.254’
And a DNS forwarder:
set service dns forwarding cache-size ‘0’
set service dns forwarding listen-on ‘eth1’
set service dns forwarding name-server ‘8.8.8.8’
set service dns forwarding name-server ‘8.8.4.4’
Add a set of firewall policies for our “Outside” interface:
set firewall name OUTSIDE-IN default-action ‘drop’
set firewall name OUTSIDE-IN rule 10 action ‘accept’
set firewall name OUTSIDE-IN rule 10 state established ‘enable’
set firewall name OUTSIDE-IN rule 10 state related ‘enable’
set firewall name OUTSIDE-LOCAL default-action ‘drop’
set firewall name OUTSIDE-LOCAL rule 10 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 10 state established ‘enable’
set firewall name OUTSIDE-LOCAL rule 10 state related ‘enable’
set firewall name OUTSIDE-LOCAL rule 20 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 20 icmp type-name ‘echo-request’
set firewall name OUTSIDE-LOCAL rule 20 protocol ‘icmp’
set firewall name OUTSIDE-LOCAL rule 20 state new ‘enable’
set firewall name OUTSIDE-LOCAL rule 30 action ‘drop’
set firewall name OUTSIDE-LOCAL rule 30 destination port ‘22’
set firewall name OUTSIDE-LOCAL rule 30 protocol ‘tcp’
set firewall name OUTSIDE-LOCAL rule 30 recent count ‘4’
set firewall name OUTSIDE-LOCAL rule 30 recent time ‘60’
set firewall name OUTSIDE-LOCAL rule 30 state new ‘enable’
set firewall name OUTSIDE-LOCAL rule 31 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 31 destination port ‘22’
set firewall name OUTSIDE-LOCAL rule 31 protocol ‘tcp’
set firewall name OUTSIDE-LOCAL rule 31 state new ‘enable’
set firewall name OUTSIDE-LOCAL rule 32 action ‘accept’
set firewall name OUTSIDE-LOCAL rule 32 destination port ‘161’
set firewall name OUTSIDE-LOCAL rule 32 protocol ‘udp’
Note: originally firewall rules did not allow snmp. Added rule 32 to enable snmp management.
Apply the firewall policies:
set interfaces ethernet eth0 firewall in name ‘OUTSIDE-IN’
set interfaces ethernet eth0 firewall local name ‘OUTSIDE-LOCAL’
Commit changes, save the configuration, and exit configuration mode:
vyos@vyos# commit
vyos@vyos# save
Saving configuration to ‘/config/config.boot’…
Done
vyos@vyos# exit
vyos@vyos$
Communicating through VyOS:
I connected an Ethernet cable between my laptop and a switch that VyOS LAN Interface is connected to. Then I disabled my wireless interface. The laptop can communicate with my home servers (Cacti, nfsen/nfdump and a NAS server). It can also communicate outside my network. For a home user, I am not noticing any major differences between this connection and my wireless connection. My Cisco router did not have a route back to my VyOS LAN interface, so I entered a static route on the Cisco.
How about measurements?
SNMP
VyOS supports snmp. I had to open up the firewall (rule 32 noted above) to allow udp port 161, then configure snmp using the following commands:
set service snmp community (snip)
set service snmp community (snip) client 10.0.1.51
set service snmp community (snip) authorization ro
set service snmp contact “Bubba”
set service snmp location “Bubbatown”
I then started polling VyOS from Cacti (an snmp platform), getting CPU, Memory, Load and Ethernet Interface statistics. I also used the Cacti Advanced Ping tool to give me basic latency between my Cacti server running on CentOS7 in a VMWare esxi 6.0 hypervisor and VyOS router.
ToDos: I am using a generic Linux SNMP template in Cacti. I found no VyOS or Vyatta templates at Cacti site. This is okay for a home router, but I would expect to see snmp collectors for errors, discards and QoS (if configured). More information is available when walking the SNMP MIB, so it could be made available by creating a template in Cacti (or any other preferred snmp management system). I haven’t decided if I want to do this.
Netflow
I am running an OpenSource Netflow server called nfsen/nfdump. So, configure netflow on VyOS:
(I copied this directly based on Google search, just substituted in my Netflow server IP and port).
vyos@vyos# set system flow-accounting interface eth0
vyos@vyos# set system flow-accounting interface eth1
[vyos@vyos# set system flow-accounting netflow server 10.0.1.53 port 9996
vyos@vyos# set system flow-accounting netflow version
Possible completions:
1 NetFlow version 1
5 NetFlow version 5 (default)
9 NetFlow version 9
vyos@vyos# set system flow-accounting netflow version 5
On the Netflow server, configure iptables to allow port 9996 in:
Edited IP tables on nfsen server to allow port 9996:
sudo iptables -I INPUT -p udp -m state --state NEW -m udp --dport 9996 -j ACCEPT
On the Netflow server, add the new source:
Do an “nfsen stop”
Edit nfsen.conf on my CentOS7 nfsen/nfdump server. BASEDIR is /data
vi /data/nfsen/etc/nfsen.conf
Do an “nfsen reconfig”
Then “nfsen start”
nfsen/nfdump is showing zero packets, although TCPDUMP shows I am getting traffic.
Modified my VyOS Netflow configuration:
flow-accounting {
interface eth1
interface eth0
netflow {
engine-id 100
server 10.0.1.53 {
port 9996
}
timeout {
expiry-interval 60
flow-generic 3600
icmp 300
max-active-life 604800
tcp-fin 300
tcp-generic 3600
tcp-rst 120
udp 300
}
version 5
}
}
Still no luck with Netflow. I can see the packets arriving at my Netflow server by doing a TCPDUMP. VyOS and nfsen/nfdump are configured for Netflow Version 5. Setting this one aside for now.Verified flow-accounting on VyOS. VyOS looks good!
Update: I updated my IPTABLES rules, but did not save them before rebooting my nfsen/nfdump server. I was receiving packets on port 9996, but not accepting them. So – update IPTABLES and save it!
sudo iptables -I INPUT -p udp -m state --state NEW -m udp --dport 9996 -j ACCEPT
service iptables save
For now I have my personal laptop and my work laptop connected to a low-end switch connected to my PC running VyOS in KVM. I can view my utilization using snmp. I can view communicating pairs using Netflow. My next goal is to learn a little more about Software Defined Networks and see if I can figure out putting up another virtual server, run ovswitch on the new server and connect my un-used Gigabit interface cards to ovswitch. I’m considering using OpenDaylight for the SDN Controller. (if anyone has experience with ovswitch and how I would add my two un-used interfaces and the VyOS LAN interface to ovswitch, let me know. I’m trying to figure out the hardware and bridge mapping. I think I’d also need to pick up a few cross-over cables to connect my laptop directly to the PC running KVM, VyOS and hopefully ovswitch.
