1.1.0, VRRP, rfc3768, arp replies seem to be filtered

Bugs
, , , , ,
1

We have a pair of Vyos routers, setup with vrrp:

set interfaces ethernet eth0 address '192.168.0.253/24'
set interfaces ethernet eth0 vrrp vrrp-group 10 advertise-interval '2'
set interfaces ethernet eth0 vrrp vrrp-group 10 hello-source-address '192.168.0.253'
set interfaces ethernet eth0 vrrp vrrp-group 10 preempt 'true'
set interfaces ethernet eth0 vrrp vrrp-group 10 preempt-delay '20'
set interfaces ethernet eth0 vrrp vrrp-group 10 priority '200'
set interfaces ethernet eth0 vrrp vrrp-group 10 'rfc3768-compatibility'
set interfaces ethernet eth0 vrrp vrrp-group 10 sync-group 'total'
set interfaces ethernet eth0 vrrp vrrp-group 10 virtual-address '98.189.89.88/28'
set interfaces ethernet eth0 vrrp vrrp-group 10 virtual-address '98.189.89.89/28'
set interfaces ethernet eth0 vrrp vrrp-group 10 virtual-address '98.189.89.90/28'
set interfaces ethernet eth0 vrrp vrrp-group 10 virtual-address '98.189.89.91/28'
set interfaces ethernet eth0 vrrp vrrp-group 10 virtual-address '98.189.89.92/28'

This builds two interfaces, eth0 and eth0v10. When this system is the master, we need to reach the default gateway located in 98.189.89.80/28.

S>* 0.0.0.0/0 [1/0] via 98.189.89.81, eth0v10
C>* 98.189.89.80/28 is directly connected, eth0v10
C>* 127.0.0.0/8 is directly connected, lo
C>* 192.168.0.0/24 is directly connected, eth0

The system will ARP for the mac address of the default gateway. Those arp requests go out from the vrrp source mac (rfc3768-compatibility), which is what we want. The outgoing arp requests are visible to tcpdump on both the physical eth0 and vrrp eth0v10 interfaces. However, the arp replies are only visible on the physical eth0 interface. The arp cache for eth0v10 does not get updated, and the gateway is not reachable.

tcpdump -i eth0 -nn -e arp | grep --line-buffered 98.189.89.81
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:33:06.763463 00:00:5e:00:01:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 98.189.89.81 (ff:ff:ff:ff:ff:ff) tell 98.189.89.88, length 28
16:33:06.779458 00:12:00:1a:39:f7 > 00:00:5e:00:01:0a, ethertype ARP (0x0806), length 60: Reply 98.189.89.81 is-at 00:12:00:1a:39:f7, length 46
16:33:07.763495 00:00:5e:00:01:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 98.189.89.81 (ff:ff:ff:ff:ff:ff) tell 98.189.89.88, length 28
16:33:07.783566 00:12:00:1a:39:f7 > 00:00:5e:00:01:0a, ethertype ARP (0x0806), length 60: Reply 98.189.89.81 is-at 00:12:00:1a:39:f7, length 46
16:33:08.763521 00:00:5e:00:01:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 98.189.89.81 (ff:ff:ff:ff:ff:ff) tell 98.189.89.88, length 28
16:33:08.772444 00:12:00:1a:39:f7 > 00:00:5e:00:01:0a, ethertype ARP (0x0806), length 60: Reply 98.189.89.81 is-at 00:12:00:1a:39:f7, length 46
tcpdump -i eth0v10 -nn -e arp | grep --line-buffered 98.189.89.81
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0v10, link-type EN10MB (Ethernet), capture size 65535 bytes
16:33:47.082439 00:00:5e:00:01:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 98.189.89.81 (ff:ff:ff:ff:ff:ff) tell 98.189.89.88, length 28
16:33:48.082510 00:00:5e:00:01:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 98.189.89.81 (ff:ff:ff:ff:ff:ff) tell 98.189.89.88, length 28
16:33:49.082541 00:00:5e:00:01:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 98.189.89.81 (ff:ff:ff:ff:ff:ff) tell 98.189.89.88, length 28

I tried adding arp_accept to both eth0 and eth0v10, but that did not help. arp_accept on eth0 will get 98.189.89.81 added to the arp cache for eth0, but that entry won’t be used for outgoing packets on eth0v10.

2

Hi,

Are these physical routers or are they virtual? I know from working with this extensively on VMware that RFC VRRP on VMware requires the port group to be promiscuous whereas NON-RFC VRRP does NOT require promiscuous but DOES require ‘accept MAC Address Changes’

If you are using physical devices, none of this relates to you.

SteveP

3

One physical, one virtual. The tcpdumps above are from the physical one, which is the primary. The virtual one is the backup. I don’t see any difference in behavior between the two. The virtual one is running under KVM under Centos6.

4

This is getting complicated. Vyos VRRP is built on top of keepalived, which uses the macvlan module to create the VRRP interfaces dynamically. See Some notes on macvlan/macvtap « \1

The stock macvlan module supports 4 modes (private, vepa, bridge and passthru). The comment for commit 19676d75be92db9ccdf68fbeac52bde4273911b0 in pkgs/linux-image is:

=======

Add special macvlan mode for VRRP.

This mode allows VRRP to function closer to other vendor implementations. With VRRP users expect that the receive path for their traffic will be the same interface that they configure VRRP to run on, due to the way other vendor implementations work. Using macvlan the user must configure two interfaces in iptables configuration in order for all traffic to be inspected. The patch adds a VRRP mode to the macvlan driver such that any traffic destined to the virtual VRRP mac address will appear to come from the underlying interface instead of macvlan interface. The patch accomplishes this by utilizing the RX_HANDLER_PASS return value in the macvlan receive handler code. The skb is modified with the dev set to the lowerdev (the physical interface) and the appropriate pkt_type. The causes the rest of the networking stack to treat the skb as if it came from lowerdev.

In order to use this effectively, one must open two raw sockets in the userspace code acting as the VRRP daemon, one to send on the macvlan interface and the other to listen on the underlying interface for the VRRP messages.

=======

So the outbound packets will originate from eth0v10, but the inbound packets seem to arrive from eth0, not eth0v10. That is consistent with what I see with NAT rules.

set nat destination rule 10 inbound-interface 'eth0'
set nat source rule 10 outbound-interface 'eth0v10'

However, the default gateway is located in the same /28 as the virtual ip addresses that are being managed by VRRP. So when this device becomes the master router,

set protocols static route 0.0.0.0/0 next-hop '98.189.89.81'

the routing table sends everything out via eth0v10, which is what we want. Everything includs the ARP request for the mac address of the gateway. When the ARP reply comes in, the MACVLAN_MODE_VRRP causes it to appear from eth0, not from eth0v10. So the arp code ignores it, and we have no arp cache entry for the gateway on eth0v10.

I am not sure how to fix this. I think with rfc3768-compatibility it makes more sense just run ALL the traffic over eth0v10. That way the NAT inbound and outbound interfaces are consistent, the ARP replies will appear on the proper interface, and everything should just work.

Alternatively, we could hack the macvlan module to special case ARP replies in MACVLAN_MODE_VRRP, and leave those replies on the upper (eth0v10) interface.

In the case of NOT rfc3768-compatibility, I have not really looked at what mac address Vyos uses for normal inbound/outbound/arp traffic.

5

A more complete writeup, with a possible solution, is at 510 Software Group

6

Details at 510 Software Group but I have a fix attached that works for me.

7

patch to allow firewall configuration on vrrp interfaces.

8

Hi,

Is it an easy way to use these patches?

Regards
Arne

9

Not without rebuilding vyos from source. I can provide the amd64.iso that I am using in production, but I had not (at that time) built a 32 bit iso.

10

It would be great!

Is it 1.1.1 or 1.1.0 you are running in production?

11

It is not an official release, just my development build from December 4th.

Version:      VyOS 999.lithium.12041239
Description:  999.lithium.12041239
Copyright:    2014 VyOS maintainers and contributors
Built by:     root@contributors.vyos.net
Built on:     Thu Dec  4 20:39:33 UTC 2014
Build ID:     1412042039-a066514
System type:  x86 64-bit

http://www.five-ten-sg.com/util/VyOS-livecd-1412042039-a066514-amd64.iso

12

Hi Carl

I’m getting a Error 404, Object not found when I try to download your iso file.

13

Thanks!
And this is a great start on the new year!

I have tested your ISO at it looks a lot better. Is it any plans to get these changes into the standard release?

14

Hi Carl
I have bumped into another missing part in this.

When using VLAN and VRRP the possibility to configure a firewall for the interface is lacking.
Unfortunately I’m lacking the understanding of the content in your patches to be sure how many … and / to place in the correct spot in combination with vif/node.tag and vrrp/vrrp-group/node.tag.

Do you have a hint?

Thanks in advance.

15

That all seems to be in pkgs/vyatta-cfg-firewall/gen-interface-templates.pl

There are two hashs, %interface_hash and %firewall_hash, and I think if those are updated the rest of the firewall stuff should just work.

Firewalls should be allowed for all of:

set interfaces bonding TXT<bondn> vif U32<0-4094> vrrp vrrp-group
set interfaces bonding TXT<bondn> vif-s U32<0-4094> vif-c U32<0-4094> vrrp vrrp-group
set interfaces bonding TXT<bondn> vif-s U32<0-4094> vrrp vrrp-group
set interfaces bonding TXT<bondn> vrrp vrrp-group
set interfaces ethernet TXT<ethn> vif U32<0-4094> vrrp vrrp-group
set interfaces ethernet TXT<ethn> vif-s U32<0-4094> vif-c U32<0-4094> vrrp vrrp-group
set interfaces ethernet TXT<ethn> vif-s U32<0-4094> vrrp vrrp-group
set interfaces ethernet TXT<ethn> vrrp vrrp-group

I only added it for the last one. We should add some documentation in that code to make it easier to add such stuff. The interface hash computes the interface name - eth0v10 for vrrp over ethernet. I presume that should be eth0.10v20 for vrrp over ethernet vlan, etc. I don’t recall now exactly what the firewall hash was computing.

16

I managed to get it working. But I don’t understand the ‘The keys of this hash are the partial pathname under the config template tree “interfaces/”’. Is there any other documentation which explains the config template tree?

17

On an installed system:

ls -al /opt/vyatta/share/vyatta-cfg/templates/interfaces

pkgs/vyatta-cfg-firewall/gen-interface-templates.pl uses those hashes to create files that ultimately get installed in the above path.

If you have a git format patch for the vyatta-cfg-firewall package, I can add it to mine and rebuild an iso image.

This should also be put into the bug tracker since I presume others might want firewalls on vrrp over vlans.

18

I will try to create the patch.

But I think it may be time for some more formal work also. As I’m new to “behind the scenes” here I don’t have the knowledge of the documentation of the requirements and impact analysis. Because as we just changes things we may break other parts. I discovered that the DHCP-Relay don’t work with this approach. What about the routing protocols etc?

I think it’s nice to fix things but we must fix it right.

19

What interface are you using for ‘set service dhcp-relay interface’ ? That should be eth0v10 rather than eth0 now. The only traffic on eth0 (assuming vrrp with eth0v10) should be the multicast stuff between the two vrrp routers. All the other traffic - (arp, dhcp, firewalls, etc) should be on eth0v10.

But if that does not work, then yes, there is a bug here.

20

I have created a different peer of routers for testing this again:

vyos@vyos-03# set service dhcp-relay interface eth2v21
[edit]
vyos@vyos-03# commit
[ service dhcp-relay ]
DHCP relay configuration error.  DHCP relay interface "eth2v21" specified has not been configured.

[[service dhcp-relay]] failed
Commit failed
[edit]
vyos@vyos-03# set service dhcp-relay interface eth2
[edit]
vyos@vyos-03# del service dhcp-relay interface eth2v21
[edit]
vyos@vyos-03# commit
[ service dhcp-relay ]
Stopping dhcrelay:  OK
Starting dhcrelay:  OK

[edit]

vyos@vyos-03# run sh int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.168.111.10/31                 u/u
eth0v21          172.16.6.18/29                    u/u
eth1             192.168.111.12/31                 u/u
eth1v21          172.16.7.1/24                     u/u
eth2             192.168.111.14/31                 u/u
eth2v21          172.16.8.1/24                     u/u
eth3             192.168.254.9/30                  u/u
lo               127.0.0.1/8                       u/u
                 192.168.255.4/32
                 ::1/128

So it don’t like to run the dhcp-relay on the virtual interface.

Setting OSPF interface parameters on the virtual interface is also broken.