Conntrack-sync whitelisting IP/TCP/UDP ports for state replication

General questions
, ,
1

Hi VyOs community!
I am setting up a community built version of VyOs 1.3 on GCP, running as HA NAT gateway with VRRP failover configuration.

On the current test setup, VRRP and state replication works as expected: when the primary VyOs instance is switched off, the connections survive the failure and traffic is “handled” by the second VyOs instance. However, to make this work, I’ve whitelisted ALL traffico from Vyos1 to Vyos2 and vice versa.

Now, in order to move this setup into production, I’d like to understand what UDP/TCP/IP flows must be opened between the two VyOs instances in order to make VRRP and conntrack-sync work as expected.

For VRRP, I know I should whitelist traffic at L3, protocol number 112. How about conntrack-sync?

EDIT

I’ve seen from /etc/conntrackd/conntrackd.conf that the sync module is configured to use UDP on port 3780. Is this the only port I need to whitelist? Is there any way I can set this port at VyOs configuration level ?

2

Greetings albertogeniola, I hope you’re doing well,

You’re right, VRRP uses IP protocol number 112.

Now about conntrack-sync you can choose between multicast:
set service conntrack-sync mcast-group <x.x.x.x>
where x.x.x.x is the multicast group (by default is 225.0.0.50)
or UDP unicast:
set service conntrack-sync interface <name> peer <address>

As you said, default port for the conntrack daemon is 3780 for UDP. I suggest to use the default settings, but is technically possible to edit the port value, the thing is you’ll need to restart the daemon every time you boot VyOS (you can make a script to be executed when booting up, but if you ask me its messy)

You can edit the conntrackd.conf port value to change the used port to the peer dst address
See the example below i’m setting the port 3030 for the sync

nano /etc/conntrackd/conntrackd.conf

Sync {
	Mode FTFW {
	}
	UDP {
		IPv4_Destination_Address 30.20.20.5
        Port 3030
		Interface eth0
		SndSocketBuffer 104857600
		RcvSocketBuffer 104857600
		Checksum on
	}

then


vyos@vyos:~$ config
[edit]
vyos@vyos# save
Saving configuration to '/config/config.boot'...
Done

Finally restart the conntrack daemon

vyos@vyos:~$ sudo su
root@vyos:/home/vyos# systemctl restart conntrackd
root@vyos:/home/vyos# exit
exit
vyos@vyos:~$ show conntra
conntrack       conntrack-sync  
vyos@vyos:~$ show conntrack table ipv4
TCP state codes: SS - SYN SENT, SR - SYN RECEIVED, ES - ESTABLISHED,
                 FW - FIN WAIT, CW - CLOSE WAIT, LA - LAST ACK,
                 TW - TIME WAIT, CL - CLOSE, LI - LISTEN

CONN ID    Source                 Destination            Protocol         TIMEOUT             
**3065503419 192.168.0.17:49128     30.20.20.5:3030        udp [17]         29**                  
175236397  127.0.0.1:42787        127.0.0.1:53           udp [17]         26                  
2399091370 192.168.0.1:9431       192.168.0.255:9431     udp [17]         29                  
3217183247 192.168.0.17           224.0.0.22             igmp [2]         526                 
2907196130 127.0.0.1:40784        127.0.0.1:53           udp [17]         26                  
1232083918 192.168.0.100:51746    192.168.0.17:22        tcp [6] ES       432000              
3187166620 127.0.0.1:33052        127.0.0.1:53           udp [17]         26                  
2082549832 127.0.0.1:53501        127.0.0.1:53           udp [17]         26                  
17203877   127.0.0.1:41238        127.0.0.1:53           udp [17]         26                  
2667057888 127.0.0.1:54693        127.0.0.1:53           udp [17]         26                  
2562212056 192.168.0.1            224.0.0.1              igmp [2]         519                 
1480496972 192.168.0.17           224.0.0.18             vrrp [112]       599
3

Hi M1nus,

Thanks for th answer. About multicast, that is not supported on GCP, so I went on Unicast peering.

About the port change configuration, I see that your suggestion is to edit th conntrackd cond file directly and then save the configuration. Will this store the custom UDP port in the VyOs configuration file that is applied at boot? What happens if I change the conntrackd file and restart (without saving on VyOs)?

4

Hi albertogeniola,

If you reboot your VyOS instance or you run a configuration command for conntrack, a perl script will build the conntrackd.conf again, and changes will be overwritten. That’s why I suggested to run this the default way.

However, you can set up a script to run everytime you boot your VyOS and change the port at conntrackd.conf.

For example:
I’ve created a conntrackd.conf file at /config/scripts

vyos@vyos:~$ cd /config/scripts
vyos@vyos:/config/scripts$ ls
conntrackd.conf  vyos-postconfig-bootup.script

Then you can copy /etc/conntrackd/conntrackd.conf content
nano /config/scripts/conntrackd.conf
paste the content and change only the port value.

vyos-postconfig-bootup.script it is a script that runs immediatly after booting, so here we will replace /etc/conntrackd/conntrackd.conf with /config/scripts/conntrackd.conf that we’ve created

vyos@vyos:/config/scripts$ sudo nano vyos-postconfig-bootup.script

!/bin/sh
#This script is executed at boot time after VyOS configuration is fully applied.
#Any modifications required to work around unfixed bugs
#or use services not available through the VyOS CLI system can be placed here.
sudo su
systemctl stop conntrackd
cp /config/scripts/conntrackd.conf /etc/conntrackd/conntrackd.conf
systemctl start conntrackd
exit

the script says that everytime you boot VyOS you’re going to: stop conntrack daemon > replace conntrackd.conf with the one we’ve created and changed the port > run conntrack daemon

vyos@vyos:~$ show conntrack table ipv4
TCP state codes: SS - SYN SENT, SR - SYN RECEIVED, ES - ESTABLISHED,
                 FW - FIN WAIT, CW - CLOSE WAIT, LA - LAST ACK,
                 TW - TIME WAIT, CL - CLOSE, LI - LISTEN

CONN ID    Source                 Destination            Protocol         TIMEOUT             
1475121917 192.168.0.100:43920    192.168.0.17:22        tcp [6] ES       432000              
3379404573 127.0.0.1:35646        127.0.0.1:53           udp [17]         9                   
418802928  192.168.0.17:54820     10.10.5.5:9999         udp [17]         29                  
1715118500 127.0.0.1:41998        127.0.0.1:53           udp [17]         9                   
3047112562 192.168.0.17           224.0.0.18             vrrp [112]       599                 
3098682239 192.168.0.1:9431       192.168.0.255:9431     udp [17]         29                  
3263140913 127.0.0.1:33912        127.0.0.1:53           udp [17]         3                   
1569728376 127.0.0.1:44081        127.0.0.1:53           udp [17]         1                   
2212209515 127.0.0.1:47051        127.0.0.1:53           udp [17]         3                   
1993901470 127.0.0.1:46658        127.0.0.1:53           udp [17]         1                   
2488012074 192.168.0.4:57621      192.168.0.255:57621    udp [17]         0                   
2016708421 127.0.0.1:60911        127.0.0.1:53           udp [17]         3                   
4003937088 127.0.0.1:49845        127.0.0.1:53           udp [17]         3                   
3638826111 192.168.0.17           224.0.0.22             igmp [2]         421

You can see it shows at the conntrack table that I changed the port to UDP 9999 for unicast dst peer 10.10.5.5

5

Hi m1nus,

thanks for the hint. However, I feel like your proposal (which I came across on my own too) is not optimal: most of the conntrack configuration is issued via set service conntrack-sync ... commands which produce a nice configuration state file (within /config directory) that is applied at boot time. So far, the target configuration of the VyOs instance is described in those file and hacking with custom scripts simply doesn’t feel right. I believe we should ask for a feature request to add a port subcommand for set service conntrack-sync interface <name> peer <address>: something like set service conntrack-sync interface <name> peer <address> port <port>.

6

Why require a non-standard port to begin with?
Changing port hardly adds security.
As this is a VRRP setup , VyOs instances are close to each other. On hardware devices, normally a dedicated port and crosslink cable in between are used. And you can firewall on source IP address, input interface…
If you really want to protect conntrack-sync, consider using a tunnel

7

Hi 16again,
I never said it’s for security, quite the opposite. The reason why I wanted to be able to specify a VRRP port is because I am setting up a VyOs HA nat module using Terraform on GCP and the desired vyos configuration is injected at Terraform level. I just want to override the default VRRP port with a terraform value (which matches the default vrrp port): this would allow be to keep a configuration binding at terraform level so that every time the VRRP port is changed at terraform level, both the VyOs port and the GCP Firewall rules are updated accordingly.

If I don’t do that, there is a risk: the user might change the VRRP port from inside VyOs, but the GCP firewall rule won’t be updated, leading to VRRP failure.

I hope it’s a bit clearer now :slight_smile:

8

Hi @albertogeniola the port 3780 is harcoded so always the /etc/conntrackd/conntrackd.conf is created port will be overwritten. I’m trying to get the feature developed so you can be able to change the port.

Can you please run the following command at your VyOS and share the output with me?

show version

Thanks!

9

Hi m1nus,

image
image900×914 49.5 KB

It’ a source-built Equuleus version, pulled from master on May 21.

Thanks a lot for your support!

10

Hi @albertogeniola hope you’re doing great.

The command

vyos@vyos# set service conntrack-sync interface <interface> port 
Possible completions:
   <1-65535>    Numeric IP port

was added to the last nightly build for VyOS 1.3 (equuleus)

You’ll find the ISO here:

vyos-1.3-beta-202107032242-amd64.iso

I’ve tested it and seems to work right. Let me know if you’re able to change the default port too!

11

Hi m1nus,
that’s simply awesome. Thanks !

12

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.