Unable to bind listen on address as VRRP interfaces not yet assigned

blackhole · March 29, 2020, 4:28pm

Hi all, I am on rolling release “VyOS 1.3-rolling-202002030754” and running inside a VM on Server 2019 with Hyper-V role.

I am still playing with my config and recently moved DHCP completely from VyOS to Microsoft DHCP, also I have moved VRRP interfaces to use 10.168.[vlan].253/24 networks and virtual interfaces to 192.168.[vlan].253/24 networks, but both interfaces running on same zone for firewalling for time being.

Today I just rebooted the VyOS router for first time since these changes and I have a fair few issues.
It seems VRRP is not loaded up before the listen on interface is assigned, as a result, I do not have ability to SSH in to VyOS anymore, even after several reboots.
I may make another topic but it seems seems there may be something potentially with DHCP relay service as well after the reboot, it did not like the reboot and I had to restart the dhcp relay service after so most of my home devices on lan could connect.

All of this I am doing from the Hyper V console so bare with me, only images for time being and error.
VyOS management interface is on eth2.7v7 on ip 192.168.7.253

Error is in red above:
error: Bind to port 22 on 192.168.7.253 failed. Cannot assign requested address.
fatal: Cannot bind any address.

In screenshot 2 you can see eth2.7v7 loading up, and even later then that this VyOS becomes master

Any hints/tips/tricks please?
Your help is appreciated, many thanks!

I have found I managed to backup the config recently, this is exact config (only relevant sections shown currently)

high-availability {
vrrp {
group mgmt {
advertise-interval 1
description Management
hello-source-address 10.168.7.253
interface eth2.7
peer-address 10.168.7.254
priority 255
rfc3768-compatibility
virtual-address 192.168.7.253/24
vrid 7
}
}
}

interfaces {
ethernet eth2 {
duplex auto
hw-id 00:15:5d:01:9f:14
speed auto
vif 7 {
address 10.168.7.253/24
description Management
}
}
}

service {
dhcp-relay {
interface eth1.131v131
interface eth0.67v67
interface eth1.13v13
interface eth1.11v11
relay-options {
relay-agents-packets discard
}
server 192.168.67.241
server 192.168.67.242
}
mdns {
repeater {
interface eth1.11v11
interface eth1.13v13
}
}
ssh {
listen-address 192.168.7.253
port 22
}
}

runar · March 29, 2020, 5:44pm

Hi blackhole!

From your configuration i see two issues, for the first the vrrp virtual-address should not be the same as your interface address, the virtual-address should be a “third” ip in the vrrp setup. the third ip will be addigned to the virtual vrrp subinterface (ethX.#v# interface)
if you want to use .253 as your virtual-address, i suggest you use .252 as your interface ip.

The other fault is that you have binded your services to the virtual vrrp interface (ethX.#v# interface), these services should you bind to the configured subinterface instead (ethX.# interface).

other than that, the configuration looks fine

blackhole · March 29, 2020, 11:09pm

Thanks for the quick response @runar, very much appreciate it.
While I am digesting this information, I am slightly conflicted I guess.

My virtual-address is different, its on 192.x.x.x as opposed to the original interface addresses of 10.x.x.x. I thought this was ok, as well as only 2 addresses needed for each VRRP, per router, one virtual and one original interface? ie: 3 IP between 2 routers as the virtual would be same between them.

I have in the past used the same network of 192.x.x.x for original interfaces too, but switched to 10.x.x.x while diagnosing issues with DHCP (no longer on VyOS) as it was complaining about multiple IP and unable to assign non reserved addresses.

In regards to the bindings, this is where I am now most confused as I believe I should be assigning to the virtual interface, as that is the interface with the “correct” or desired overall end IP

As per this screenshot of interfaces, apologies on screenshots, I would rather the text but can not do it without SSH yet

And this deliberate action I took from my laptop I am on, to a port thats not allowed in DMZ, both source and destination interfaces are the virtual interfaces, as I was expecting.

I am relatively new to VRRP, and I am not dismissing anything, just logically trying to work through everything here as this is turning everything I knew upside down if it is right.

Cheers!

Frank · April 1, 2020, 11:02am

Problem is that the ip address you try to bind to is the virtual address and that it is not always available. You can enable binding to non existing addresses by specifying net.ipv4.ip_nonlocal_bind = 1 in /etc/sysctl.conf or (better) creating a separate file in /etc/sysctl.d/ with this setting. Make sure the file name ends with .conf e.g. /etc/sysctl.d/99-nonlocalbind.conf

sysctl -p should activate the new setting. Otherwise a reboot will.

blackhole · April 1, 2020, 10:24pm

Thanks @Frank for your input as well. I suspected it may actually have something to do with a non existent interface/address.

This is now raising a further question if I may please. I don’t have the backup VyOS fully in order yet till I am 100% satisfied on the 1st router, I didn’t want to introduce complexity/new problems.
The problem I see now is, I am VRRP’ing the entire VLAN structure over, including the management VLAN, this is going to create havoc on the second VyOS instance, and first if it goes faulty, I wont be able to SSH/manage apart from the console.

I am trying to think, with such a setup, what is the best practice? Am I best off assigning another ip on the original interface (10.168.7.x) that is in the same network as the virtual IP (192.168.7.x), or change the original interface back over to the same as virtual network 192.168.7.x and assign both the virtual and original addresses to the listen-address.

Seems there is some information missing, I am happy to write some documentation, but I am learning VRRP as I go and not sure what would be the best way yet.

Thanks all!

blackhole · April 1, 2020, 10:30pm

I should say, logically, to me, it makes most sense to assign the original interface on same network as the virtual address

blackhole · April 1, 2020, 11:19pm

After some coffee… the best solution appears to be

Router 1 - Original interface 192.168.x.252
Router 1 - Virtual interface 192.168.x.253

Router 2 - Original interface 192.168.x.254
Router 2 - Virtual interface 192.168.x.253

Router 1 and 2 listen-address will have just the original interface ip’s instead of the virtual…

Will test this out over the day
Cheers!

blackhole · April 3, 2020, 10:56am

Well, just as follow up in case anyone is interested down the track. What I wrote in my post just above seems to be winner configuration.

Now left to do before setting up the secondary VyOS and get VRRP actually functioning (apart from conntrack-sync not working/bug)
mDNS - put into VRRP transition script (currently having to disable and re-enable on restart of VyOS as only using on virtual interfaces)
dhcp-relay - put into VRRP transition script (as using on virtual interfaces only and they need to be up, currently manually restarting)
nat - put into VRRP transition script, according to doco
wan interface disable/enable - put into VRRP transition script as only have single WAN link

conntrack-sync has a bug in it and I can not enable it
Tracking bug via ⚓ T1538 Update conntrack-sync packages to fix VRRP issues

Cheers

blackhole · April 4, 2020, 1:00am

All transition script content is finished, all from one script, it just uses the case statements instead of having multiple scripts to maintain.
My VRRP syncs all networks, so I only configured group “lan” to have the script run, and it would take care of everything.

Tested working great! Apart from the WAN interface, seems to have had issue i wanted to just disable the interface and delete the disable part, but not sure what happened. Will retest, atm WAN interface switches between fake network and dhcp.

YAY!

#!/bin/vbash
STATE=$1
INTERFACE=$2
GROUP=$3

source /opt/vyatta/etc/functions/script-template

LOG_DIR=/config/log
LOG_FILE=$LOG_DIR/vrrp-transition.log

if [ ! -d "$LOG_DIR" ]; then
	mkdir -p $LOG_DIR
fi

things_enable() {
	# Enter configuration mode
	configure

	# Configure WAN interface
	delete interfaces ethernet eth0 vif 167 address 192.168.167.167/24
	set interfaces ethernet eth0 vif 167 address dhcp

	# Configure NAT
	delete nat destination rule 200 disable
	delete nat destination rule 201 disable
	delete nat destination rule 399 disable
	delete nat destination rule 400 disable
	delete nat destination rule 401 disable
	delete nat source rule 5010 disable

	# Configure  mDNS repeater
	set service mdns repeater interface 'eth1.11v11'
	set service mdns repeater interface 'eth1.13v13'

	# Configure DHCP relay
	set service dhcp-relay interface 'eth1.131v131'
	set service dhcp-relay interface 'eth0.67v67'
	set service dhcp-relay interface 'eth1.13v13'
	set service dhcp-relay interface 'eth1.11v11'
	set service dhcp-relay relay-options relay-agents-packets 'discard'
	set service dhcp-relay server '192.168.67.241'
	set service dhcp-relay server '192.168.67.242'

	# Commit
	commit
	exit
}

things_disable() {
	# Enter configuration mode
	configure

	# Configure NAT
	set nat destination rule 200 disable
	set nat destination rule 201 disable
	set nat destination rule 399 disable
	set nat destination rule 400 disable
	set nat destination rule 401 disable
	set nat source rule 5010 disable

	# Configure WAN interface, set "fake" interface ip instead of disabling interface (may have had issue firewall didnt load?)
	delete interfaces ethernet eth0 vif 167 address dhcp
	set interfaces ethernet eth0 vif 167 address 192.168.167.167/24

	# Configure  mDNS repeater
	delete service mdns

	# Configure DHCP relay
	delete service dhcp-relay

	# Commit
	commit
	exit
}

if [ "$GROUP" = "lan" ]; then
case $STATE in
		"master")
			echo "$(date) - VRRP Entering MASTER state" >> $LOG_FILE
			things_enable
		;;
		"backup")
			echo "$(date) - VRRP Entering BACKUP state" >> $LOG_FILE
			things_disable
		;;
		"fault")
			echo "$(date) - VRRP Entering FAULT state" >> $LOG_FILE
			things_disable
		;;
		"stop")
			echo "$(date) - VRRP Entering STOP state" >> $LOG_FILE
			things_disable
		;;
	esac
fi