I have issue with my vyos routers. It was up and running for few years now but starting this year only sometimes I’ve lost connection to the router. When I access the router via VM console, the interface ip was missing. All vti, dummy and loopback interface also was gone. Like below :
vyos@Router1:~$ sh interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 - A/D Public interface
eth1 - A/D
lo 127.0.0.1/8 u/u
::1/128
But when I run show configuration, all the configuration was there.
I need to reboot the router for all the interfaces coming back again.
Its happen to both 1.1.8 and 1.3.1 VyOS routers.
Its happened 3 times this year to my 1.1.8 and 1 time to my 1.3.1 router.
Does anyone know what is happen to my vRouter?
Is your router a physical piece of hardware, or is it a virtual machine? What do the logs show at the time? The best thing to do would be to capture the output of “show log” and also the capture of “sudo dmesg”
It sounds like a driver problem causing the underlying interfaces to be removed from the Linux kernel.
Edit: sorry I can see you say quite clearly this is running in a virtual machine. I would definitely capture those logs. Also have a look at the virtual machine. Does it experience any problems or issues that might cause the interfaces to be removed? For example, are you doing a v-motion or something odd like that that’s causing issues?
Since VyOS maps the mac-address of the network interface to which ethX it will use in VyOS config (that hw-id setting in each interface ethernet ethX) I would also check for if the mac-addresses changes over time.
As @tjh suggested perhaps vmotion or similar feature moves the vm guest to a new vm host and based on how you configured stuff in the vm host the VyOS guest then suddently will see a new set of network interfaces (based on mac-address) and because of that current ip etc gets unset.
This is the log during the issue. All interfaces was deleted. Mac address was the same during the issue and after I reboot the machine.
I’m not check dmesg during the downtime. I will do it if the issue happen again.
About the v-motion or similar feature I will check with my Infrastructure team that handle the servers.
I will check with our cloud provider as well on this and let you know.
Create /config/custom/dump_macaddress.sh and make that executable (dont forget to set TIMER to whatever you find is suitable, the value is in seconds between each dump to the logfile):
#!/bin/sh
# Script debugging
#set -x
# Set variables
TIMER=60
LOGFILE='/config/custom/dump_macaddress.log'
# Perform stuff
while true
do
NOW=$(date +"%y%m%d %H:%M:%S")
echo ${NOW} >> ${LOGFILE}
ip link >> ${LOGFILE}
echo >> ${LOGFILE}
sleep ${TIMER}
done
Then add this to /config/scripts/vyos-postconfig-bootup.script:
#!/bin/sh
# This script is executed at boot time after VyOS configuration is fully applied.
# Any modifications required to work around unfixed bugs
# or use services not available through the VyOS CLI system can be placed here.
/config/custom/dump_macaddress.sh &
And finally reboot the box.
After boot the dump_macadress.sh will be runned in a loop dumping info regarding which mac-addresses is seen by the OS every seconds defined by that TIMER variable.
You will find the dump in /config/custom/dump_macaddress.log (or whatever you set the LOGFILE variable to).
To make it stop either kill the process by finding out its pid:
Or remove the call to the script from /config/scripts/vyos-postconfig-bootup.script and reboot.
Hopefully above (depending on TIMER value) should be enough to spot if something fishy is going on with the mac addresses (if they are involved in this at all).
Edit: Originally posted this reply in the wrong thread.
