Can't SSH Suddenly

Not sure what happened, been using VyOS for years and suddenly today I cannot access both my VyOS routers via SSH despite that no recent config changes were made at all. Services are up otherwise. Strange thing is, I also am no longer able to access by sub Cisco switches that are used as L2 below the VyOS routers. I rebooted both routers and strangely was able to access everything via SSH for a minute, then again no longer again afterwards. I tried via my wifi and vpn, I tried via two SSH programs too.

I’m running version 1.2.6-S1 on both routers.

What could possibly cause this? I have IPMI windows open for both routers at least.

Hi @Bashed , try to check free space. Also via IPMI try to monitor logging when you connecting via SSH

I’m not too technically inclined here as my normal network tech has disappeared on me. I know some basic things with VyOS. What’s the command to log VyOS when trying to connect via SSH?

Below disk space looks good but not sure about that loop partition on each router. Is this normal?

I ran ‘show log’ and this is the output on each router.

Further update. I am unable to ping from my home wifi to any VyOS / Cisco devices, but the gateway IPs do ping. All public device IPs however do ping, from online tools and the data center NOC confirmed they can ping as well. My local home IP is static, that hasn’t changed and been whitelisted according to previous network tech.

I think you need to begin debug with traceroute from your PC to VyOS router. And then try to get this packets on VyOS WAN interface via monitor traffic interface ethX filter icmp
Also try to temporary disable firewall.

Thanks for the follow up. Here’s the update and a strange issue.

If I shut down VyOS #2 router, I can finally access all Cisco switches and VyOS #1. However, as a result though some of my clients go down when #2 VyOS is down. But, at least I’ve troubleshooted to this point.

What could possibly cause this odd behavior? Hopefully we can figure this out.

Hi. It can be overlaped ip route or duplicate IP address.

Could I trouble you for the commands/steps I need to run to check this?

I’m familiar with the basic stuff:
show protocols static route
show ip bgp
show protocols bgp xxxxxx address-family ipv4-unicast network

Network topology with IP addresses will be useful

How is this done via VyOS? My network tech is MIA and I need some help here please. Thank you.

Can you show configs of these two VyOS routers?

Yes. Had to x out some private info obviously for security.

Pastebin below.

VyOS 1
https://pastebin.com/hz4Un3NU
pw: 6cJjEU29RU

VyOS 2
https://pastebin.com/cFKKz5KZ
pw: vzDVxSDdKL

OK - I would suggest to do the below thing and confirm. The same thing happened as the device stopeed responding to ICMP as well as SSH.

Give any IP address to any of the interface. Could be fake and say commit and save.

As I said same thing happened and I was troubleshooting for almost 5 hours. Then I decide to run commit so I gave IP address to other not used interface and then ran commit.

Voila - the device came back

Sorry I really do not understand what you’re saying. I’ve already been modifying IP assignment per VLAN (by logging in via IPMI console) and already committed/saved. Issue not resolved.

Ahh - Then that could be a bug. Well in my case; I wasnt sure what had happened however as I said my appliance loaded with 1.3.4 stopped responding suddenly and then I had to commit and save that brought the device back.

  1. If you do not use ssh-in acl, delete it.
  2. Check ssh listen-addresses on VyOS#2. Are they actual?
  3. Look at the routing table and find information about IP from which you try to ping. And try to understand how packet can flow from and back.
  4. Check VRRP status on both routers.

I think your problem is between routing, vrrp, and maybe listen-addresses.

Please provide me the commands & steps to do this. My network guy is MIA and I have to figure this out myself.

As I have understood, you have a problem with ssh access not only to VyOS but to Cisco too.
So.
On your host where you could not get ssh access to Cisco

  1. Run ping and traceroute to Cisco IP
  2. Run ping and traceroute to each VyOS

On each VyOS

  1. Try ssh to Cisco IP
  2. Run show vrrp
  3. Run show ip route Your_host_Public_IP

Then show all results.

VyOS to Cisco

Ping: good from VyOS (both 1 and 2) to all 4 Cisco switches
Traceroute: immediate timeout first hop “*****” results from both VyOS to any Cisco switch.

I was able to ssh into Cisco 1 from VyOS 1. However, “show vrrp” gave blank results.

Please see attached for “show ip route”

I was able to ping from Cisco 1 to VyOS (both).