Problem with SSH

Bugs
, , , ,
1

I tried this in the users forum but couldn’t make any progress. Whenever I reboot Vyos, I cannot ssh into the system. In order to do so, I need to connect through the console and make a configuration change that causes the the SSHD service to restart. Once that happens, I can get in and everything is good.

Typically, I’ll log in through the console, add another listen-address to the SSH configuration and commit. This causes SSHD to restart and that allows me to connect.

Here’s my ssh config:

ssh {
listen-address 100.64.20.1
listen-address 100.64.10.1
port 22
}

The configuration is saved and after I reboot, I see everything intact when I log into the console. Apparently the SSHD program either doesn’t start on reboot or doesn’t correctly read its configuration. Any ideas anyone?

2

I guess this isn’t a bug but a feature because nobody is answering…

3

it’s probably because nobody can reproduce it and don’t know where to begin looking…

or you know, it could be because this is a volunteer forum…

PS. The source is open, feel free to dig around.

4

It is interesting because it occurs on the 3 servers on which I have installed vyos. It is apparently only reproducible by me and not on purpose!

5

I have 3 installs as well, and ssh works fine on all of them.

6

Is there a way for me to run a script that would restart sshd after vyos completes booting? Restarting sshd is what fixes the issue.

7

I should say that ssh works fine for me as soon as I log in through the console and make a change that causes the ssh daemon to restart. Then I can ssh in but not before.

Again, is there a way to run a script that would cause sshd to restart after vyos boots up all the way?

8

I understand what you are saying, because you have said it so many times.

http://vyos.net/wiki/Command_scripting

This is a dirty work around. This doesn’t fix anything

Please post your config because you have a configuration messed up somewhere.

9

Any hints in /var/log/messages?

10

interfaces {
bonding bond0 {
hash-policy layer3+4
mode 802.3ad
mtu 9000
vif 10 {
address 100.64.10.253/24
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address 100.64.10.1
}
}
}
vif 16 {
address 100.64.16.30/27
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address 100.64.16.1
}
}
}
vif 20 {
address 100.64.20.62/26
description san
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address 100.64.20.1
}
}
}
vif 40 {
address 100.64.40.253/24
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address 100.64.40.1
}
}
}
vif 200 {
address 100.64.200.17/24
description IPMI
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address 100.64.200.1
}
}
}
vif 210 {
address 100.64.210.254/24
description management
vrrp {
vrrp-group 1 {
advertise-interval 1
preempt true
virtual-address 100.64.210.1
}
}
}
}
ethernet eth0 {
duplex auto
hw-id a0:36:9f:02:62:38
smp_affinity auto
speed auto
}
ethernet eth1 {
duplex auto
hw-id a0:36:9f:02:62:39
smp_affinity auto
speed auto
}
ethernet eth2 {
bond-group bond0
duplex auto
hw-id a0:36:9f:02:62:3a
mtu 9000
smp_affinity auto
speed auto
}
ethernet eth3 {
bond-group bond0
duplex auto
hw-id a0:36:9f:02:62:3b
mtu 9000
smp_affinity auto
speed auto
}
ethernet eth4 {
duplex auto
hw-id 00:26:9e:58:ec:b2
smp_affinity auto
speed auto
}
ethernet eth5 {
address dhcp
duplex auto
hw-id 00:26:9e:58:ec:b3
smp_affinity auto
speed auto
}
loopback lo {
}
}
nat {
source {
rule 10 {
outbound-interface eth5
source {
address 100.64.0.0/10
}
translation {
address 192.168.16.111
}
}
}
}
service {
ssh {
listen-address 192.168.16.111
listen-address 100.64.10.1
port 22
}
}
system {
config-management {
commit-revisions 20
}
console {
device ttyS0 {
speed 9600
}
}
host-name xxxxxxx
login {
user bruceg {
authentication {
encrypted-password ****************
plaintext-password ****************
}
level admin
}
user vyos {
authentication {
encrypted-password ****************
}
level admin
}
}
ntp {
server 0.pool.ntp.org {
}
server 1.pool.ntp.org {
}
server 2.pool.ntp.org {
}
}
package {
auto-sync 1
repository community {
components main
distribution helium
password ****************
url http://packages.vyos.net/vyos
username “”
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone UTC
}

I just looked and saw this.

Apr 8 15:31:24 xxxxx sshd[3588]: error: Bind to port 22 on 192.168.16.111 failed: Cannot assign requested address.

Apr 8 15:31:24 xxxxx sshd[3588]: error: Bind to port 22 on 100.64.10.1 failed: Cannot assign requested address.

Apr 8 15:31:24 xxxxx sshd[3588]: fatal: Cannot bind any address.

Apr 8 15:31:24 xxxxx kernel: [ 24.720942] sshd (3588): /proc/3588/oom_adj is deprecated, please use /proc/3588/oom_score_adj instead.

Apr 8 16:26:16 xxxxx sshd[4077]: error: Bind to port 22 on 1.1.1.1 failed: Cannot assign requested address.

11

This would be your problem. 192.168.16.111 does not exist in your config, and 100.64.10.1 is a VRRP address, it’s probably not initialized yet when sshd starts.

You might try using a VRRP transition script to restart sshd when the system becomes the active node, or just bind to a valid non-VRRP address.

12

192.168.16.111 was supplied via DHCP and apparently the ssh daemon starts before the interface gets its IP address.

Further, I can assign non-VRRP addresses to the ssh config.

Thanks for your help - appreciate it very much.

1 Like
13

I don’t think it will work properly to bind ssh service to a dhcp address. I don’t really understand your config. your NAT rule doesn’t make any sense to me either (doesn’t mean it is wrong, but I don’t get it).

Why would you have an interface that is an internal one grabbing a DHCP address? Why would you not statically assign that interface an address?

14

OpenSSH daemon will happily bind to anything that exists if you explicitly configure listen addresses, bu the catch is the IPs must exist & configured on an interface. In the absence of a listen-address directive, openssh (and any other daemon that is configured to listen on all IPs) will bind to INADDR_ANY which survives IPs coming and going.

To the OP, you will always have a race condition when booting up and having daemons expecting to bind to an explicit IP, but that IP doesn’t yet exist on an interface. Systemd handles this kind of stuff quite gracefully (compared to init scripts that frequently have random sleeps to work around these kind of behaviours).

For VyOS specifically, the easiest way to deal with this is probably to add some logic to /opt/vyatta/etc/config/scripts/vyatta-postconfig-bootup.script which will survive image upgrades & is invoked from /etc/rc.rclocal. You may need to background a script that sleeps a bit longer and then starts sshd explicitly.

15

Looks like you’re trying to run ssh listening on a vrrp vip. I would remove the listen-address portion of the ssh configuration for starters to see if it works. Then I would listen on one of the local IP addresses on an internal interface. I see no reason to make sshd listen on a vrrp virtual address.

What is probably happening is ssh tries to start before vrrp brings up the virtual interface IP and ssh failed because the IP it’s trying to bind to doesn’t exist yet. After the machine has been up long enough for vrrp to come up, then the IPs are present, so a restart of ssh works and listens properly. if you must listen on a vip for some reason look at sysctl kernel tweak net.ipv4.ip_nonlocal_bind=1
and copy the host keys from this node to it’s vrrp partner.

16

This was set up at my house prior to bringing it down to my data center. I used DHCP to get an address for it. Now that it is at the data center, I have statically assigned an IP address to it.

17

I just had the same issue on a test machine; a Supermicro box. I had the management interface still set to DHCP, and SSH would work when I configured it, but after a reboot the service was no longer available even though the configuration changes were still there.