External traffic stops routing when IPSEC tunnel comes up

General questions
1

Hi all,

Having this really strange issue where my VyOS install will stop routing external traffic once IPSEC tunnel comes up. When the tunnel isnt up, i can ping, ssh etc but when the tunnel comes up I can only access the install via the tunnel. Cannot ping or ssh externally. Traceroute from the install falls flat at first hop, so it seems like it isn’t really trying?

Have tried several different builds of the OS, including today’s latest.

Config is below, I have done a full reinstall and started adding only the required config line by line to see when it breaks and everything is fine until tunnel comes up.

interfaces {
    ethernet eth0 {
        address 172.20.201.254/24
        hw-id 00:16:3e:e1:b9:31
    }
    loopback lo {
    }
    vti vti0 {
        address 172.19.1.5/30
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 172.20.201.1 {
            }
        }
    }
}
service {
    ssh {
        disable-password-authentication
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password ****************
                public-keys key123 {
                    key ****************
                    type ssh-rsa
                }
            }
        }
    }
    ntp {
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
vpn {
    ipsec {
        esp-group ESP {
            compression disable
            lifetime 3600
            mode tunnel
            pfs dh-group2
            proposal 1 {
                encryption aes256
                hash sha1
            }
        }
        ike-group IKE {
            dead-peer-detection {
                action restart
                interval 15
                timeout 30
            }
            ikev2-reauth yes
            key-exchange ikev2
            lifetime 28800
            proposal 1 {
                dh-group 2
                encryption aes256
                hash sha1
            }
        }
        interface eth0
        site-to-site {
            peer x.x.x.116 {
                authentication {
                    id x.x.x.159
                    mode pre-shared-secret
                    pre-shared-secret ****************
                    remote-id x.x.x.116
                }
                connection-type initiate
                ike-group IKE
                ikev2-reauth inherit
                local-address 172.20.201.254
                vti {
                    bind vti0
                    esp-group ESP
                }
            }
        }
    }
}

Routing Table:

S>* 0.0.0.0/0 [1/0] via 172.20.201.1, eth0, weight 1, 00:55:07
C>* 172.19.1.4/30 is directly connected, vti0, 00:01:37
C>* 172.20.201.0/24 is directly connected, eth0, 00:55:09

Any help or hints would be greatly appreciated.

2

Check all routting tables:

show ip route table all
3

Thanks for your reply, don’t know much about VRFs but here is the entire print out

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 254:
S>* 0.0.0.0/0 [1/0] via 172.20.201.1, eth0, weight 1, 00:02:51
C>* 172.19.1.4/30 is directly connected, vti0, 00:02:53
C>* 172.20.201.0/24 is directly connected, eth0, 00:02:54

Looks fine from face value

4

And from remote site?

5

Remote side is an Ubiquiti ERX

default via 119.17.156.1 dev eth0 proto zebra
119.17.156.0/22 dev eth0 proto kernel scope link src 119.17.159.116
172.19.1.4/30 dev vti0 proto kernel scope link src 172.19.1.6
172.20.205.0/24 dev eth1 proto kernel scope link src 172.20.205.254
broadcast 119.17.156.0 dev eth0 table local proto kernel scope link src x.x.x.116
local 119.17.159.116 dev eth0 table local proto kernel scope host src 1x.x.x.116
broadcast 119.17.159.255 dev eth0 table local proto kernel scope link src x.x.x.116
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 172.19.1.4 dev vti0 table local proto kernel scope link src 172.19.1.6
local 172.19.1.6 dev vti0 table local proto kernel scope host src 172.19.1.6
broadcast 172.19.1.7 dev vti0 table local proto kernel scope link src 172.19.1.6
broadcast 172.20.205.0 dev eth1 table local proto kernel scope link src 172.20.205.254
local 172.20.205.254 dev eth1 table local proto kernel scope host src 172.20.205.254
broadcast 172.20.205.255 dev eth1 table local proto kernel scope link src 172.20.205.254

However the VyOS install becomes uncontactable from all external regardless of site or network

6

Hi mistersock, welcome to the VyOS community!

Is this is the table before IPsec comes UP? How it looks after?

7

Hi Nikolay,

Thanks for your reply and introduction to the forums.

Routing table is identical when IPSEC is connected and not connected :slight_smile:

Thanks

8

I tried making a lab one with a similar setup.
It works on 1.2.8 and 1.4, both Internet and IPsec at the same time.
Maybe this helps (IKEv2 example):
https://docs.vyos.io/en/crux/configuration/vpn/site2site_ipsec.html#examples
Here is my example:


interfaces {
    ethernet eth0 {
        address 172.18.202.10/24
    }
    loopback lo {
    }
    vti vti10 {
        address 10.0.0.3/31
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 172.18.202.254 {
            }
        }
    }
}
system {
    host-name R2
}
vpn {
    ipsec {
        esp-group ESP_DEFAULT {
            compression disable
            lifetime 3600
            mode tunnel
            pfs dh-group19
            proposal 10 {
                encryption aes256gcm128
                hash sha256
            }
        }
        ike-group IKEv2_DEFAULT {
            dead-peer-detection {
                action restart
                interval 30
                timeout 120
            }
            ikev2-reauth no
            key-exchange ikev2
            lifetime 10800
            mobike disable
            proposal 10 {
                dh-group 19
                encryption aes256gcm128
                hash sha256
            }
        }
        site-to-site {
            peer 172.18.201.10 {
                authentication {
                    id 172.18.202.10
                    mode pre-shared-secret
                    pre-shared-secret ****************
                    remote-id 172.18.201.10
                }
                connection-type initiate
                ike-group IKEv2_DEFAULT
                ikev2-reauth inherit
                local-address 172.18.202.10
                vti {
                    bind vti10
                    esp-group ESP_DEFAULT
                }
            }
        }
    }
}
9

Hi Nikolay,

Thank you very much for your time you’re putting into this. I performed the config changes exactly how you have it written above only changing the IKE encryption method because aes256gcm128 is not supported for IKE on the Ubiquiti Edgerouter which is the remote party. I’ve changed it to

set vpn ipsec ike-group IKEv2_DEFAULT proposal 10 encryption aes256

As soon as the tunnel comes up all external connections drop again. super strange!

Current config:

interfaces {
    ethernet eth0 {
        address 172.20.201.254/24
        hw-id 00:16:3e:e1:b9:31
    }
    loopback lo {
    }
    vti vti0 {
        address 172.19.1.5/30
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 172.20.201.1 {
            }
        }
    }
}
service {
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name rt1
    login {
        user vyos {
            authentication {
                encrypted-password ****************
                public-keys sshkey {
                    key ****************
                    type ssh-rsa
                }
            }
        }
    }
    ntp {
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
vpn {
    ipsec {
        esp-group ESP_DEFAULT {
            compression disable
            lifetime 3600
            mode tunnel
            pfs dh-group19
            proposal 10 {
                encryption aes256gcm128
                hash sha256
            }
        }
        ike-group IKEv2_DEFAULT {
            dead-peer-detection {
                action restart
                interval 30
                timeout 120
            }
            ikev2-reauth no
            key-exchange ikev2
            lifetime 10800
            mobike disable
            proposal 10 {
                dh-group 19
                encryption aes256
                hash sha256
            }
        }
        site-to-site {
            peer x.x.x.116 {
                authentication {
                    id x.x.x.159
                    mode pre-shared-secret
                    pre-shared-secret ****************
                    remote-id x.x.x.116
                }
                connection-type initiate
                ike-group IKEv2_DEFAULT
                ikev2-reauth inherit
                local-address 172.20.201.254
                vti {
                    bind vti0
                    esp-group ESP_DEFAULT
                }
            }
        }
    }
}

Current Version: 1.4-rolling-202109050613

Sorry I’m not reporting good news :slightly_frowning_face:

10

OK I just found an option to emulate an Intel E1000 NIC on my cloud hosting provider. It’s up and no drop to external connections. I’ll monitor overnight and report back with my findings

11

Hi Mistersock,

Yes, its strange
If changing the adapter type helps, it would be great.
In any case, your answer will help, waiting for the results.
Can you tell the adapter type you were having problems with?

12

Unfortunately it seems to be a fluke as network traffic has stopped again. Have tried setting the default route with the ip of the router plus the interface in case its going out the wrong interface but no change. Can still ping through the tunnel but can’t ping to and from externally, ssh etc. Connections drop as soon as tunnel comes up

I have also configured the machine to have the external IP on the interface just incase NAT was doing something strange, no change.

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 254:
S>* 0.0.0.0/0 [1/0] via 203.x.x.1, eth0, weight 1, 00:06:22
C>* 172.19.1.4/30 is directly connected, vti0, 00:03:48
C>* 203.x.x.0/24 is directly connected, eth0, 00:07:1

Sorry to report the bad news

13

do extra troubleshooting:
use tcpdump, to see if packets arrive and are sent back. And if ARP works.
Add logging to firewall rule, so you’re sure packet hits intended rule
run trceroute from vyos itself to non-working destination. Responses might reveal path chosen

14

same issue! other network can not ping !

15

Hello @crazycen,
As @Dmitry suggested:
did you try to add disable-route-autoinstall option?
set vpn ipsec options disable-route-autoinstall

Can you also provide the config?
show configuration commands | strip-private

And routing tables before and after the IPsec comes up

16

bgp over ipsec.txt (3.8 KB)
@Nikolay
Router A
eth0 10.10.100.41 (uplink,gateway 10.10.100.1)
eth1 172.16.254.1 (ipsec vti 172.16.250.1/32)
eth2 172.16.252.1 (connect to nsx-gateway 172.16.252.2)

Router A
eth0 10.10.200.41 (uplink,gateway 10.10.200.1)
eth1 172.16.254.1 (ipsec vti 172.16.250.2/32)
eth2 172.16.253.1 (connect to nsx-gateway 172.16.253.2)

when ipsec up ,RouterA&B ony eth1 can ping (eth1&vti),eth0&eth2 can not ping gateway !!

i try many times ,Problems can recur !

i try set vpn ipsec options disable-route-autoinstall ,Not working!

Snipaste_2021-09-13_09-34-46
Snipaste_2021-09-13_09-34-462034×767 85.6 KB

17

@crazycen
can you show the output of the commands?:
show configuration commands | strip-private

And
show ip route
(before and after the tunnel is up)

18

@Nikolay

:joy: ipsec up ,i can not ssh to router !

Router01a&Router01b ipsec up route

image
image3256×899 474 KB

when ipsec up ,ipsec port and vti port can ping !
image
image1590×1298 229 KB

Router01a&Router01b ipsec down route
i disable ipsec port

image
image1920×869 110 KB

can not ping direct interface
image
image1502×814 156 KB

if i want to recovery , only to delete vpn ipsec and reboot router !

19

@crazycen

can you paste the output of the commands?:
show configuration commands | strip-private

20

@Nikolay Sorry,when ipsec up , I can’t SSH to the router, I can only screenshot!

1 (1)
1 (1)1199×1551 194 KB

1 (2)
1 (2)1199×1551 197 KB

1 (3)
1 (3)1199×1551 130 KB

1 (4)
1 (4)1199×1551 97 KB

1 (5)
1 (5)1199×1551 199 KB