VyOS VyOS 1.4-rolling-202204140217 AWS tunnel BGP configuration issues

Hi

I am trying to setup tunnels with an AWS VPC, and using bgp.
I noticed oddly that BGP responses aren’t being sent through the VTI interfaces (or at all).
I checked with journalctl -x | grep bgp and interestingly I found this:

Apr 14 09:03:06 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.144.29 : Cannot assign requested address
Apr 14 09:03:13 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.177.37 : Cannot assign requested address
Apr 14 09:05:06 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.144.29 : Cannot assign requested address
Apr 14 09:05:13 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.177.37 : Cannot assign requested address
Apr 14 09:07:06 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.144.29 : Cannot assign requested address
Apr 14 09:07:13 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.177.37 : Cannot assign requested address
Apr 14 09:09:06 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.144.29 : Cannot assign requested address
Apr 14 09:09:13 gw00 bgpd[1162]: [VZ06J-9VWXE][EC 100663299] can’t bind socket for 169.254.177.37 : Cannot assign requested address

any reason why bgp will be complaining about this?

My Ipsec config is like this:

set vpn ipsec esp-group to-aws-1 compression ‘disable’
set vpn ipsec esp-group to-aws-1 lifetime ‘3600’
set vpn ipsec esp-group to-aws-1 mode ‘tunnel’
set vpn ipsec esp-group to-aws-1 pfs ‘enable’
set vpn ipsec esp-group to-aws-1 proposal 1 encryption ‘aes128’
set vpn ipsec esp-group to-aws-1 proposal 1 hash ‘sha1’
set vpn ipsec esp-group to-aws-2 compression ‘disable’
set vpn ipsec esp-group to-aws-2 lifetime ‘3600’
set vpn ipsec esp-group to-aws-2 mode ‘tunnel’
set vpn ipsec esp-group to-aws-2 pfs ‘enable’
set vpn ipsec esp-group to-aws-2 proposal 1 encryption ‘aes128’
set vpn ipsec esp-group to-aws-2 proposal 1 hash ‘sha1’
set vpn ipsec ike-group to-aws-1 dead-peer-detection action ‘restart’
set vpn ipsec ike-group to-aws-1 dead-peer-detection interval ‘15’
set vpn ipsec ike-group to-aws-1 dead-peer-detection timeout ‘30’
set vpn ipsec ike-group to-aws-1 lifetime ‘28800’
set vpn ipsec ike-group to-aws-1 proposal 1 dh-group ‘2’
set vpn ipsec ike-group to-aws-1 proposal 1 encryption ‘aes128’
set vpn ipsec ike-group to-aws-1 proposal 1 hash ‘sha1’
set vpn ipsec ike-group to-aws-2 dead-peer-detection action ‘restart’
set vpn ipsec ike-group to-aws-2 dead-peer-detection interval ‘15’
set vpn ipsec ike-group to-aws-2 dead-peer-detection timeout ‘30’
set vpn ipsec ike-group to-aws-2 lifetime ‘28800’
set vpn ipsec ike-group to-aws-2 proposal 1 dh-group ‘2’
set vpn ipsec ike-group to-aws-2 proposal 1 encryption ‘aes128’
set vpn ipsec ike-group to-aws-2 proposal 1 hash ‘sha1’
set vpn ipsec site-to-site peer a.a.a.a authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer a.a.a.a authentication pre-shared-secret ‘’
set vpn ipsec site-to-site peer a.a.a.a description ‘AWS transit VPC tunnel 1’
set vpn ipsec site-to-site peer a.a.a.a ike-group ‘to-aws-1’
set vpn ipsec site-to-site peer a.a.a.a local-address ‘x.x.x.x’
set vpn ipsec site-to-site peer a.a.a.a vti bind ‘vti02’
set vpn ipsec site-to-site peer a.a.a.a vti esp-group ‘to-aws-1’
set vpn ipsec site-to-site peer b.b.b.b authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer b.b.b.b authentication pre-shared-secret ‘’
set vpn ipsec site-to-site peer b.b.b.b description ‘AWS transit VPC tunnel 2’
set vpn ipsec site-to-site peer b.b.b.b ike-group ‘to-aws-2’
set vpn ipsec site-to-site peer b.b.b.b local-address ‘x.x.x.x’
set vpn ipsec site-to-site peer b.b.b.b vti bind ‘vti03’
set vpn ipsec site-to-site peer b.b.b.b vti esp-group ‘to-aws-2’

My bgp configuration is like this:

set protocols bgp address-family ipv4-unicast network 10.100.0.0/16
set protocols bgp local-as ‘65003’
set protocols bgp neighbor 169.254.9.253 address-family ipv4-unicast maximum-prefix ‘50’
set protocols bgp neighbor 169.254.9.253 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor 169.254.9.253 address-family ipv4-unicast prefix-list export ‘AWS_64513-OUT’
set protocols bgp neighbor 169.254.9.253 address-family ipv4-unicast prefix-list import ‘AWS_64513-IN’
set protocols bgp neighbor 169.254.9.253 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp neighbor 169.254.9.253 capability dynamic
set protocols bgp neighbor 169.254.9.253 description ‘BGP AWS VPC 1’
set protocols bgp neighbor 169.254.9.253 disable-connected-check
set protocols bgp neighbor 169.254.9.253 ebgp-multihop ‘2’
set protocols bgp neighbor 169.254.9.253 remote-as ‘64513’
set protocols bgp neighbor 169.254.9.253 timers holdtime ‘30’
set protocols bgp neighbor 169.254.9.253 timers keepalive ‘10’
set protocols bgp neighbor 169.254.9.253 update-source ‘169.254.9.254’
set protocols bgp neighbor 169.254.176.177 address-family ipv4-unicast maximum-prefix ‘50’
set protocols bgp neighbor 169.254.176.177 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor 169.254.176.177 address-family ipv4-unicast prefix-list export ‘AWS_64513-OUT’
set protocols bgp neighbor 169.254.176.177 address-family ipv4-unicast prefix-list import ‘AWS_64513-IN’
set protocols bgp neighbor 169.254.176.177 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp neighbor 169.254.176.177 capability dynamic
set protocols bgp neighbor 169.254.176.177 description ‘BGP AWS VPC 2’
set protocols bgp neighbor 169.254.176.177 disable-connected-check
set protocols bgp neighbor 169.254.176.177 ebgp-multihop ‘2’
set protocols bgp neighbor 169.254.176.177 remote-as ‘64513’
set protocols bgp neighbor 169.254.176.177 timers holdtime ‘30’
set protocols bgp neighbor 169.254.176.177 timers keepalive ‘10’
set protocols bgp neighbor 169.254.176.177 update-source ‘169.254.176.178’
set protocols bgp parameters router-id ‘10.0.2.1’

VyOS just does not reply to bgp requests from AWS.
Please what am I doing wrong?

Hello @sinaowolabi

You can submit the result of the command:

vyos@vyos:~$ show ip route

Hello @RyVolodya
current output is:

S>* 0.0.0.0/0 [1/0] via 81.29.69.193, eth0, weight 1, 10:29:07
C>* 10.100.10.0/23 is directly connected, eth1, 10:29:21
C>* 10.100.12.0/23 is directly connected, eth2, 10:29:21
C>* 10.100.14.0/23 is directly connected, eth3, 10:29:22
C>* 10.100.16.0/23 is directly connected, eth2, 10:29:21
C>* 10.100.18.0/23 is directly connected, eth3, 10:29:22
C>* 10.100.20.0/23 is directly connected, eth5, 10:29:23
C>* 10.100.22.0/23 is directly connected, eth5, 10:29:23
C>* x.x.x.x/27 is directly connected, eth0, 10:29:20
C>* 169.254.95.176/30 is directly connected, vti02, 10:28:46
C>* 169.254.183.248/30 is directly connected, vti03, 10:28:46
C>* 172.16.17.188/30 is directly connected, vti0, 09:00:57
C>* 172.16.18.200/30 is directly connected, vti01, 10:28:46

I have to state I rolled the system image back to 1.3.0-rc6 out of frustration, and recreated the AWS VPN side to use a transit gateway instead of a virtual private gateway. I don’t have the bgp problems seen in the logs anymore, but now the AWS side is unresponsive to bgp requests, pings or anything. The AWS VTI interfaces for AWS are vti02 and vti03.

vti02 169.254.95.178/30 u/u aws transit VPC tunnel 1
vti03 169.254.183.250/30 u/u aws transit VPC tunnel 2

ping 169.254.95.177 count 1
PING 169.254.95.177 (169.254.95.177) 56(84) bytes of data.
From 169.254.95.178 icmp_seq=1 Destination Host Unreachable

— 169.254.95.177 ping statistics —
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

ping 169.254.183.249 count 1
PING 169.254.183.249 (169.254.183.249) 56(84) bytes of data.
From 169.254.183.250 icmp_seq=1 Destination Host Unreachable

— 169.254.183.249 ping statistics —
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

You have different vti addresses and neighbors are not in the same network. There is no route to the neighbors bgp.

You can send your current configuration:
vyos@vyos# run show configuration commands | strip-private
and other information:

vyos@vyos# run show ip route
vyos@vyos# run show ip bgp
vyos@vyos# run show vpn ipsec sa

Thanks.
I attached them, I hope that’s fine.

conf-commands.txt (27.3 KB)
show-ip-bgp.txt (35 Bytes)
show-ip-route.txt (1.7 KB)
show-vpn-ipsec-sa.txt (1.5 KB)

Could you try removing the range from the NAT?
169.254.0.0/24
set nat source rule XX exclude

I did that but it did not seem to have any effect …

set nat source rule 51 destination address ‘169.254.0.0/24’
set nat source rule 51 exclude
set nat source rule 51 outbound-interface ‘eth0’

The VTIs:

vti02 169.254.94.238/30 u/u aws transit VPC tunnel 1
vti03 169.254.236.174/30 u/u aws transit VPC tunnel 2

run ping 169.254.94.237 count 3
PING 169.254.94.237 (169.254.94.237) 56(84) bytes of data.
From 169.254.94.238 icmp_seq=1 Destination Host Unreachable
From 169.254.94.238 icmp_seq=2 Destination Host Unreachable
From 169.254.94.238 icmp_seq=3 Destination Host Unreachable

run ping 169.254.236.173 count 3
PING 169.254.236.173 (169.254.236.173) 56(84) bytes of data.
From 169.254.236.174 icmp_seq=1 Destination Host Unreachable
From 169.254.236.174 icmp_seq=2 Destination Host Unreachable

run show ip route
S>* 0.0.0.0/0 [1/0] via x.x.x.193, eth0, weight 1, 15:53:21
C>* 10.100.10.0/23 is directly connected, eth1, 15:54:03
C>* 10.100.12.0/23 is directly connected, eth2, 15:54:04
C>* 10.100.14.0/23 is directly connected, eth3, 15:54:00
C>* 10.100.16.0/23 is directly connected, eth2, 15:54:04
C>* 10.100.18.0/23 is directly connected, eth3, 15:54:00
C>* 10.100.20.0/23 is directly connected, eth5, 15:54:02
C>* 10.100.22.0/23 is directly connected, eth5, 15:54:02
S>* 10.101.0.0/16 [1/0] is directly connected, vti0, weight 1, 00:00:41
S>* 10.103.0.0/16 [1/0] is directly connected, vti02, weight 1, 00:06:11
C>* x.x.x.192/27 is directly connected, eth0, 15:54:01
C>* 169.254.94.236/30 is directly connected, vti02, 15:41:20
C>* 169.254.236.172/30 is directly connected, vti03, 15:41:21

run show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.0.2.1, local AS number 65003 vrf-id 0
BGP table version 1
RIB entries 1, using 184 bytes of memory
Peers 2, using 1447 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
169.254.94.237 4 64512 0 0 0 0 0 never Active 0 BGP to-AWS 1
169.254.236.173 4 64512 0 0 0 0 0 never Active 0 BGP to-AWS 2

Total number of neighbors 2

You need your VTIs to work first in order to BGP to connect. Are your SA’s in order?

Figure out why your VTIs dont work. If it’s the case, do a reboot.

Otherwise the config looks alright.

Its really strange, config is all right but it never works unless I first

  1. Create AWS site-to-site connection with transit gateway to desired VPC, BGP fails to work
  2. create AWS site-to-site connection with virtual private gateway with desired VPC, BGP fails to work
  3. Create a new VPC, setup transit gateway, and attachments to site-to-site vpn connection, and desired VPC and new VPC (previous configurations left in place), THEN BGP starts to work.
  4. Delete previous connections.

This seems to happen every time. Im really stumped.
Ive created multiple site-to-sites using the transit gateway to two VyOS routers now (all version 1.4-rolling-202204140217) now.