Routing between two IPSec Site-To-Site Tunnels, traffic being routed to WAN

I have a VyOS instance installed as a VM in ProxMox.

It has 1 x Interface attached, eth0, which is sat on our LAN, with a static IP.

It’s next hop for 0.0.0.0 is set to our current router (Unifi USG), and the device can reach the internet and local network fine, similarly we can access it fine from the LAN.

We have 2 x IPSec Site-To-Site tunnels configured, as follow:

1 - AWS, this is setup using BGP and is advertising two routes, the LAN subnet (10.11.12.0/23), and the second VPN tunnels range (10.0.100.0/24), this works fine and is propagated into AWS as expected.

2 - Oracle, this is setup using static routing, static routes are defined on the other side of the tunnel for the LAN Subnet 1(0.11.12.0/23) and the AWS Subnet (10.12.0.0/16).

From the LAN I can ping an access devices on both tunnels, from AWS I can access devices on the LAN. I am unable to test directly from Oracle to LAN as I don’t have access to that site, but I can access it from the LAN so it seems to be working fine.

However what I also need to be able to do it access the Oracle site from the AWS site, and that’s where I’m failing.

I can see the Static Route for the oracle 10.0.100.0/24 network being propagated into the AWS VPC as expected.

But if I attempt a traceroute from a machine on AWS I can see the the traffic is routed over the tunnel as expected, but is then passed from the eth0 of the VyOS system to our internet gateway device.

I do have static routes setup on the gateway for both remote networks pointing to the VyOS instance, and this works internally, but in this context causes an infinite loop as the gateway routes the packet back to VyOS, and it then sends it back to the gateway.

So, I’m obviously missing a step here to tell VyOS to not send the traffic arriving from the AWS address range, with a destination of the Oracle network, to the LAN and instead route it directly to the VPN Tunnel, but I’m not sure how to achieve this, as their isn’t a VTI interface here to route to as it’s a static route based tunnel.

So, if anyone could offer any suggestions that would be amazing, save me having to just setup a second instance and have a tunnel on each :face_vomiting:

Here’s the current configuration, but some bits redacted!

interfaces {
    ethernet eth0 {
        address 10.11.12.4/23
        description DMZ
    }
    loopback lo {
    }
    vti vti2 {
        address 169.#.#.#/30
        description "AWS 1"
        mtu 1436
    }
    vti vti3 {
        address 169.#.#.#/30
        description “AWS 2“
        mtu 1436
    }
}
nat {
    source {
        rule 100 {
            outbound-interface eth0
            source {
                address 10.11.12.0/23
            }
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    bgp 65026 {
        address-family {
            ipv4-unicast {
                network 10.0.100.0/24 {
                }
                network 10.11.12.0/23 {
                }
            }
        }
        neighbor 169.#.#.# {
            address-family {
                ipv4-unicast {
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            remote-as 64512
            timers {
                holdtime 30
                keepalive 10
            }
        }
        neighbor 169.#.#.# {
            address-family {
                ipv4-unicast {
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            remote-as 64512
            timers {
                holdtime 30
                keepalive 10
            }
        }
    }
    static {
        route 0.0.0.0/0 {
            next-hop 10.11.12.1 {
            }
        }
    }
}
service {
    dns {
    }
    ssh {
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name vyos
    login {
        user # {
            authentication {
                encrypted-password ****************
                plaintext-password ****************
            }
            level admin
        }
    }
    name-server 10.11.12.1
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}
vpn {
    ipsec {
        esp-group ESP2-AES256-SHA1-LT3600-DH5 {
            lifetime 3600
            mode tunnel
            pfs dh-group5
            proposal 1 {
                encryption aes256
                hash sha1
            }
        }
        esp-group ESP-AES256-SHA1-LT3600 {
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes256
                hash sha1
            }
        }
        ike-group IKE-AES256-SHA1-LT28800 {
            dead-peer-detection {
                action restart
                interval 15
                timeout 30
            }
            lifetime 28800
            proposal 1 {
                dh-group 2
                encryption aes128
                hash sha1
            }
        }
        ipsec-interfaces {
            interface eth0
        }
        nat-traversal enable
        site-to-site {
            peer 3.#.#.# {
                authentication {
                    mode pre-shared-secret
                    pre-shared-secret ****************
                }
                connection-type initiate
                description "AWS Tunnel 1"
                ike-group IKE-AES256-SHA1-LT28800
                local-address 10.11.12.4
                vti {
                    bind vti2
                    esp-group ESP-AES256-SHA1-LT3600
                }
            }
            peer 35.#.#.# {
                authentication {
                    mode pre-shared-secret
                    pre-shared-secret ****************
                }
                connection-type initiate
                description "AWS Tunnel 2"
                ike-group IKE-AES256-SHA1-LT28800
                local-address 10.11.12.4
                vti {
                    bind vti3
                    esp-group ESP-AES256-SHA1-LT3600
                }
            }
            peer 132.#.#.# {
                authentication {
                    id 88.98.164.150
                    mode pre-shared-secret
                    pre-shared-secret ****************
                }
			    connection-type initiate
                default-esp-group ESP2-AES256-SHA1-LT3600-DH5
                description "OCI 2"
                ike-group IKE-AES256-SHA1-LT28800
                local-address 10.11.12.4
                tunnel 1 {
                    local {
                        prefix 10.11.12.0/23
                    }
                    remote {
                        prefix 10.0.100.0/24
                    }
                }
            }
            peer 132.#.#.# {
                authentication {
                    id 88.98.164.150
                    mode pre-shared-secret
                    pre-shared-secret ****************
                }
                connection-type initiate
                default-esp-group ESP2-AES256-SHA1-LT3600-DH5
                description "OCI 1"
                ike-group IKE-AES256-SHA1-LT28800
                local-address 10.11.12.4
                tunnel 1 {
                    local {
                        prefix 10.11.12.0/23
                    }
                    remote {
                        prefix 10.0.100.0/24
                    }
                }
            }
        }
    }
}

Thanks

Gareth