Understanding VRF and debug tunnels

Khanivore · October 25, 2025, 4:33pm

Hey, since a few month I use VyOS as router. A few days ago I expand my network with a clue. Now I just have external interfaces which are assigned to VRFs. There is no external interface without a VRF to setup tunnels correctly. Also setting up a static route with leak to default VRF does not help to establish the connections of the tunnel.

To get a familiar setup I requested a third interface with a “normal” uplink to just establish tunnel connections which was declined. Now I try to figure out how I should use a loopback interface or dummy interface to bind the tunnel…but without any ethernet port which is not in any VRF.

Can anyone help me to understand how it can be done anyway?

If the one interface which is also used as uplink via BGP is not in a VRF the tunnel works fine, so it seems not be an issue with the tunnel itself. I use ip6gretap for encapsulation of the tunnel.

ip6gretap seems bad for debugging…I just can debug if it works but not why it does not work…or has anyone a hint how to debug ip6gretap tunnel why the connection can not be established?

It is funny that both can reach each others endpoint normally…so it is very hard to understand why the tunnel breaks.

skynw · October 26, 2025, 6:20pm

Hey.

To understand your setup.

Could you share your config and or a drawing ?

From your pure description at least for me it’s difficult to understand what’s your goal and what’s not working.

Cheers

Marcel

Khanivore · October 26, 2025, 6:28pm

firewall {
    ipv6 {
        input {
            filter {
                rule 1 {
                    action accept
                    destination {
                        port 222
                    }
                    protocol tcp
                    source {
                        address xxxx:xxxx:1c1c:7113::/64
                    }
                }
                rule 2 {
                    action accept
                    destination {
                        port 222
                    }
                    protocol tcp
                    source {
                        address xxxx:xxxx:c0c:abb2::/64
                    }
                }
                rule 3 {
                    action accept
                    protocol ipv6-icmp
                }
                rule 5 {
                    action accept
                    state established
                    state related
                }
                rule 10 {
                    action drop
                    destination {
                        port 222
                    }
                    protocol tcp
                }
                rule 11 {
                    action drop
                    state invalid
                }
            }
        }
    }
}
interfaces {
    dummy dum0 {
        address xxxx:xxxx:d41::ce:12/128
        address xxxx:xxxx:60::ce:12/128
        vrf net
    }
    dummy dum1 {
        address xxxx:xxxx:60::cf:12/128
    }
    ethernet eth0 {
        address xxxx:xxxx:2511:101::190c/64
        hw-id xx:xx:xx:xx:xx:4b
        offload {
            gro
            gso
            sg
            tso
        }
        vrf net
    }
    ethernet eth1 {
        address xxxx:xxxx:ca:1::21:3036:1/64
        hw-id xx:xx:xx:xx:xx:11
        offload {
            gro
            gso
            sg
            tso
        }
        vrf net
    }
    loopback lo {
    }
    tunnel tun011 {
        address xxxx:xxxx:d41::8008/127
        description core-w0.draphago.de
        encapsulation ip6gretap
        mtu 1358
        remote xxxx:xxxx:1c1c:f2c:1001:101:1:2
        source-address xxxx:xxxx:2511:101::190c
        vrf net
    }
}
policy {
    as-path-list apl-bogon-asns {
        rule 10 {
            action permit
            regex 23456
        }
        rule 20 {
            action permit
            regex 64496-131071
        }
        rule 30 {
            action permit
            regex 4200000000-4294967295
        }
    }
    large-community-list cm-learnt-downstream {
        rule 1 {
            action permit
            regex "213036:4:*"
        }
    }
    large-community-list lcm-own-communities {
        rule 1 {
            action permit
            regex "213036:*:*"
        }
    }
    prefix-list6 pl6-bogons {
        rule 10 {
            action permit
            le 128
            prefix ::/8
        }
        rule 20 {
            action permit
            le 128
            prefix 100::/64
        }
        rule 30 {
            action permit
            le 128
            prefix xxxx:xxxx::/48
        }
        rule 40 {
            action permit
            le 128
            prefix xxxx:xxxx::/28
        }
        rule 50 {
            action permit
            le 128
            prefix xxxx:xxxx::/32
        }
        rule 60 {
            action permit
            le 128
            prefix 2002::/16
        }
        rule 70 {
            action permit
            le 128
            prefix 3ffe::/16
        }
        rule 80 {
            action permit
            le 128
            prefix fc00::/7
        }
        rule 90 {
            action permit
            le 128
            prefix fe80::/10
        }
        rule 100 {
            action permit
            le 128
            prefix fec0::/10
        }
        rule 110 {
            action permit
            le 128
            prefix ff00::/8
        }
        rule 120 {
            action permit
            le 128
            prefix 3fff::/20
        }
        rule 130 {
            action permit
            le 128
            prefix 5f00::/16
        }
    }
    prefix-list6 pl6-draphago-net-0 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:60::/48
        }
    }
    prefix-list6 pl6-draphago-net-1 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:61::/48
        }
    }
    prefix-list6 pl6-draphago-net-2 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:62::/48
        }
    }
    prefix-list6 pl6-draphago-net-3 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:63::/48
        }
    }
    prefix-list6 pl6-draphago-net-4 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:64::/48
        }
    }
    prefix-list6 pl6-draphago-net-5 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:65::/48
        }
    }
    prefix-list6 pl6-draphago-net-6 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:66::/48
        }
    }
    prefix-list6 pl6-draphago-net-7 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:67::/48
        }
    }
    prefix-list6 pl6-draphago-net-8 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:68::/48
        }
    }
    prefix-list6 pl6-draphago-net-9 {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:69::/48
        }
    }
    prefix-list6 pl6-draphago-net-a {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:6a::/48
        }
    }
    prefix-list6 pl6-draphago-net-b {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:6b::/48
        }
    }
    prefix-list6 pl6-draphago-net-c {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:6c::/48
        }
    }
    prefix-list6 pl6-draphago-net-d {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:6d::/48
        }
    }
    prefix-list6 pl6-draphago-net-e {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:6e::/48
        }
    }
    prefix-list6 pl6-draphago-net-f {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:6f::/48
        }
    }
    prefix-list6 pl6-ixp-lan {
        rule 1 {
            action permit
            ge 64
            le 128
            prefix xxxx:xxxx:701::/64
        }
        rule 2 {
            action permit
            ge 64
            le 128
            prefix xxxx:xxxx:1000:46::/64
        }
        rule 3 {
            action permit
            ge 64
            le 128
            prefix xxxx:xxxx:15b:1::/64
        }
    }
    prefix-list6 pl6-own-prefixes {
        rule 1 {
            action permit
            ge 48
            le 48
            prefix xxxx:xxxx:d41::/48
        }
    }
    prefix-list6 pl6-tiny-prefix {
        rule 10 {
            action permit
            ge 49
            le 128
            prefix ::/0
        }
    }
    route-map rm-noexport {
        rule 1 {
            action deny
        }
    }
    route-map rm-noimport {
        rule 1 {
            action deny
        }
    }
    route-map rm-as34927-in {
        rule 1 {
            action deny
            match {
                as-path apl-bogon-asns
            }
        }
        rule 2 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-bogons
                    }
                }
            }
        }
        rule 3 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-tiny-prefix
                    }
                }
            }
        }
        rule 4 {
            action deny
            match {
                rpki invalid
            }
        }
        rule 5 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-ixp-lan
                    }
                }
            }
        }
        rule 65535 {
            action permit
            set {
                large-community {
                    add 213036:1:34927
                    add 213036:0:0
                }
                local-preference 50
            }
        }
    }
    route-map rm-internal-in {
        rule 1 {
            action permit
        }
    }
    route-map rm-internal-out {
        rule 1 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-own-prefixes
                    }
                }
            }
        }
        rule 2 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-0
                    }
                }
            }
        }
        rule 3 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-1
                    }
                }
            }
        }
        rule 4 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-2
                    }
                }
            }
        }
        rule 5 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-3
                    }
                }
            }
        }
        rule 6 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-4
                    }
                }
            }
        }
        rule 7 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-5
                    }
                }
            }
        }
        rule 8 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-6
                    }
                }
            }
        }
        rule 9 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-7
                    }
                }
            }
        }
        rule 10 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-8
                    }
                }
            }
        }
        rule 11 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-9
                    }
                }
            }
        }
        rule 12 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-a
                    }
                }
            }
        }
        rule 13 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-b
                    }
                }
            }
        }
        rule 14 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-c
                    }
                }
            }
        }
        rule 15 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-d
                    }
                }
            }
        }
        rule 16 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-e
                    }
                }
            }
        }
        rule 17 {
            action deny
            match {
                ipv6 {
                    address {
                        prefix-list pl6-draphago-net-f
                    }
                }
            }
        }
        rule 65535 {
            action permit
        }
    }
    route-map rm-peer-out {
        rule 1 {
            action deny
            match {
                rpki invalid
            }
        }
        rule 2 {
            action permit
            match {
                large-community {
                    large-community-list cm-learnt-downstream
                }
            }
        }
    }
    route-map rm-set-src {
        rule 1 {
            action permit
            set {
                src xxxx:xxxx:d41::ce:12
            }
        }
    }
    route-map rm-tag-downstream {
        rule 1 {
            action permit
            set {
                large-community {
                    add 213036:4:213036
                }
            }
        }
    }
    route-map rm-upstream-out {
        rule 1 {
            action deny
            match {
                rpki invalid
            }
        }
        rule 2 {
            action permit
            match {
                large-community {
                    large-community-list cm-learnt-downstream
                }
            }
        }
    }
}
protocols {
    static {
        route6 xxxx:xxxx:1c1c:f2c:1001:101:1:2/128 {
            next-hop xxxx:xxxx:2511:101::1 {
                vrf net
            }
        }
    }
}
service {
    ntp {
        allow-client xxxxxx
            address xxx.xxx.0.0/8
            address xxx.xxx.0.0/16
            address xxx.xxx.0.0/8
            address xxx.xxx.0.0/12
            address xxx.xxx.0.0/16
            address ::1/128
            address fe80::/10
            address fc00::/7
        }
        server xxxxx.tld {
        }
        server xxxxx.tld {
        }
        server xxxxx.tld {
        }
    }
    ssh {
        access-control {
            allow {
                user xxxxxx
            }
        }
        listen-address xxxx:xxxx:2511:101::190c
        port 222
        vrf net
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name xxxxxx
    login {
        user xxxxxx {
            authentication {
                encrypted-password xxxxxx
                public-keys xxxx@xxx.xxx {
                    key xxxxxx
                    type ssh-ed25519
                }
                public-keys xxxx@xxx.xxx {
                    key xxxxxx
                    type ssh-ed25519
                }
                public-keys xxxx@xxx.xxx {
                    key xxxxxx
                    type ssh-ed25519
                }
            }
        }
        user xxxxxx {
            authentication {
                encrypted-password xxxxxx
                plaintext-password xxxxxx
            }
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility local7 {
                level debug
            }
        }
    }
}
vrf {
    bind-to-all
    name net {
        ipv6 {
            nht {
                no-resolve-via-default
            }
        }
        protocols {
            bgp {
                neighbor xxxx:xxxx:2511:101::1 {
                    address-family {
                        ipv6-unicast {
                            nexthop-self {
                            }
                            route-map {
                                export rm-upstream-out
                                import rm-as34927-in
                            }
                            soft-reconfiguration {
                                inbound
                            }
                        }
                    }
                    description iFog
                    local-role customer {
                    }
                    remote-as XXXXXX
                }
                neighbor xxxx:xxxx:d41::ce:1 {
                    description core-w0.draphago.de
                    peer-group internal
                }
                neighbor xxxx:xxxx:d41::ce:2 {
                    description core-w1.draphago.de
                    peer-group internal
                }
                neighbor xxxx:xxxx:d41::ce:3 {
                    description core-g0.draphago.de
                    peer-group internal
                }
                neighbor xxxx:xxxx:d41::ce:4 {
                    description core-g1.draphago.de
                    peer-group internal
                }
                neighbor xxxx:xxxx:d41::ce:11 {
                    description nbg-gw.draphago.de
                    peer-group internal
                }
                parameters {
                    always-compare-med
                    bestpath {
                        as-path {
                            multipath-relax
                        }
                        med confed
                    }
                    ebgp-requires-policy
                    log-neighbor-changes
                    router-id xxx.xxx.0.2
                }
                peer-group internal {
                    address-family {
                        ipv6-unicast {
                            nexthop-self {
                            }
                            route-map {
                                export rm-internal-out
                                import rm-internal-in
                            }
                            soft-reconfiguration {
                                inbound
                            }
                        }
                    }
                    bfd {
                    }
                    remote-as internal
                }
                system-as 213036
            }
        }
        table 100
    }
}

That is still the complete nonworking configuration.

I have 2 interfaces: eth0 (bgp uplink) and eth1 (IXP) I need a tunnel to my own network over physical eth0…but after I set eth0 into vrf the tunnel breaks. eth0 is required in VRF for BGP…so my description is an configuration independent question to understand in what way it is possible to configure as it seems outside my understanding of VRF. I really do not understand why the tunnel breaks even if both endpoints can reach each other.

skynw · October 26, 2025, 7:30pm

Hey,

I meant set config
But anyway I think I could read it more or less.
to summarize:
tunnel 11 with encapsulation ip6gretap
routing should go out on interface eth0
Eth1 for this use case not relevant

What is in your BGP table for vrf net
What is in your route table for vrf net

I dont see any RD and Route-targets configured for your vrf
IMHO you need them for BGP and vrf

Cheers

Marcel

Khanivore · October 26, 2025, 7:48pm

You mean the summarize of BGP? It just shows that BGP on uplink is configured and all internal connections to core and other gateways are down…

The routing table for vrf net is too large with over 200k prefixes but they are all “valid and best” as there is just one bgp uplink still available. But that does not have anything to do with the general question.

I have already tried much more like set up a second VRF, bind tunnel to loopback interface and also tried to make a bridge between loopback and eth0 but it can not be established because eth0 is in a VRF and VyOS declined it. The current configuration can be non helping anyway as I try everytime some other clues to make it possible. But now I do not have any idea about the two questions below:

WHY does a tunnel break if the physical interface where the tunnel is bind to, is in a VRF?
How can I use a tunnel if all ethernet interfaces are in any vrf? They said they recommend if I create the tunnel through loopback as it is not in a VRF but I still do not understand how I bind the loopback interface to the correct ethernet interface…as they can not be bridged…

Edit: It can not be caused by routing itself as both endpoints can reach each other without any issues. Only the tunnel breaks if the ethernet interface is in a VRF.

L0crian · October 26, 2025, 8:49pm

The tunnel will source its traffic from the default VRF unless a source-interface is defined. So the tunnel is trying to source traffic from a table that doesn’t know about the routes in vrf net.

You would want to define a source-interface (dev in linux) for that traffic, which you’ve probably seen isn’t allowed for encapsulation ip6gretap. Do you need to use that encap, or is ip6gre fine? That will allow you to set the source-interface. If you specifically need the bridging of ip6gretap, then you can use a vxlan interface, which also allows the source-interface to be set.

Like I mentioned above, you can change the encap to ip6gre or use a vxlan interface, which will allow you to set the source-interface. If you have to use ip6gretap, then you have a couple of options:

Create a task on vyos.dev and request relaxing that constraint, but it may have been added for a specific reason.
Remove that constraint yourself in the code, but you’d need to do that every time you upgrade to a new VyOS version.
Use route leaking or a veth pair to route traffic between the default VRF and vrf net.

Khanivore · October 26, 2025, 9:40pm

That solved this issue, the tunnel is able to be used with ip6gre and ip6ip6. Now I can figure out how to bring iBGP up on the tunnel with real debugging.

system · October 28, 2025, 9:41pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.