Unable to get DNAT to work on pseudo-ethernet

Hi All,

As I prep for a new Kube install, I am adding a pseudo-ethernet interface. I can ping the external interface peth3 without issue from our side of the network, yet I can’t seem to get DNAT to pass. I’m not sure where to look; perhaps you could check my configuration. Sorry, it’s a bit of a mess right now. I’m working on cleaning it up as I learn more. My next firewall change will be to move to zone-based. Anyhow, here’s my configuration: I appreciate any input. I’ve tried for hours without any luck. I’m on Rolling 1.5, latest.

firewall {
    global-options {
        source-validation strict
        state-policy {
            established {
                action accept
            }
            invalid {
                action drop
            }
            related {
                action accept
            }
        }
        twa-hazards-protection enable
    }
    group {
        interface-group LAN {
            interface eth2
        }
        interface-group WAN {
            description "250 Address"
            interface eth3
        }
        interface-group WAN2 {
            description "252 Address"
            interface peth3
        }
        network-group NET-INSIDE-v4 {
            network xxx.xxx.25.0/24
        }
    }
    ipv4 {
        input {
            filter {
                default-action drop
                default-log
                rule 1 {
                    action accept
                    description "Allow all from LAN"
                    inbound-interface {
                        group LAN
                    }
                }
                rule 20 {
                    action jump
                    destination {
                        port 2237
                    }
                    jump-target VyOS_MANAGEMENT
                    protocol tcp
                }
                rule 30 {
                    action accept
                    description "Allow ICMP"
                    limit {
                        rate 5/second
                    }
                    protocol icmp
                }
                rule 40 {
                    action accept
                    destination {
                        port 53
                    }
                    protocol tcp_udp
                    source {
                        group {
                            network-group NET-INSIDE-v4
                        }
                    }
                }
                rule 50 {
                    action accept
                    source {
                        address xxx.xxx.0.0/8
                    }
                }
                rule 60 {
                    action accept
                    destination {
                        address xxx.xxx.25.56
                        port 53764
                    }
                    inbound-interface {
                        group WAN
                    }
                    protocol tcp
                }
                rule 70 {
                    action accept
                    destination {
                        address xxx.xxx.25.76
                        port 443
                    }
                    inbound-interface {
                        group WAN
                    }
                    protocol tcp
                }
                rule 80 {
                    action accept
                    description "NPM 80"
                    destination {
                        address xxx.xxx.25.76
                        port 80
                    }
                    inbound-interface {
                        group WAN
                    }
                    protocol tcp
                }
                rule 90 {
                    action accept
                    description Plex
                    destination {
                        address xxx.xxx.25.51
                        port 32400
                    }
                    inbound-interface {
                        group WAN
                    }
                    protocol tcp
                }
                rule 100 {
                    action accept
                    description Tailscale
                    destination {
                        address xxx.xxx.25.64
                        port 41641
                    }
                    inbound-interface {
                        group WAN
                    }
                    protocol udp
                }
                rule 110 {
                    action accept
                    description "Speedtest Port 3000"
                    destination {
                        address xxx.xxx.25.71
                        port 3002
                    }
                    inbound-interface {
                        group WAN
                    }
                    protocol tcp
                }
                rule 120 {
                    action accept
                    description "Test Extenal to New IP"
                    destination {
                        address xxx.xxx.25.71
                        port 3002
                    }
                    inbound-interface {
                        group WAN2
                    }
                    protocol tcp
                }
            }
        }
        name VyOS_MANAGEMENT {
            default-action return
            rule 15 {
                action accept
                inbound-interface {
                    group LAN
                }
            }
            rule 20 {
                action drop
                inbound-interface {
                    group WAN
                }
            }
        }
    }
}
interfaces {
    ethernet eth0 {
        disable
        hw-id xx:xx:xx:xx:xx:4e
    }
    ethernet eth1 {
        disable
        hw-id xx:xx:xx:xx:xx:4f
    }
    ethernet eth2 {
        address xxx.xxx.25.1/24
        description LAN
        disable-flow-control
        hw-id xx:xx:xx:xx:xx:87
        ip {
            adjust-mss clamp-mss-to-pmtu
            arp-cache-timeout 60
        }
        ipv6 {
            address {
                no-default-link-local
            }
            disable-forwarding
        }
        offload {
            gro
            gso
            rps
        }
        ring-buffer {
            rx 128
            tx 128
        }
    }
    ethernet eth3 {
        address xxx.xxx.0.250/27
        description "250 Address"
        disable-flow-control
        hw-id xx:xx:xx:xx:xx:88
        ip {
            adjust-mss clamp-mss-to-pmtu
            arp-cache-timeout 60
        }
        ipv6 {
            address {
                no-default-link-local
            }
            disable-forwarding
        }
        offload {
            gro
            gso
            rps
        }
        ring-buffer {
            rx 128
            tx 128
        }
    }
    loopback lo {
    }
    pseudo-ethernet peth3 {
        address xxx.xxx.0.252/27
        description "252 WAN"
        ip {
            adjust-mss clamp-mss-to-pmtu
            arp-cache-timeout 60
        }
        ipv6 {
            address {
                no-default-link-local
            }
            disable-forwarding
        }
        source-interface eth3
    }
}
nat {
    destination {
        rule 100 {
            description qBittorrent
            destination {
                address xxx.xxx.0.250
                port 53764
            }
            inbound-interface {
                group WAN
            }
            protocol tcp
            translation {
                address xxx.xxx.25.56
                port 53764
            }
        }
        rule 110 {
            description NPM
            destination {
                address xxx.xxx.0.250
                port 443
            }
            inbound-interface {
                group WAN
            }
            protocol tcp
            translation {
                address xxx.xxx.25.76
                port 443
            }
        }
        rule 120 {
            description "NPM 80"
            destination {
                address xxx.xxx.0.250
                port 443
            }
            inbound-interface {
                group WAN
            }
            protocol tcp
            translation {
                address xxx.xxx.25.76
                port 80
            }
        }
        rule 130 {
            description Plex
            destination {
                address xxx.xxx.0.250
                port 40402
            }
            inbound-interface {
                group WAN
            }
            protocol tcp
            translation {
                address xxx.xxx.25.51
                port 32400
            }
        }
        rule 150 {
            description Tailscale
            destination {
                address xxx.xxx.0.250
                port 41641
            }
            inbound-interface {
                group WAN
            }
            protocol udp
            translation {
                address xxx.xxx.0.250
                port 41641
            }
        }
        rule 160 {
            description "Speedtest Port 3000"
            destination {
                address xxx.xxx.0.250
                port 3000
            }
            inbound-interface {
                group WAN
            }
            log
            protocol tcp
            translation {
                address xxx.xxx.25.71
                port 3002
            }
        }
        rule 170 {
            description "Speedtest Port 3000"
            destination {
                address xxx.xxx.0.252
                port 3000
            }
            inbound-interface {
                group WAN2
            }
            log
            protocol tcp
            translation {
                address xxx.xxx.25.71
                port 3002
            }
        }
    }
    source {
        rule 100 {
            outbound-interface {
                name eth3
            }
            source {
                address xxx.xxx.25.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 110 {
            outbound-interface {
                name peth3
            }
            source {
                address xxx.xxx.25.0/24
            }
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    static {
        route xxx.xxx.0.0/0 {
            next-hop xxx.xxx.0.249 {
            }
        }
    }
}
service {
    dns {
        forwarding {
            allow-from xxx.xxx.25.0/24
            cache-size 0
            domain xxx.xxx.16.172.in-addr.arpa {
                name-server xxx.xxx.25.20 {
                }
                name-server xxx.xxx.25.21 {
                }
            }
            domain cotb.local.lan {
                name-server xxx.xxx.25.20 {
                }
                name-server xxx.xxx.25.21 {
                }
            }
            domain cotb.syberlogic.com {
                name-server xxx.xxx.25.20 {
                }
                name-server xxx.xxx.25.21 {
                }
            }
            exclude-throttle-address xxx.xxx.25.0/24
            listen-address xxx.xxx.25.1
            name-server xxx.xxx.1.1 {
            }
            name-server xxx.xxx.8.8 {
            }
        }
    }
    lldp {
        interface eth3 {
        }
        interface peth3 {
        }
    }
    ntp {
        allow-client xxxxxx
            address xxx.xxx.25.0/24
        }
        server xxxxx.tld {
        }
        server xxxxx.tld {
        }
        server xxxxx.tld {
        }
    }
    ssh {
        port 2237
    }
}
system {
    config-management {
        commit-archive {
            location xxxxxx
            source-address xxx.xxx.25.1
        }
        commit-revisions 1000
    }
    conntrack {
        flow-accounting
        log {
            other {
                destroy
                new
                update
            }
            tcp {
                destroy
                new
                update
            }
            udp {
                destroy
                new
                update
            }
        }
        modules {
            ftp
            h323
            sip
        }
        table-size 1000000
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    domain-name xxxxxx
    domain-search xxxxxx
    host-name xxxxxx
    ip {
        arp {
            table-size 32768
        }
        tcp {
            mss {
                probing on-icmp-black-hole
            }
        }
    }
    login {
        user xxxxxx {
            authentication {
                encrypted-password xxxxxx
            }
        }
    }
    logs {
        logrotate {
            messages {
                max-size 5
            }
        }
    }
    name-server xxx.xxx.25.20
    option {
        kernel {
            disable-mitigations
            disable-power-saving
        }
        performance throughput
        reboot-on-panic
        root-partition-auto-resize
        startup-beep
        time-format 24-hour
    }
    syslog {
        global {
            facility all {
            }
            facility local7 {
            }
        }
    }
    time-zone America/Los_Angeles
    update-check {
        auto-check
        url xxxxxx
    }
}

Thanks again!

Keith

Bumping, this is a hard stop for me. I want to do a zone-based firewall, yet without this capability, I am kind of out of luck.

Thanks again,

Keith

Maybe packets from WAN always enter on eth3, not the peth interface
If so dNAT rules on peth are ineffective.
I always try to avoid peth , its use case is pulling 2nd DHCP address from ISP using different MAC address
why not assign both WAN addresses to eth3?

I believe the root of the issue might indeed be what you’re suggesting, though I’m not entirely certain that it captures the whole picture. It appears that while packets are successfully arriving via PETH3, they seem to be lost when trying to return. My suspicion is that this might be due to internal asymmetric routing affecting LAN clients. Given that their gateway is set to eth3, my guess is that their responses are being sent via eth3 and subsequently dropped. While some firewalls manage this scenario without any issues, I’m not entirely confident about Vyos’s capabilities in this regard.

The reason I’m considering adding a secondary IP is to enable more than one client to handle packets on the same external and internal ports (80/443). I’ve been contemplating whether Vyos’s internal load balancer might offer a solution, though I’m uncertain if it’s the optimal approach in this context. Adding another IP to the primary could be a workaround, but I’m skeptical it would fully address the problem, especially since I’m not too familiar with the nuances of peth interfaces and if they’re causing an issue or if I’m just running up against a routing problem.

I’m planning to experiment with your idea of a secondary IP to see if it offers a straightforward fix, though my expectations are tempered. While employing the Vyos load balancer could introduce greater flexibility, it also presents additional complexity. My goal is to direct traffic into a Kubernetes cluster, and ideally, I’d prefer using an internal cluster load balancer for redundancy. However, considering some of my clients are external to the cluster, this necessitates further consideration.

I’m considering using the load balancer to solve for the initial issue while also implementing policy-based routing to be sure the packets are routed out over the correct interface. This seems rather complex; therefore, I’d appreciate further insight.

Is the Vyos load balancer itself perhaps the solution? Considering all of my traffic is coming from the WAN, I suppose that, being that I have a single firewall, it doesn’t really affect my overall redundancy. I could also consider a load balancer inside the LAN, yet this is one more system to maintain. I’d rather keep it as straight-forward as possible.

I’d appreciate any insights or recommendations you might have. Admittedly, this topic stretches beyond my current networking expertise, and I’m aiming for the most efficient configuration possible.

Thank you immensely for your support!