Serious connection issues - please help

,

Hello,

I have tried switching from OPNsense to VyOS for a second time and again, something fails.
First time I tried, it first worked for a few days and suddenly I had issues getting DHCP replies and some devices stopped communication in the network. My only solution was to return to OPNsense since I didn’t get it running smoothly.

This time, my config is close to the first and I can’t get it working at all. I can ping and resolve everything from the VyOS device itself, but clients have serious issues and I don’t quite understand why.

Ping works from VyOS Device to anywhere
Ping to almost all websites do not work. Some do.
Ping to Fritzbox (WAN Router) works.

My setup is like this:
Clients → managed but not configured switch → VyOS (DHCP: 192.168.11.0/24) on ETH0 → Fritzbox (192.168.10.10) on ETH1 → WWW
No VLANs
Version: 1.5-rolling-202412060007

When i did look for arp cache, almost all outside addresses showed “FAILED” or “UNREACHABLE”

container {
    name AdGuard {
        allow-host-networks
        disable
        image "adguard/adguardhome:latest"
        volume work {
            destination "/opt/adguardhome/conf"
            source "/config/containers/adguard"
        }
    }
}
firewall {
    global-options {
        all-ping "enable"
        ip-src-route "enable"
        log-martians "enable"
        syn-cookies "enable"
        twa-hazards-protection "disable"
    }
    ipv4 {
        name IOT_to_LAN {
            default-action "drop"
            default-log
            rule 10 {
                action "accept"
                description "allow return traffic from LAN"
                state "related"
                state "established"
            }
            rule 11 {
                action "drop"
                description "drop invalid packets"
                state "invalid"
            }
        }
        name IOT_to_LOCAL {
            rule 200 {
                action "accept"
                description "allow DNS"
                destination {
                    port "53"
                }
                protocol "tcp_udp"
            }
        }
        name IOT_to_MGMT {
            default-action "drop"
        }
        name IOT_to_WAN {
            default-action "drop"
            default-log
            rule 300 {
                action "accept"
                description "allow HTTPS to Internet from IOT"
                destination {
                    port "443"
                }
                protocol "tcp_udp"
            }
        }
        name LAN_to_IOT {
            default-action "accept"
        }
        name LAN_to_LOCAL {
            default-action "accept"
            default-log
            rule 50 {
                action "accept"
                description "allow DHCP"
                destination {
                    port "67-68"
                }
                log
                protocol "udp"
                source {
                    port "67-68"
                }
            }
            rule 100 {
                action "accept"
                description "allow ping"
                icmp {
                    type-name "echo-request"
                }
                log
            }
            rule 200 {
                action "accept"
                description "allow DNS"
                destination {
                    port "53"
                }
                log
                protocol "tcp_udp"
            }
            rule 800 {
                action "accept"
                description "allow SSH from LAN"
                destination {
                    port "22"
                }
                log
                protocol "tcp"
            }
            rule 1000 {
                action "accept"
                description "allow traffic to Internet"
                destination {
                    address-mask "0.0.0.0"
                }
                log
            }
        }
        name LAN_to_MGMT {
            default-action "drop"
        }
        name LAN_to_WAN {
            default-action "accept"
            default-log
            rule 1000 {
                action "accept"
                description "allow all from LAN to INTERNET"
                log
            }
        }
        name LOCAL_to_IOT {
            default-action "accept"
        }
        name LOCAL_to_LAN {
            default-action "accept"
            default-log
        }
        name LOCAL_to_MGMT {
            default-action "accept"
        }
        name LOCAL_to_WAN {
            default-action "accept"
            default-log
        }
        name MGMT_to_IOT {
            default-action "accept"
        }
        name MGMT_to_LAN {
            default-action "accept"
        }
        name MGMT_to_LOCAL {
            default-action "accept"
            default-log
        }
        name MGMT_to_WAN {
            default-action "accept"
        }
        name WAN_to_IOT {
            rule 10 {
                action "accept"
                description "allow return traffic from Internet to IOT"
                state "established"
                state "related"
            }
            rule 11 {
                action "drop"
                description "drop invalid"
                state "invalid"
            }
        }
        name WAN_to_LAN {
            default-action "accept"
            default-log
            rule 10 {
                action "accept"
                description "allow return traffic from Internet to LAN"
                log
                state "established"
                state "related"
            }
            rule 11 {
                action "drop"
                description "drop invalid"
                log
                state "invalid"
            }
        }
        name WAN_to_LOCAL {
            default-action "accept"
            default-log
            rule 10 {
                action "accept"
                description "allow established/related traffic"
                log
                state "established"
                state "related"
            }
            rule 11 {
                action "drop"
                description "drop traffic with invalid packets"
                log
                state "invalid"
            }
        }
        name WAN_to_MGMT {
            default-action "drop"
        }
    }
    zone IOT {
        default-action "drop"
        default-log
        from LAN {
            firewall {
                name "LAN_to_IOT"
            }
        }
        from LOCAL {
            firewall {
                name "LOCAL_to_IOT"
            }
        }
        from MGMT {
            firewall {
                name "MGMT_to_IOT"
            }
        }
        from WAN {
            firewall {
                name "WAN_to_IOT"
            }
        }
        interface "eth0.20"
    }
    zone LAN {
        default-log
        from IOT {
            firewall {
                name "IOT_to_LAN"
            }
        }
        from LOCAL {
            firewall {
                name "LOCAL_to_LAN"
            }
        }
        from MGMT {
            firewall {
                name "MGMT_to_LAN"
            }
        }
        from WAN {
            firewall {
                name "WAN_to_LAN"
            }
        }
        interface "eth0"
    }
    zone LOCAL {
        from IOT {
            firewall {
                name "IOT_to_LOCAL"
            }
        }
        from LAN {
            firewall {
                name "LAN_to_LOCAL"
            }
        }
        from MGMT {
            firewall {
                name "MGMT_to_LOCAL"
            }
        }
        from WAN {
            firewall {
                name "WAN_to_LOCAL"
            }
        }
        local-zone
    }
    zone MGMT {
        description "Allow Access to Firewall from SECURE PORT ETH2 (ETH3 on DEVICE)"
        from IOT {
            firewall {
                name "IOT_to_MGMT"
            }
        }
        from LAN {
            firewall {
                name "LAN_to_MGMT"
            }
        }
        from LOCAL {
            firewall {
                name "LOCAL_to_MGMT"
            }
        }
        from WAN {
            firewall {
                name "WAN_to_MGMT"
            }
        }
        interface "eth2"
    }
    zone WAN {
        default-action "drop"
        default-log
        from IOT {
            firewall {
                name "IOT_to_WAN"
            }
        }
        from LAN {
            firewall {
                name "LAN_to_WAN"
            }
        }
        from LOCAL {
            firewall {
                name "LOCAL_to_WAN"
            }
        }
        from MGMT {
            firewall {
                name "MGMT_to_WAN"
            }
        }
        interface "eth1"
    }
}
interfaces {
    ethernet eth0 {
        address "192.168.11.1/24"
        description "LAN"
        hw-id "bc:24:11:7a:16:ee"
        offload {
            gro
            gso
            sg
            tso
        }
        
    }
    ethernet eth1 {
        address "192.168.10.2/24"
        description "WAN"
        hw-id "bc:24:11:64:71:fb"
        offload {
            gro
            gso
            sg
            tso
        }
    }
    ethernet eth2 {
        address "172.100.200.1/24"
        description "MGMT"
        hw-id "bc:24:11:1c:39:b8"
    }
    loopback lo {
    }
}
nat {
    source {
        rule 11 {
            outbound-interface {
                name "eth1"
            }
            protocol "all"
            source {
                address "192.168.11.0/24"
            }
            translation {
                address "masquerade"
            }
        }
        rule 20 {
            description "IOT"
            outbound-interface {
                name "eth1"
            }
            protocol "all"
            source {
                address "192.168.20.0/24"
            }
            translation {
                address "masquerade"
            }
        }
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            interface eth1 {
            }
            next-hop 192.168.10.10 {
            }
        }
    }
}
service {
    dhcp-server {
        dynamic-dns-update
        hostfile-update
        shared-network-name berts.local {
            authoritative
            option {
            }
            subnet 192.168.11.0/24 {
                option {
                    default-router "192.168.11.1"
                    name-server "192.168.10.10"
                }
                range clients {
                    start "192.168.11.120"
                    stop "192.168.11.199"
                }
                subnet-id "1"
            }
        }
    }
    dns {
    }
    lldp {
        interface all {
        }
        interface eth0 {
        }
        interface eth2 {
        }
        management-address "192.168.11.1"
    }
    ntp {
        allow-client {
            address "127.0.0.0/8"
            address "169.254.0.0/16"
            address "10.0.0.0/8"
            address "172.16.0.0/12"
            address "192.168.0.0/16"
            address "::1/128"
            address "fe80::/10"
            address "fc00::/7"
        }
        server 192.53.103.108 {
        }
    }
    ssh {
        port "22"
    }
}
system {
    config-management {
        commit-revisions "100"
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sqlnet
        }
    }
    console {
        device ttyS0 {
            speed "115200"
        }
    }
    host-name "VyOSFW"
    login {
        user xxxx {
            authentication {
                encrypted-password "*"
            }
        }
    }
    name-server "192.168.10.10"
    option {
        keyboard-layout "de"
        ssh-client {
            source-interface "eth0"
        }
        time-format "24-hour"
    }
    syslog {
        global {
            facility all {
                level "info"
            }
            facility local7 {
                level "debug"
            }
        }
    }
    time-zone "Europe/Berlin"
}

// Warning: Do not remove the following line.
// vyos-config-version: "bgp@5:broadcast-relay@1:cluster@2:config-management@1:conntrack@5:conntrack-sync@2:container@2:dhcp-relay@2:dhcp-server@11:dhcpv6-server@6:dns-dynamic@4:dns-forwarding@4:firewall@17:flow-accounting@1:https@7:ids@1:interfaces@33:ipoe-server@4:ipsec@13:isis@3:l2tp@9:lldp@2:mdns@1:monitoring@1:nat@8:nat66@3:ntp@3:openconnect@3:openvpn@4:ospf@2:pim@1:policy@8:pppoe-server@11:pptp@5:qos@3:quagga@11:reverse-proxy@2:rip@1:rpki@2:salt@1:snmp@3:ssh@2:sstp@6:system@28:vrf@3:vrrp@4:vyos-accel-ppp@2:wanloadbalance@3:webproxy@2"
// Release version: 1.5-rolling-202412060007

LAN_to_LOCAL rule 10000 makes no sense, but this isn’t related.
Outside addresses shouldn’t end up in ARP cache, only next hop 192.168.10.10 should be.
So I’d focus on reachability to Fritzbox when connection fails.

But as I said, the device itself has no issues pinging any address. Its only related to devices over dhcp hence I do not know what i am missing or misconfigured.

Any idea how, if it is not my config, i could find out what the malfunction is?

If, as you mean, it is the Connection to the FRITZ!Box, then the VyOS Host should itself has Problems…

I have tried to do a tcpdump during a request on heise.de

this is the log:

tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:10:00.669884 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
    berts-pc.52527 > fritz.box.domain: [udp sum ok] 37066+ A? heise.de. (26)
    fritz.box.domain > berts-pc.52527: [udp sum ok] 37066 q: A? heise.de. 1/0/0 heise.de. A 193.99.144.80 (42)
    fritz.box.domain > 192.168.10.1.45418: [udp sum ok] 55521 q: PTR? 80.144.99.193.in-addr.arpa. 1/0/0 80.144.99.193.in-addr.arpa. PTR redirector.heise.de. (77)
08:10:01.726610 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:02.750536 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:03.923115 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:04.926609 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:05.950539 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:07.670972 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:08.702540 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:09.726536 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:15.672168 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:16.702588 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28
08:10:17.726676 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has redirector.heise.de tell 192.168.10.1, length 28

The VyOS seems to get no answer from the FritzBox , what is the reason for this? Any ideas?

Delete firewall section and check connectivity without

No difference. Just done. Still loading for ever

Is it normal, that my Desktop Client is shown with MAC and IP from the vyos System?!

fritz.box.domain > 192.168.10.1.45418: [udp sum ok] 55521 q: PTR? 80.144.99.193.in-addr.arpa. 1/0/0 80.144.99.193.in-addr.arpa. PTR redirector.heise.de. (77)

This line looks like it got a reply from the fritz.box with DNS resolution. So looks like NAT isn’t getting return traffic back as it keeps asking who has redirector.heise.de

if you delete firewall you say it still does this ?

If you’re seeing arp requests for off link IPs, I think the default route may actually be the problem. Linux does very strange things when it binds to an interface rather than IP.

If you delete the reference to eth1 under the default route, and either leave no interface ref at all or move it to be a property of the next hop so it looks like this instead:

protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.10.10 {
                interface eth1
            }
        }
    }

does it change the behavior so you no longer send arp requests for off link IPs?

1 Like

Indeed ,leave out eth interface.
Route should be something like:
set protocols static route 0.0.0.0/0 next-hop 192.168.10.10 distance ‘1’

I guess that solved it!
Thanks and whi would Imagine that something like this bricks everything…

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.