VyOS in DNS forwarding mode resolving some hosts but not others

I’m configuring VyOS as a firewall to replace an alternate firewall/switching equipment I’m currently using.

The the VyOS machine and client computer (a macmini) is configured as per the image below:

I previously had a fully working configuration with VyOS plugged into my existing network via the 24-port switch. I’ve now connected it directly to internet and it is partially working for DNS resolution.

I can connect to some sites such as google.com or vyos.io and the page loads are instantaneous. But for other sites such as “forum.vyos.io the browser just waits to load the page indefinitely and eventually fails.

I’ve tried simplifying my configuration, looking at log files and at help online but haven’t found any smoking gun to point out what is wrong so wondering if the community might have insight.

A summary of what has been changed or tried to fix the above:

  1. Added pppoe authentication info to configuration.
  2. Configured DNS forwarding, initially through a container running Adguardhome and had the same issue as above. Thought there might be an issue with container networking so I simplified to DNS forwarding through PowerDNS and removing the container altogether.
  3. Added firewall rules per the blog post on blog.kroy.io. Note: I’ve already tried disabling the firewall and running “naked” and it didn’t fix the problem with DNS above. I don’t believe the issue to be firewall related.
  4. I only have pdns listening on port 53 for tcp/udp

Netstat output:

xxxxxxx@vyos:~$ sudo netstat -ntulp | grep 53
tcp        0      0 10.0.255.254:53         0.0.0.0:*               LISTEN      2479/pdns_recursor  
tcp        0      0 127.0.0.1:2612          0.0.0.0:*               LISTEN      1533/ldpd           
udp        0      0 10.0.255.254:53         0.0.0.0:*                           2479/pdns_recursor  
xxxxxxx@vyos:~$

Traceroute output from macmini (it never completes)

xxxxxxx@Macmini ~ % traceroute forum.vyos.io
traceroute: Warning: forum.vyos.io has multiple addresses; using 34.233.30.99
traceroute to vyos-forum-1044462705.us-east-1.elb.amazonaws.com (34.233.30.99), 64 hops max, 52 byte packets
 1  10.0.20.1 (10.0.20.1)  2.653 ms  1.087 ms  1.027 ms
 2  10.0.255.254 (10.0.255.254)  0.624 ms  0.388 ms  0.396 ms
 3  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)  2.034 ms  2.568 ms  2.978 ms
 4  * * *
 5  * * *
 6  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 4.952 ms  4.379 ms  4.175 ms
 7  xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) 3.928 ms  3.033 ms  2.980 ms
 8  * * *
 9  * * *
10  52.94.82.1 (52.94.82.1)  2.573 ms
    52.94.81.237 (52.94.81.237)  12.146 ms
    52.94.81.235 (52.94.81.235)  2.517 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
continues with * * *

VyOS config:

xxxxxx@vyos:~$ show conf
firewall {
    ipv4 {
        name LAN-LOCAL {
            default-action accept
            description "Allow access from LAN to firewall"
        }
        name LAN-WAN {
            default-action accept
            description "Allow access from LAN to WAN"
        }
        name LOCAL-LAN {
            default-action accept
            description "Allow access from firewall to LAN"
        }
        name LOCAL-WAN {
            default-action accept
            description "Allow access from firewall to WAN"
        }
        name WAN-LAN {
            default-action drop
            rule 1 {
                action accept
                description "Allow EST/Related Traffic"
                state established
                state related
            }
            rule 100 {
                action accept
                description "Allow ICMP from WAN to LAN"
                protocol icmp
                state new
            }
        }
        name WAN-LOCAL {
            default-action drop
            rule 1 {
                action accept
                description "Allow EST/Related Traffic"
                state established
                state related
            }
            rule 100 {
                action accept
                description "Allow ICMP from WAN to firewall"
                protocol icmp
                state new
            }
        }
    }
    zone LAN {
        default-action drop
        from LOCAL {
            firewall {
                name LOCAL-LAN
            }
        }
        from WAN {
            firewall {
                name WAN-LAN
            }
        }
        interface eth1
    }
    zone LOCAL {
        default-action drop
        from LAN {
            firewall {
                name LAN-LOCAL
            }
        }
        from WAN {
            firewall {
                name WAN-LOCAL
            }
        }
        local-zone
    }
    zone WAN {
        default-action drop
        from LAN {
            firewall {
                name LAN-WAN
            }
        }
        from LOCAL {
            firewall {
                name LOCAL-WAN
            }
        }
        interface pppoe0
    }
}
interfaces {
    ethernet eth0 {
        description WAN
        hw-id xx:xx:xx:xx:xx:xx
    }
    ethernet eth1 {
        address 10.0.255.254/24
        description LAN
        hw-id xx:xx:xx:xx:xx:xx
    }
    ethernet eth2 {
        hw-id xx:xx:xx:xx:xx:xx
    }
    loopback lo {
    }
    pppoe pppoe0 {
        authentication {
            password ****************
            username ****************
        }
        description xxxxx-xxxxx
        source-interface eth0
    }
}
nat {
    source {
        rule 100 {
            outbound-interface {
                name pppoe0
            }
            source {
                address 10.0.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    static {
        route 10.0.0.0/16 {
            next-hop 10.0.255.1 {
            }
        }
    }
}
service {
    dhcp-server {
        host-decl-name
        shared-network-name LAN {
            authoritative
            subnet 10.0.255.0/24 {
                default-router 10.0.255.1
                range 0 {
                    start 10.0.255.50
                    stop 10.0.255.250
                }
            }
        }
        shared-network-name vl10_mgmt {
            authoritative
            domain-name home.arpa
            subnet 10.0.10.0/24 {
                default-router 10.0.10.1
                range 0 {
                    start 10.0.10.50
                    stop 10.0.10.250
                }
            }
        }
        shared-network-name vl20_main {
            authoritative
            subnet 10.0.20.0/24 {
                default-router 10.0.20.1
                name-server 10.0.255.254
                range 0 {
                    start 10.0.20.50
                    stop 10.0.20.250
                }
            }
        }
        shared-network-name vl99_sec {
            authoritative
            domain-name home.arpa
            subnet 10.0.99.0/24 {
                default-router 10.0.99.1
                range 0 {
                    start 10.0.99.50
                    stop 10.0.99.250
                }
            }
        }
    }
    dns {
        forwarding {
            allow-from 10.0.0.0/16
            listen-address 10.0.255.254
            name-server 9.9.9.9 {
            }
            name-server 149.112.112.112 {
            }
        }
    }
    ntp {
        allow-client {
            address 0.0.0.0/0
            address ::/0
        }
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    ssh {
        listen-address 10.0.255.254
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name xxxxxxxx
    login {
        user xxxxxxxx {
            authentication {
                encrypted-password ****************
            }
        }
    }
    name-server 10.0.255.254
    syslog {
        global {
            facility all {
                level info
            }
            facility local7 {
                level debug
            }
        }
    }
    time-zone US/Eastern
}
}

You say the page loads for ages but then fails.
That doesn’t sound like DNS, that sounds like MTU or similar.

Have you tested that DNS is actually failing?

What does “dig @10.0.255.254 forum.vyos.io” give you?

Your DHCP configuration isn’t giving clients a DNS server to use either - not sure if this by design or not? Unless you’re statically configuring DNS on your clients they won’t be able to resolve DNS at all unless you define a “name-server” statement under your subnet statements:

 dhcp-server {
        shared-network-name EXAMPLE {
            authoritative
            description "EXAMPLE DHCP Server"
            subnet 192.168.0.0/24 {
                default-router 192.168.0.1
                domain-name example.com
                domain-search example.com
                lease 86400
                name-server 192.168.0.1
                ntp-server 192.168.0.1
<snip>

Also: What version of Vyos?

Hi @tjh
I’m on VyOS 1.4 RC1

The name server is configured for vlan 20 (see below), where the macmini is connected, and for LAN (eth1) on the VyOS machine (see end of the config file).

        shared-network-name vl20_main {
            authoritative
            subnet 10.0.20.0/24 {
                default-router 10.0.20.1
                name-server 10.0.255.254
                range 0 {
                    start 10.0.20.50
                    stop 10.0.20.250
                }
            }
        }

I’ll try the dig command and report back.

Sorry, so it is, I scanned the DHCP config and I missed you’d configured it under that vlan.

Here is the output from dig:

xxxxxx@Macmini ~ % dig @10.0.255.254 forum.vyos.io 

; <<>> DiG 9.10.6 <<>> @10.0.255.254 forum.vyos.io
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63315
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;forum.vyos.io.			IN	A

;; ANSWER SECTION:
forum.vyos.io.		300	IN	CNAME	vyos-forum-1044462705.us-east-1.elb.amazonaws.com.
vyos-forum-1044462705.us-east-1.elb.amazonaws.com. 60 IN A 34.233.30.99
vyos-forum-1044462705.us-east-1.elb.amazonaws.com. 60 IN A 52.200.30.47

;; Query time: 25 msec
;; SERVER: 10.0.255.254#53(10.0.255.254)
;; WHEN: Mon Jan 15 17:43:36 EST 2024
;; MSG SIZE  rcvd: 137

xxxxxx@Macmini ~ % dig forum.vyos.io 

; <<>> DiG 9.10.6 <<>> forum.vyos.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61304
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;forum.vyos.io.			IN	A

;; ANSWER SECTION:
forum.vyos.io.		287	IN	CNAME	vyos-forum-1044462705.us-east-1.elb.amazonaws.com.
vyos-forum-1044462705.us-east-1.elb.amazonaws.com. 47 IN A 34.233.30.99
vyos-forum-1044462705.us-east-1.elb.amazonaws.com. 47 IN A 52.200.30.47

;; Query time: 0 msec
;; SERVER: 10.0.255.254#53(10.0.255.254)
;; WHEN: Mon Jan 15 17:43:49 EST 2024
;; MSG SIZE  rcvd: 137

xxxxxx@Macmini ~ % dig +trace forum.vyos.io

; <<>> DiG 9.10.6 <<>> +trace forum.vyos.io
;; global options: +cmd
.			42311	IN	NS	g.root-servers.net.
.			42311	IN	NS	i.root-servers.net.
.			42311	IN	NS	e.root-servers.net.
.			42311	IN	NS	h.root-servers.net.
.			42311	IN	NS	f.root-servers.net.
.			42311	IN	NS	d.root-servers.net.
.			42311	IN	NS	b.root-servers.net.
.			42311	IN	NS	c.root-servers.net.
.			42311	IN	NS	k.root-servers.net.
.			42311	IN	NS	l.root-servers.net.
.			42311	IN	NS	j.root-servers.net.
.			42311	IN	NS	m.root-servers.net.
.			42311	IN	NS	a.root-servers.net.
.			42311	IN	RRSIG	NS 8 0 518400 20240128170000 20240115160000 30903 . Oqy8Fk/8exXTkO6s1RY3s5XueIyKgYCNpHl/gdhuh5WvxNFcjfuWWlKb J2LAsi89J8RIXYbzATzOvhp6NuVTpGS1whWMsLroP2trFXow4pLCsxcJ raYgwhzIeAzy5R1E942zSr0rc2bhdjc5MhOED45UeXCfZINPgHS9ND3x cbJosCOPsHxZtRScg0m/Y3zIqkHooIxIemc9OHqpGRie/RHU1BWAawNv xtO2hR9Q5fa1Hu3dGxgAgMOa2z8bIJ1KBK2/TFjUgwhc/4spOXeZlmN+ P+X5pWXXB51zEGha60mCswvkTf6/Grwu3OORr/injeUPABjFPCZaOVRG uKyvAQ==
;; Received 525 bytes from 10.0.255.254#53(10.0.255.254) in 0 ms

io.			172800	IN	NS	a0.nic.io.
io.			172800	IN	NS	a2.nic.io.
io.			172800	IN	NS	b0.nic.io.
io.			172800	IN	NS	c0.nic.io.
io.			86400	IN	DS	57355 8 2 95A57C3BAB7849DBCDDF7C72ADA71A88146B141110318CA5BE672057 E865C3E2
io.			86400	IN	RRSIG	DS 8 1 86400 20240128170000 20240115160000 30903 . 5JPGFN9HszxUfS0W+RTvdZspnD/TRa8yrRAiogvMH7XLXdGSNerOnbrj xye+yjslknVZIeiFOo7uLEEluIG9olgvYytASIiGbV4owYPdXZhOypWk MpUIab3yz5qgwWylTAVXDr1T1Xx3v2emU12ijCfm/fV7bkIVa4x1+nd7 A29CFQ2WZJoJzJtlcNmsgEXwKuIRFSi7NXfLxyeBGxPVjOCOf0eEZXGF 6BCpRqKfPE7xScix2KPSY/tdWC07oNZCHtLRu1PGmsGo//ulZq/G2w9D PMttydtyA1a6DIWodNx3Pdw9R67OSmjA4YYKvBFpttgebJ8txUx2M2m1 u1YonA==
;; Received 629 bytes from 193.0.14.129#53(k.root-servers.net) in 42 ms

vyos.io.		3600	IN	NS	sofia.ns.cloudflare.com.
vyos.io.		3600	IN	NS	igor.ns.cloudflare.com.
u92tce82j4l1t382opcath2ulsjsm9qg.io. 3600 IN NSEC3 1 1 100 332539EE7F95C32A U92UR6UH0NP6MTNKM7M61MG4FQGG6K45  NS SOA RRSIG DNSKEY NSEC3PARAM
u92tce82j4l1t382opcath2ulsjsm9qg.io. 3600 IN RRSIG NSEC3 8 2 3600 20240205224725 20240115214725 25893 io. OdXdslgiqe+IFdTBhH5gSoNQT/Ed/zB+doqpmXXbNe9b95jttcGSRFs8 m8eIXrrFzwf8JxpqxS7bU18w7bmz997r3lq21KeTjPe+cJbunG0aqFyM Eref0joAvbOfPGdUG7PsYNjUmnlUMjqZGmdYgg1piDNAV/q0H4x8eNZ9 rp8=
4t86moun0qqr99bbaa1p74t7phr3jhgv.io. 3600 IN NSEC3 1 1 100 332539EE7F95C32A 4T976CNRU76TLK4SSBC9A9UNUVBDAQR1  NS DS RRSIG
4t86moun0qqr99bbaa1p74t7phr3jhgv.io. 3600 IN RRSIG NSEC3 8 2 3600 20240201155417 20240111145417 25893 io. cxhpwpGz2cKuK3xExS+N2ANUmxGduoaDc32g4ESoR81q7ziUE3i1pfl9 kQzP2w8jusH6r8F+P/X/ec9y4mQOxUiww5ATFfC6MQkHB0ZTky8ob/95 p9loxTMRhTnpwU42WS2blOI02l+0PovXqM4Dy4hzF5c7w1vDfbK/kY28 A3E=
;; Received 597 bytes from 65.22.162.17#53(c0.nic.io) in 190 ms

forum.vyos.io.		300	IN	CNAME	vyos-forum-1044462705.us-east-1.elb.amazonaws.com.
;; Received 105 bytes from 173.245.58.223#53(sofia.ns.cloudflare.com) in 6 ms

xxxxxx@Macmini ~ % ```

Ok, so test a few more things and make sure they’re also resolving correctly.

But I think we can agree that actually, DNS is just fine and not the problem based on your output above.

I suspect you have a TCP/MSS problem. Try MSS clamping:

set firewall options interface pppoe0 adjust-mss 1452 [That’s the command for 1.3 I think it’ll be the same for 1.4 but I don’t have a 1.4 handy to test quickly]

1 Like

hi @tjh you nailed it, it was a tcp/mss issue.

I’ve fixed it by applying # set interfaces pppoe pppoe0 ip adjust-mss 'clamp-mss-to-pmtu and am able to access forum.vyos.io from the macmini.

Will continue testing various sites to see if I need to adjust MSS down further or not.

Cheers!

Fantastic!
You can try and get one step fancier - it depends if your ISP/PPPoE provider support a full 1500 MTU on their service (RFC4638). My ISP allows a full 1500 MTU on PPPoE as they support the RFC on their Nokia BNGs. You have to set the MTU of the WAN Ethernet port doing PPPoE to at least 1508, then you can set MTU of 1500 on PPPoE and you don’t have to do TCP MSS Clamping:

set interfaces ethernet eth0 mtu '1508'
set interfaces pppoe pppoe0 mtu '1500'

You can apply that and then check what MTU you get with:

show interfaces pppoe pppoe0 | match mtu
pppoe0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN group default qlen 3
1 Like

thanks @tjh, i tried the suggestion to use baby jumbo frames and from what I can tell my ISP supports it but I run into the same problem where I can’t connect to some websites (such as this one forum.vyos.io)

What I tried:

1st try:

set interfaces ethernet eth0 mtu '1508'
set interfaces pppoe pppoe0 mtu '1500'
delete interfaces pppoe pppoe0 ip adjust-mss clamp-mss-to-pmtu

Result:

  • cannot connect to forum.vyos.io but bug ping / dig work
  • removing the mtu values and adding back clamp-mss-to-pmtu fixed things

2nd try: (I read that that MRU should default to MTU per T5630 but tried hardcoding anyway)

set interfaces ethernet eth0 mtu '1508'
set interfaces pppoe pppoe0 mtu '1500'
set interfaces pppoe pppoe0 mru '1500'
delete interfaces pppoe pppoe0 ip adjust-mss clamp-mss-to-pmtu

Result:

3rd try (this time left clamp-mss-to-pmtu enabled to see if that helps)

set interfaces ethernet eth0 mtu '1508'
set interfaces pppoe pppoe0 mtu '1500'
set interfaces pppoe pppoe0 mru '1500'
set interfaces pppoe pppoe0 ip adjust-mss clamp-mss-to-pmtu

Result:

  • No success → same result as with 2nd try
  • Keeping the MRU 1500 setting or deleting it makes no difference.
  • I’ve reverted back to the default ‘1500’/‘1492’ with clamp-mss-to-pmtu to get things working again.

Worth a try, it would seem your ISP doesn’t really support 1500 MTU on PPPoE. Support for it seems very uncommon. [You didn’t include any show commands on the PPPoE interface, but I suspect it probably stayed at 1492]

All good, clamp your MSS and you’ll be fine.

@tjh
I didn’t but a “show interfaces” or “ifconfig” did show the MTU as 1500 for pppoe0 when the MTU for eth0 was set to 1508.

I do have VyOS and the client PC connected through the L3 router/switch so thinking I’ll adjust the MTU size to 1508 and try again in case that’s the issue.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.