the dns forwarding cache is very responsive and low latency for about a minute after reboot then it slows down noticeably. As soon as I run reset dns forwarding all the system returns “DNS Forwarding cache reset for all domains!” the cache becomes fast again for about a minute or two. This is odd and I’m not sure where the issue lies. I’ve tested all the different pdns-recursor caching configs including with packet cache on/off and changing cache sizes but nothing seem to keep the cache fast as reset cache all.I’m running nightly.
How many entries does your cache fill up with when it starts to go slow?
Im thinking in case you got some malware in your network trying to enumerate various DNS queries which quickly fill up the cache to an unnatural size?
Also check so there is no swapfile that the dns software is trying to utilize.
1 Like
cache slows down really early about 2000 entries only
Cache entries Max cache entries Cache size
--------------- ------------------- -------------
2369 1000000 700.05 kbytes
Generally speaking I strongly recommend to NOT use features such as DNS etc on your router.
They are handy for cornercases but generally speaking get a proper DNS which is easily setup with lets say Alpine Linux and then using Bind9 or whatever you prefer.
If you still want to run something locally in your router then using something like AdGuard Home as a container is probably a better option (and you get better visiblity).
But having that said this should be looked into.
I assume your box have plenty of unused RAM when this happens?
What about swap utilization?
1 Like
when i say cache slow down i mean its even slower than cache set to 0
output from free -h
total used free shared buff/cache available
Mem: 15Gi 702Mi 14Gi 2.6Mi 423Mi 14Gi
Swap: 0B 0B 0B
I’ll look into bind9
How are you timing cache speed?
I use pdns on my Vyos router all day every day and I’ve never noticed a problem with resolver speed.
But yeah, how are you timing resolution speed?
1 Like
Yeah whats slow for me doesnt necessary mean its slow for you…
So are we talking about 1ms vs 2ms or 1ms vs 25000ms ?
Also if possible try to verify this using wireshark both on LAN and WAN side of your router to figure out whats going on (is it that your cache sends some request towards WAN which it then sits and wait for an answer for 2 seconds before giving up and returning an answer from the cache?).
The output of “free -h” shows that the swap isnt involved and you got plenty of unused memory so we can rule that thing out.
Other things that comes to mind is how is the conntrack table is configured if you got thousands of connections in a short timeframe along with alot uses DNSSEC nowadays which means 53/TCP and not only 53/UDP when it comes to DNS resolving. If you run out of space in the conntrack table that can probably give funny results.
Also could the ISP or whatever upstream provider you got have some kind of “anti-DDoS” going on so like if they see more than 1000 DNS requests in less than 1 min (or so) they start to throttle things for you which gives that your cache that tries to reach whatever servers will get affacted aswell?
Other classics specially if you see a 2 or 4 second delay could be that some PTR-records are missing (common when you want to connect to a SSH-server) or that the cache perhaps is configured to log the FQDN instead of just the IP itself (which would result in that the cache receives the query, then try to do a PTR-lookup of the srcip, wait until fail (2 seconds per request so if you got 2 servers configured the timeout might end up at 2 x 2 seconds aka 4 seconds) then log the IP (since it couldnt figure out the PTR-record) and return reply to the srcip from its cache).
1 Like
I’d dump the local dns forwarder in vyos. replace it with mosdns . Local DNS forwarder is very slow.
1 Like
What’s slow about it??
{19:52}~ ➭ dig @192.168.0.1 slashdot.org | grep time
;; Query time: 10 msec
{19:52}~ ➭ dig @192.168.0.1 slashdot.org | grep time
;; Query time: 0 msec
{19:52}~ ➭ dig @192.168.0.1 bbc.co.uk | grep time
;; Query time: 300 msec
{19:53}~ ➭ dig @192.168.0.1 bbc.co.uk | grep time
;; Query time: 0 msec
It takes 10ms for it to lookup slashdot the first time, after that it’s cached and it takes 0ms.
bbc.co.uk takes 300ms first time, because it’s not in my ISPs cache. After that it caches it, again 0ms.
The 300ms is the correct time, I query my ISPs resolvers and they take ~300ms to look it up, I’m in NZ and has to go over to the US/UK to resolve.
What’s slow here? It’s just as performant as querying my ISPs namesevers, except once it’s cached it, it’s 0ms instead of the 10ms trip time it is to them each time.
This bizzare idea people seem to have that one caching nameserver is “so slow” compared to another. Even if one was 10ms slower than the other, are you really going to tell me you/your users are going to notice 10ms on each lookup?
I keep reading on multiple various forums people saying “X resolver is slow” but they never back it up with any evidence. There was a thread here that discussed it too but it never went anywhere, just someone “feeling it was slow”
I’m happy to be shown I’m wrong, by the way. But what’s slow about powerdns? How is mosdns that much faster?
2 Likes
pdns-recursor resolves fast, there is no issue with resolving speed. My issue is the cache slows down and I have to run reset dns forwarding all to see fast cache response. After 5 mins the cache becomes less responsive than set service dns forwarding cache-size 0.
as a work around I use cache-size 200, yes only 200. anything higher seems to cause slower response on my system, coffeelake quad core 2x8gb. I’m still trying to figure out why.
Would be nice if it was possible to have a script to trigger this issue on demand.
Like requesting the same records and see at which point it start to slow down and if restarted it would slow down at the same place in the script (like after unique request 300 it starts to slow down) then me and tjh and others could try to replicate your findings to find out if there is something local to your installation (or place in network and/or ISP) or if there really is an issue with whatever the dns forwarding cache is doing in VyOS?
Also what do you mean by slowdown?
Like from 10ms to 20ms or from 10ms to 25000ms?
1 Like
@Thisistheoldplan
Can you please show us how you’re measuring it being slow? You keep saying it’s slow but don’t provide anything to indicate it.
I don’t doubt you’re experiencing an issue but you’ve not shown anything that supports your slowness issue.
Please provide some dig captures showing resolution speed, or something, that shows the problem both when you’ve got cache set to 200 and then above.
Also have a read of the PowerDNS Recursor Turning Guide and see if anything jumps out.
1 Like
Currently dns forwarder didn’t support base much domain name to send to different DNS server. If set mosdns as the upstream DNS server for local DNS forwarder in vyos, when you access the website. you need to accessthe website twice . the first try will always get DNS timeout.
1 Like
Sorry echowings I don’t follow what you’re saying, can you provide an example?
You originally said that PowerDNS recursor is very slow, now you seem to be discussing something different?
It’s the same thing. local dns forwarder is powerdns recursor.
Clinet—>local DNS forwarder(PowerDNS recursor)----->MosDNS------>Public DNS, The client easy error prone to get DNS request timeout
I’m really sorry, how is this at all related to your statement before that PowerDNS is slow? This sounds like MosDNS is having issues.
At stage I think we have to dismiss this statement as being totally invalid.
Currently I directly use MosDNS as the DNS server. it works well.
1 Like
+15 replies and still @echowing nor @Thisistheoldplan can answer on the simple question as in what are the horrific latencies you see when using the builtin dns forwarding in VyOS?
1 Like
Do you understand how annoying you are being? You said “The PowerDNS resolver is slow” and then have just wobbled all over the place like a drunken sailor, totally ignoring this statement you’ve made or providing any evidence about it. I could not care less about whatever stupid resolver you use. Route your DNS to a resolver that’s hosted on the moon for all I care.
Can you either:
A) Provide some evidence/facts to back up your “PowerDNS is slow” statement or
B) Admit you were speaking about something you have no actual idea about and just throwing random statements out there with zero basis in fact.
Thank you.
1 Like