DNS forward domain disappears

Hi all,

I configured VyOS to forward DNS requests. All requests get forwarded to Freenom (80.80.80.80 & 80.80.81.81) except for requests for one specific domain, those go to an internal server. However, I noticed domain fowarding didn’t go well. Apparently when I apply the configuration or reboot the system the configuration gets applied properly, but after a while the forwarding for the specific domain disappears. The generic forwarding stays there though.

I was running a self-built v1.2.3 and got myself the official 1.2.5, but also on the official v1.2.5 image I got the same problem.

When checking the actual PowerDNS configuration in /etc/powerdns/recursor.conf I noticed that the forward-zones-recurse configuration line changed after ~30 minutes. I’m running two VyOS instances and both have this problem. Removing the DNS service configuration and setting it up again also doesn’t seem to help.

I have the feeling that this is a bug, but if anyone has another solution, please let me know.

The configuration I used:

set service dns forwarding allow-from '10.0.0.0/8'
set service dns forwarding domain mydomain.net server '10.0.0.100'
set service dns forwarding listen-address '0.0.0.0'
set service dns forwarding name-server '80.80.80.80'
set service dns forwarding name-server '80.80.81.81'

The PowerDNS recurser configuration that gets generated on boot/config change:

$ cat /etc/powerdns/recursor.conf 

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=10.0.0.0/8
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=10000

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=yes

# listen-on
local-address=0.0.0.0

# dnssec
dnssec=process-no-validate

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=mydomain.net=10.0.0.100, .=80.80.80.80;80.80.81.81

The changed configuration when the system is running for a while (notice the missing mydomain.net=x.x.x.x

$ cat /etc/powerdns/recursor.conf 

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=10.0.0.0/8
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=10000

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=yes

# listen-on
local-address=0.0.0.0

# dnssec
dnssec=process-no-validate

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=.=80.80.80.80;80.80.81.81

Additional command outputs:

$ w
20:13:19 up 45 min,  1 user,  load average: 0.47, 0.14, 0.05
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
vyos     pts/0    10.0.0.2        19:29    0.00s  0.25s  0.00s w
$ ls -l /etc/powerdns/recursor.conf 
-rw-r--r-- 1 root root 685 Jun 21 19:57 /etc/powerdns/recursor.conf

I did some more digging, and it seems that this behavior is related to dhclient and vyos-hostsd. As you can see in the logs below it starts with dhclient performing a request, followed by vyos-hostsd doing some stuff with the data dhclient received. When vyos-hostsd is finished you see systemd performing a restart of PowerDNS after which my forwarding configuration is gone.

What’s also interesting is that the configured domain and search-domain (in resolv.conf) get overwritten with the one that DHCP offers. But I would assume that statically configured domains should take presedence.

Logs in /var/log/messages:

Jun 24 09:28:04 vyos Keepalived_vrrp[2940]: message repeated 10 times: [ Warning: Failed to connect to the agentx master agent ([NIL]):]
Jun 24 09:28:06 vyos dhclient: DHCPREQUEST on eth4 to 192.168.0.1 port 67
Jun 24 09:28:06 vyos dhclient: DHCPACK from 192.168.0.1
Jun 24 09:28:06 vyos vyos-hostsd[660]: Received a configuration change request
Jun 24 09:28:06 vyos vyos-hostsd[660]: Request data: {"op": "set", "data": {"host_name": null, "domain_name": null, "search_domains": ["home"]}, "type": "host_name"}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/resolv.conf
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/hosts
Jun 24 09:28:06 vyos vyos-hostsd[660]: Saving state to /var/lib/vyos/hostsd.state
Jun 24 09:28:06 vyos vyos-hostsd[660]: Sent response: {'data': None}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Received a configuration change request
Jun 24 09:28:06 vyos vyos-hostsd[660]: Request data: {"type": "name_servers", "tag": "dhcp-eth4", "op": "delete"}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/resolv.conf
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/hosts
Jun 24 09:28:06 vyos vyos-hostsd[660]: Saving state to /var/lib/vyos/hostsd.state
Jun 24 09:28:06 vyos vyos-hostsd[660]: Sent response: {'data': None}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Received a configuration change request
Jun 24 09:28:06 vyos vyos-hostsd[660]: Request data: {"type": "name_servers", "data": ["8.8.8.8", "8.8.4.4"], "tag": "dhcp-eth4", "op": "add"}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/resolv.conf
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/hosts
Jun 24 09:28:06 vyos vyos-hostsd[660]: Saving state to /var/lib/vyos/hostsd.state
Jun 24 09:28:06 vyos vyos-hostsd[660]: Sent response: {'data': None}
Jun 24 09:28:07 vyos systemd[1]: Stopping PowerDNS Recursor...
Jun 24 09:28:07 vyos systemd[1]: Starting PowerDNS Recursor...
Jun 24 09:28:07 vyos pdns_recursor[6897]: PowerDNS Recursor 4.1.15 (C) 2001-2018 PowerDNS.COM BV
Jun 24 09:28:07 vyos pdns_recursor[6897]: Using 64-bits mode. Built using gcc 4.9.2 on Dec  3 2019 13:02:24 by root@613ec8350b02.
Jun 24 09:28:07 vyos pdns_recursor[6897]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Jun 24 09:28:07 vyos pdns_recursor[6897]: Reading random entropy from '/dev/urandom'
Jun 24 09:28:07 vyos pdns_recursor[6897]: Enabling IPv6 transport for outgoing queries
Jun 24 09:28:07 vyos pdns_recursor[6897]: Only allowing queries from: 10.0.0.0/8
Jun 24 09:28:07 vyos pdns_recursor[6897]: Will not send queries to: 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10, 0.0.0.0/8, 192.0.0.0/24, 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 240.0.0.0/4, ::/96, ::ffff:0:0/96, 100::/64, 2001:db8::/32, 0.0.0.0, ::
Jun 24 09:28:07 vyos pdns_recursor[6897]: Redirecting queries for zone '.' with recursion to: 80.80.80.80:53, 80.80.81.81:53
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting forward zone 'localhost' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting reverse zone '1.0.0.127.in-addr.arpa' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting forward zone 'vyos' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting reverse zone '1.1.0.127.in-addr.arpa' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting rfc 1918 private space zones
Jun 24 09:28:07 vyos pdns_recursor[6897]: Listening for UDP queries on 0.0.0.0:53
Jun 24 09:28:07 vyos pdns_recursor[6897]: Enabled TCP data-ready filter for (slight) DoS protection
Jun 24 09:28:07 vyos pdns_recursor[6897]: Listening for TCP queries on 0.0.0.0:53
Jun 24 09:28:07 vyos pdns_recursor[6897]: Operating unthreaded
Jun 24 09:28:07 vyos systemd[1]: Started PowerDNS Recursor.
Jun 24 09:28:07 vyos pdns_recursor[6897]: Done priming cache with root hints
Jun 24 09:28:07 vyos pdns_recursor[6897]: Done priming cache with root hints
Jun 24 09:28:07 vyos pdns_recursor[6897]: Enabled 'epoll' multiplexer
Jun 24 09:28:07 vyos dhclient: bound to 192.168.0.163 -- renewal in 1498 seconds.
Jun 24 09:28:12 vyos pdns_recursor[6897]: PowerDNS Security Update Mandatory: Upgrade now, see https://doc.powerdns.com/recursor/security-advisories/powerdns-advisory-2020-01.html https://doc.powerdns.com/recursor/security-advisories/powerdns-advisory-2020-02.html https://doc.powerdns.com/recursor/security-advisories/powerdns-adv" "isory-2020-03.html
Jun 24 09:28:19 vyos Keepalived_vrrp[2940]: Warning: Failed to connect to the agentx master agent ([NIL]):

Content of /var/lib/vyos/hostsd.state:

{
	"host_name": "vyos",
	"name_servers": {
		"80.80.80.80": {
			"tag": "static"
		},
		"8.8.8.8": {
			"tag": "dhcp-eth4"
		},
		"8.8.4.4": {
			"tag": "dhcp-eth4"
		},
		"80.80.81.81": {
			"tag": "static"
		}
	},
	"hosts": {},
	"search_domains": [
		"mydomain.net"
	],
	"domain_name": "mydomain.net"
}

Content of /etc/resolv.conf:

### Autogenerated by VyOS ###
### Do not edit, your changes will get overwritten ###

nameserver 80.80.80.80
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 80.80.81.81

search home