I did some more digging, and it seems that this behavior is related to dhclient and vyos-hostsd. As you can see in the logs below it starts with dhclient performing a request, followed by vyos-hostsd doing some stuff with the data dhclient received. When vyos-hostsd is finished you see systemd performing a restart of PowerDNS after which my forwarding configuration is gone.
What’s also interesting is that the configured domain and search-domain (in resolv.conf) get overwritten with the one that DHCP offers. But I would assume that statically configured domains should take presedence.
Logs in /var/log/messages:
Jun 24 09:28:04 vyos Keepalived_vrrp[2940]: message repeated 10 times: [ Warning: Failed to connect to the agentx master agent ([NIL]):]
Jun 24 09:28:06 vyos dhclient: DHCPREQUEST on eth4 to 192.168.0.1 port 67
Jun 24 09:28:06 vyos dhclient: DHCPACK from 192.168.0.1
Jun 24 09:28:06 vyos vyos-hostsd[660]: Received a configuration change request
Jun 24 09:28:06 vyos vyos-hostsd[660]: Request data: {"op": "set", "data": {"host_name": null, "domain_name": null, "search_domains": ["home"]}, "type": "host_name"}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/resolv.conf
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/hosts
Jun 24 09:28:06 vyos vyos-hostsd[660]: Saving state to /var/lib/vyos/hostsd.state
Jun 24 09:28:06 vyos vyos-hostsd[660]: Sent response: {'data': None}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Received a configuration change request
Jun 24 09:28:06 vyos vyos-hostsd[660]: Request data: {"type": "name_servers", "tag": "dhcp-eth4", "op": "delete"}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/resolv.conf
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/hosts
Jun 24 09:28:06 vyos vyos-hostsd[660]: Saving state to /var/lib/vyos/hostsd.state
Jun 24 09:28:06 vyos vyos-hostsd[660]: Sent response: {'data': None}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Received a configuration change request
Jun 24 09:28:06 vyos vyos-hostsd[660]: Request data: {"type": "name_servers", "data": ["8.8.8.8", "8.8.4.4"], "tag": "dhcp-eth4", "op": "add"}
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/resolv.conf
Jun 24 09:28:06 vyos vyos-hostsd[660]: Writing /etc/hosts
Jun 24 09:28:06 vyos vyos-hostsd[660]: Saving state to /var/lib/vyos/hostsd.state
Jun 24 09:28:06 vyos vyos-hostsd[660]: Sent response: {'data': None}
Jun 24 09:28:07 vyos systemd[1]: Stopping PowerDNS Recursor...
Jun 24 09:28:07 vyos systemd[1]: Starting PowerDNS Recursor...
Jun 24 09:28:07 vyos pdns_recursor[6897]: PowerDNS Recursor 4.1.15 (C) 2001-2018 PowerDNS.COM BV
Jun 24 09:28:07 vyos pdns_recursor[6897]: Using 64-bits mode. Built using gcc 4.9.2 on Dec 3 2019 13:02:24 by root@613ec8350b02.
Jun 24 09:28:07 vyos pdns_recursor[6897]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Jun 24 09:28:07 vyos pdns_recursor[6897]: Reading random entropy from '/dev/urandom'
Jun 24 09:28:07 vyos pdns_recursor[6897]: Enabling IPv6 transport for outgoing queries
Jun 24 09:28:07 vyos pdns_recursor[6897]: Only allowing queries from: 10.0.0.0/8
Jun 24 09:28:07 vyos pdns_recursor[6897]: Will not send queries to: 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10, 0.0.0.0/8, 192.0.0.0/24, 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 240.0.0.0/4, ::/96, ::ffff:0:0/96, 100::/64, 2001:db8::/32, 0.0.0.0, ::
Jun 24 09:28:07 vyos pdns_recursor[6897]: Redirecting queries for zone '.' with recursion to: 80.80.80.80:53, 80.80.81.81:53
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting forward zone 'localhost' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting reverse zone '1.0.0.127.in-addr.arpa' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting forward zone 'vyos' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting reverse zone '1.1.0.127.in-addr.arpa' based on hosts file
Jun 24 09:28:07 vyos pdns_recursor[6897]: Inserting rfc 1918 private space zones
Jun 24 09:28:07 vyos pdns_recursor[6897]: Listening for UDP queries on 0.0.0.0:53
Jun 24 09:28:07 vyos pdns_recursor[6897]: Enabled TCP data-ready filter for (slight) DoS protection
Jun 24 09:28:07 vyos pdns_recursor[6897]: Listening for TCP queries on 0.0.0.0:53
Jun 24 09:28:07 vyos pdns_recursor[6897]: Operating unthreaded
Jun 24 09:28:07 vyos systemd[1]: Started PowerDNS Recursor.
Jun 24 09:28:07 vyos pdns_recursor[6897]: Done priming cache with root hints
Jun 24 09:28:07 vyos pdns_recursor[6897]: Done priming cache with root hints
Jun 24 09:28:07 vyos pdns_recursor[6897]: Enabled 'epoll' multiplexer
Jun 24 09:28:07 vyos dhclient: bound to 192.168.0.163 -- renewal in 1498 seconds.
Jun 24 09:28:12 vyos pdns_recursor[6897]: PowerDNS Security Update Mandatory: Upgrade now, see https://doc.powerdns.com/recursor/security-advisories/powerdns-advisory-2020-01.html https://doc.powerdns.com/recursor/security-advisories/powerdns-advisory-2020-02.html https://doc.powerdns.com/recursor/security-advisories/powerdns-adv" "isory-2020-03.html
Jun 24 09:28:19 vyos Keepalived_vrrp[2940]: Warning: Failed to connect to the agentx master agent ([NIL]):
Content of /var/lib/vyos/hostsd.state:
{
"host_name": "vyos",
"name_servers": {
"80.80.80.80": {
"tag": "static"
},
"8.8.8.8": {
"tag": "dhcp-eth4"
},
"8.8.4.4": {
"tag": "dhcp-eth4"
},
"80.80.81.81": {
"tag": "static"
}
},
"hosts": {},
"search_domains": [
"mydomain.net"
],
"domain_name": "mydomain.net"
}
Content of /etc/resolv.conf:
### Autogenerated by VyOS ###
### Do not edit, your changes will get overwritten ###
nameserver 80.80.80.80
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 80.80.81.81
search home