I actually tried to post this to phabricator, where I’ve successfully posted bugs in the past, but I’m getting a Cloudflare block error when I press submit ticket, so here it is
Bug Report
On first boot, my client device has a DHCP assigned address as though it were not configured with a static mapping.
After dhclient -r && dhclient, it has the correct address. However, dhclient uses the default configuration on the system, and not the one rendered by systemd-networkd.
Analysis indicates this is a quirk interaction between systemd-networkd and VyOS latest. It did not occur on 1.3.x.
The root cause of the quirk appears to be that systemd-networkd is sending a client identifier, and the new dhcp server is prioritizing using this identifier instead of the mac address (which it will also receive).
from netplan man pages: Ubuntu Manpage: netplan - YAML network configuration abstraction for various backends
• dhcp-identifier (scalar)
(networkd backend only) Sets the source of DHCPv4 client identifier. If mac is
specified, the MAC address of the link is used. If this option is omitted, or if
duid is specified, networkd will generate an RFC4361-compliant client identifier
for the interface by combining the link's IAID and DUID.
Changing netplan to
network:
ethernets:
eno1:
dhcp4: true
dhcp-identifier: mac
version: 2
i.e. adding the dhcp-identifier: mac line now has the correct behavior.
For context below, the client is: appmana-014
Analysis:
Is the static-mapping configured correctly? Yes:
vyos@vyos:~$ show configuration commands | grep appmana-014
set service dhcp-server shared-network-name LAN subnet 10.2.0.0/24 static-mapping appmana-014 ip-address '10.2.0.2'
set service dhcp-server shared-network-name LAN subnet 10.2.0.0/24 static-mapping appmana-014 mac '04:0e:3c:92:23:0c'
set system static-host-mapping host-name appmana-014 inet '10.2.0.2'
set system static-host-mapping host-name appmana-014.i.appmana.com inet '10.2.0.2'
On the client:
appmana-014:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 04:0e:3c:92:23:0c brd ff:ff:ff:ff:ff:ff
altname enp0s31f6
inet 10.2.0.27/24 metric 100 brd 10.2.0.255 scope global dynamic eno1
valid_lft 85326sec preferred_lft 85326sec
inet6 fe80::60e:3cff:fe92:230c/64 scope link
valid_lft forever preferred_lft forever
Observe the mac address is correct.
Are there other DHCP servers? No:
# nmap --script broadcast-dhcp-discover
Starting Nmap 7.80 ( https://nmap.org ) at 2024-06-13 17:58 UTC
Pre-scan script results:
| broadcast-dhcp-discover:
| Response 1 of 1:
| IP Offered: 10.2.0.16
| DHCP Message Type: DHCPOFFER
| Subnet Mask: 255.255.255.0
| Router: 10.2.0.1
| Domain Name Server: 10.2.0.1
| Domain Name: i.appmana.com
| IP Address Lease Time: 1d00h00m00s
|_ Server Identifier: 10.2.0.1
Observe this is the wrong IP.
Did the client record the correct lease? Apparently yes, which is surprising:
appmana-014:~# cat /var/lib/dhcp/dhclient.leases
lease {
interface "eno1";
fixed-address 10.2.0.2;
option subnet-mask 255.255.255.0;
option dhcp-lease-time 86400;
option routers 10.2.0.1;
option dhcp-message-type 5;
option dhcp-server-identifier 10.2.0.1;
option domain-name-servers 10.2.0.1;
option domain-search "i.appmana.com.";
option host-name "appmana-014";
option domain-name "i.appmana.com";
renew 4 2024/06/13 03:26:57;
rebind 4 2024/06/13 14:02:14;
expire 4 2024/06/13 17:02:14;
}
lease {
interface "eno1";
fixed-address 10.2.0.2;
option subnet-mask 255.255.255.0;
option dhcp-lease-time 86400;
option routers 10.2.0.1;
option dhcp-message-type 5;
option dhcp-server-identifier 10.2.0.1;
option domain-name-servers 10.2.0.1;
option domain-search "i.appmana.com.";
option host-name "appmana-014";
option domain-name "i.appmana.com";
renew 4 2024/06/13 14:43:56;
rebind 5 2024/06/14 00:26:57;
expire 5 2024/06/14 03:26:57;
}
lease {
interface "eno1";
fixed-address 10.2.0.2;
option subnet-mask 255.255.255.0;
option dhcp-lease-time 86400;
option routers 10.2.0.1;
option dhcp-message-type 5;
option dhcp-server-identifier 10.2.0.1;
option domain-name-servers 10.2.0.1;
option domain-search "i.appmana.com.";
option host-name "appmana-014";
option domain-name "i.appmana.com";
renew 5 2024/06/14 00:39:04;
rebind 5 2024/06/14 11:43:56;
expire 5 2024/06/14 14:43:56;
}
Does VyOS show valid lease state? No:
Observe two active entries:
vyos@vyos:~$ show dhcp server leases state all
IP Address MAC address State Lease start Lease expiration Remaining Pool Hostname Origin
------------ ----------------- ------- ------------------- ------------------- ----------- ------ --------------------------- --------
10.2.0.2 04:0e:3c:92:23:0c active 2024/06/13 17:37:43 2024/06/14 17:37:43 23:36:49 LAN appmana-014 local
...
10.2.0.27 04:0e:3c:92:23:0c active 2024/06/13 17:38:05 2024/06/14 17:38:05 23:37:11 LAN appmana-014 local
Is there a workaround?: Yes, running dhclient -r && dhclient resolves the issue. Here is the state of everything after that is run:
$ ssh [email protected] -- ip address
...
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 04:0e:3c:92:23:0c brd ff:ff:ff:ff:ff:ff
altname enp0s31f6
inet 10.2.0.2/24 brd 10.2.0.255 scope global dynamic eno1
valid_lft 86332sec preferred_lft 86332sec
inet6 fe80::60e:3cff:fe92:230c/64 scope link
valid_lft forever preferred_lft forever
$ ssh [email protected] -- cat /var/lib/dhcp/dhclient.leases
lease {
interface "eno1";
fixed-address 10.2.0.2;
option subnet-mask 255.255.255.0;
option dhcp-lease-time 86400;
option routers 10.2.0.1;
option dhcp-message-type 5;
option dhcp-server-identifier 10.2.0.1;
option domain-name-servers 10.2.0.1;
option domain-search "i.appmana.com.";
option host-name "appmana-014";
option domain-name "i.appmana.com";
renew 4 2024/06/13 18:03:37;
rebind 4 2024/06/13 18:03:37;
expire 4 2024/06/13 18:03:37;
}
lease {
interface "eno1";
fixed-address 10.2.0.2;
option subnet-mask 255.255.255.0;
option dhcp-lease-time 86400;
option routers 10.2.0.1;
option dhcp-message-type 5;
option dhcp-server-identifier 10.2.0.1;
option domain-name-servers 10.2.0.1;
option domain-search "i.appmana.com.";
option host-name "appmana-014";
option domain-name "i.appmana.com";
renew 5 2024/06/14 05:54:05;
rebind 5 2024/06/14 15:03:39;
expire 5 2024/06/14 18:03:39;
}
VyOS correctly shows only one lease entry:
IP Address MAC address State Lease start Lease expiration Remaining Pool Hostname Origin
------------ ----------------- ------- ------------------- ------------------- ----------- ------ --------------------------- --------
10.2.0.2 04:0e:3c:92:23:0c active 2024/06/13 18:03:39 2024/06/14 18:03:39 23:44:41 LAN appmana-014 local
Is the client misconfigured? No, netplan is vanilla:
appmana-014:~# cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
eno1:
dhcp4: true
version: 2
for a long time now, the client receives the wrong dhcp address via systemd-networkd. However, under 1.3.1, these logs were identical in character but with the correct address, not the incorrect one.
Jun 12 14:27:55 appmana-014 systemd-networkd[564]: eno1: Lost carrier
Jun 12 14:27:55 appmana-014 systemd-networkd[564]: eno1: DHCP lease lost
Jun 12 14:27:55 appmana-014 systemd-networkd[564]: eno1: DHCPv6 lease lost
Jun 12 14:27:58 appmana-014 systemd-networkd[564]: eno1: Gained carrier
Jun 12 14:28:02 appmana-014 systemd-networkd[564]: eno1: DHCPv4 address 10.2.0.24/24 via 10.2.0.1
Jun 13 16:03:50 appmana-014 systemd-networkd[564]: eno1: Lost carrier
Jun 13 16:03:50 appmana-014 systemd-networkd[564]: eno1: DHCP lease lost
Jun 13 16:03:50 appmana-014 systemd-networkd[564]: eno1: DHCPv6 lease lost
Jun 13 16:03:53 appmana-014 systemd-networkd[564]: eno1: Gained carrier
Jun 13 16:03:56 appmana-014 systemd-networkd[564]: eno1: DHCPv4 address 10.2.0.24/24 via 10.2.0.1
Jun 13 17:36:01 appmana-014 systemd-networkd[564]: eno1: DHCP lease lost
Jun 13 17:36:01 appmana-014 systemd-networkd[564]: eno1: DHCPv6 lease lost
Jun 13 17:37:59 appmana-014 systemd-networkd[566]: eno1: Link UP
Jun 13 17:38:01 appmana-014 systemd-networkd[566]: eno1: Gained carrier
Jun 13 17:38:02 appmana-014 systemd-networkd[566]: eno1: Gained IPv6LL
Jun 13 17:38:05 appmana-014 systemd-networkd[566]: eno1: DHCPv4 address 10.2.0.27/24 via 10.2.0.1
I don’t believe this dhclient.conf is used by systemd-networkd, but it is used by dhclient which resolves the issue:
appmana-014:~# cat /etc/dhcp/dhclient.conf
# Configuration file for /sbin/dhclient.
#
# This is a sample configuration file for dhclient. See dhclient.conf's
# man page for more information about the syntax of this file
# and a more comprehensive list of the parameters understood by
# dhclient.
#
# Normally, if the DHCP server provides reasonable information and does
# not leave anything out (like the domain name, for example), then
# few changes must be made to this file, if any.
#
option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;
send host-name = gethostname();
request subnet-mask, broadcast-address, time-offset, routers,
domain-name, domain-name-servers, domain-search, host-name,
dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers,
netbios-name-servers, netbios-scope, interface-mtu,
rfc3442-classless-static-routes, ntp-servers;
...
netplan’s rendered systemd-networkd configuration:
# cat /run/systemd/network/10-netplan-eno1.network
[Match]
Name=eno1
[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6
[DHCP]
RouteMetric=100
UseMTU=true
incorrect status from networkctl
appmana-014:~# networkctl status eno1
● 2: eno1
Link File: /usr/lib/systemd/network/99-default.link
Network File: /run/systemd/network/10-netplan-eno1.network
Type: ether
State: routable (configured)
Online state: online
Alternative Names: enp0s31f6
Path: pci-0000:00:1f.6
Driver: e1000e
Vendor: Intel Corporation
Model: Ethernet Connection (7) I219-LM
HW Address: 04:0e:3c:92:23:0c (HP Inc.)
MTU: 1500 (min: 68, max: 9000)
QDisc: fq_codel
IPv6 Address Generation Mode: eui64
Queue Length (Tx/Rx): 1/1
Auto negotiation: yes
Speed: 1Gbps
Duplex: full
Port: tp
Address: 10.2.0.27 (DHCP4 via 10.2.0.1)
fe80::60e:3cff:fe92:230c
Gateway: 10.2.0.1
DNS: 10.2.0.1
Search Domains: i.appmana.com
Activation Policy: up
Required For Online: yes
DHCP4 Client ID: IAID:0xb6220feb/DUID
DHCP6 Client DUID: DUID-EN/Vendor:0000ab118067c6c52548008b0000
Connected To: QSW-M2116P on port 6 (2.5GigabitEthernet 1/6)
Jun 13 18:25:46 appmana-014 systemd-networkd[566]: eno1: Lost carrier
Jun 13 18:25:46 appmana-014 systemd-networkd[566]: eno1: DHCP lease lost
Jun 13 18:25:46 appmana-014 systemd-networkd[566]: eno1: DHCPv6 lease lost
Jun 13 18:25:48 appmana-014 systemd-networkd[566]: eno1: Gained carrier
Jun 13 18:25:53 appmana-014 systemd-networkd[566]: eno1: DHCPv4 address 10.2.0.27/24 via 10.2.0.1
Jun 13 18:26:09 appmana-014 systemd-networkd[566]: eno1: Lost carrier
Jun 13 18:26:09 appmana-014 systemd-networkd[566]: eno1: DHCP lease lost
Jun 13 18:26:09 appmana-014 systemd-networkd[566]: eno1: DHCPv6 lease lost
Jun 13 18:26:11 appmana-014 systemd-networkd[566]: eno1: Gained carrier
Jun 13 18:26:15 appmana-014 systemd-networkd[566]: eno1: DHCPv4 address 10.2.0.27/24 via 10.2.0.1
Client Versions
# lsb_release -idrc
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
# uname -a
Linux appmana-014 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
VyOS versions:
vyos@vyos:~$ show version
Version: VyOS 1.5-rolling-202406130020
Release train: current
Release flavor: generic
Built by: [email protected]
Built on: Thu 13 Jun 2024 03:16 UTC
Build UUID: 84281102-574f-443d-83f7-af82b6ba3512
Build commit ID: f2154b4252535e
Architecture: x86_64
Boot via: installed image
System type: bare metal
Hardware vendor: HP
Hardware model: HP EliteDesk 800 G5 Desktop Mini
Hardware S/N: MXL95025NY
Hardware UUID: 800b5dc3-e6c8-ba65-0bcb-dc6bfdfbccb2
Copyright: VyOS maintainers and contributors