I’m currently in VyOS build 1.3-rolling-202008311805, and I’m trying to use set-mss pmtu but it keep kernel panicking once I set the policy route onto my interfaces, the kernel panic didn’t happen with iptables command and only happen with set tcp-mss pmtu
Regards,
Raphiel
Yes, I’m trying to set set-mss pmtu to Wireguard interface
set policy route MSS-CLAMP rule 10 protocol tcp
set policy route MSS-CLAMP rule 10 set tcp-mss pmtu
set policy route MSS-CLAMP rule 10 tcp flags SYN
set interfaces wireguard wg0 policy route MSS-CLAMP
After I do commit, it went kernel panics immediately
EDIT : It’s also panics when I set it to ethernet interface, tried to apply the same policy route to eth0.5 it also panics, and then tried to set it to eth0 it also still panics
Hello @raphielscape, thanks for the description.
I can’t reproduce this issue on Qemu KVM hypervisor with VyOS 1.3-rolling-202009030118 version.
Which hypervisor using in your case? Or this is bare-metal?
I’m using a bare-metal, I’m able to reproduce this behavior on three machines
Can you try to reproduce on the latest rolling images?
Yes, I’m still able to reproduce it in 1.3-rolling-202009030118
Got it, thanks
[ 695.375390] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 695.380568] #PF: supervisor read access in kernel mode
[ 695.384282] #PF: error_code(0x0000) - not-present page
[ 695.387973] PGD 0 P4D 0
[ 695.389995] Oops: 0000 [#1] SMP PTI
[ 695.392685] CPU: 0 PID: 12 Comm: kworker/0:1 Not tainted 5.8.5-amd64-vyos #1
[ 695.397655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org
[ 695.405985] Workqueue: wg-crypt-wg01 wg_packet_decrypt_worker [wireguard]
[ 695.410832] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[ 695.415232] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 28 d0 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 8
[ 695.427246] RSP: 0018:ffffaba4000038c8 EFLAGS: 00010246
[ 695.430213] RAX: 000000000000058c RBX: ffff929348bb708e RCX: 0000000000000001
[ 695.433802] RDX: 0000000000000000 RSI: ffff9293493ffa18 RDI: 0000000000000000
[ 695.437290] RBP: ffffaba400003978 R08: ffff92935fc23808 R09: 0000000000000014
[ 695.440992] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[ 695.444938] R13: ffff92935ea17200 R14: 0000000000000028 R15: ffffaba400003bb0
[ 695.448912] FS: 0000000000000000(0000) GS:ffff92935fc00000(0000) knlGS:0000000000000000
[ 695.453332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 695.456468] CR2: 0000000000000008 CR3: 000000000622a000 CR4: 00000000000006f0
[ 695.460371] Call Trace:
[ 695.462005] <IRQ>
[ 695.463479] tcpmss_tg4+0x2c/0xa0 [xt_TCPMSS]
[ 695.466070] nft_target_eval_xt+0x30/0x50 [nft_compat]
[ 695.469145] nft_do_chain+0x149/0x4c0 [nf_tables]
[ 695.472396] ? check_preempt_curr+0x75/0x90
[ 695.474955] ? try_to_wake_up+0x199/0x3c0
[ 695.477344] ? __queue_work+0xd9/0x310
[ 695.479803] ? wg_packet_receive+0x3cf/0x6a0 [wireguard]
[ 695.482731] ? send6+0x3a0/0x3a0 [wireguard]
[ 695.485229] ? send6+0x3a0/0x3a0 [wireguard]
[ 695.487807] ? wg_receive+0x1d/0x30 [wireguard]
[ 695.490499] ? udp_queue_rcv_one_skb+0x1d4/0x460
[ 695.493214] ? udp_unicast_rcv_skb.isra.66+0x6f/0x80
[ 695.496097] ? __udp4_lib_rcv+0x553/0xb70
[ 695.498243] nft_do_chain_ipv4+0x61/0x80 [nf_tables]
[ 695.500824] nf_hook_slow+0x3f/0xc0
[ 695.503371] ? ip_local_deliver_finish+0x3f/0x50
[ 695.506556] nf_hook_slow_list+0x89/0x130
[ 695.509450] ip_sublist_rcv+0x1fb/0x210
[ 695.511909] ? ip_rcv_finish_core.isra.22+0x400/0x400
[ 695.514818] ip_list_rcv+0x132/0x156
[ 695.517005] __netif_receive_skb_list_core+0x296/0x2c0
[ 695.520005] netif_receive_skb_list_internal+0x1a1/0x2c0
[ 695.523094] ? dev_gro_receive+0x61e/0x690
[ 695.525468] gro_normal_list.part.162+0x14/0x30
[ 695.528079] napi_complete_done+0x62/0x170
[ 695.530534] wg_packet_rx_poll+0x60c/0xa10 [wireguard]
[ 695.533392] ? virtnet_poll+0x2e0/0x330 [virtio_net]
[ 695.536166] net_rx_action+0xf6/0x2e0
[ 695.538267] __do_softirq+0xd2/0x227
[ 695.540351] asm_call_on_stack+0x12/0x20
[ 695.542529] </IRQ>
[ 695.543966] do_softirq_own_stack+0x34/0x40
[ 695.546559] do_softirq.part.19+0x3c/0x40
[ 695.548906] __local_bh_enable_ip+0x46/0x50
[ 695.551332] process_one_work+0x189/0x2e0
[ 695.553736] ? create_worker+0x190/0x190
[ 695.556203] worker_thread+0x2b/0x380
[ 695.558301] ? create_worker+0x190/0x190
[ 695.560557] kthread+0x10c/0x130
[ 695.562536] ? kthread_park+0x80/0x80
[ 695.564888] ret_from_fork+0x22/0x30
[ 695.567047] Modules linked in: ip_set xt_TCPMSS xt_comment wireguard libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ipi
[ 695.607586] CR2: 0000000000000008
[ 695.609627] ---[ end trace 4a212d01f48208e2 ]---
[ 695.612307] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[ 695.615765] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 28 d0 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 8
[ 695.625532] RSP: 0018:ffffaba4000038c8 EFLAGS: 00010246
[ 695.628539] RAX: 000000000000058c RBX: ffff929348bb708e RCX: 0000000000000001
[ 695.632722] RDX: 0000000000000000 RSI: ffff9293493ffa18 RDI: 0000000000000000
[ 695.637411] RBP: ffffaba400003978 R08: ffff92935fc23808 R09: 0000000000000014
[ 695.642314] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[ 695.648322] R13: ffff92935ea17200 R14: 0000000000000028 R15: ffffaba400003bb0
[ 695.653263] FS: 0000000000000000(0000) GS:ffff92935fc00000(0000) knlGS:0000000000000000
[ 695.659701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 695.664956] CR2: 0000000000000008 CR3: 000000000622a000 CR4: 00000000000006f0
[ 695.671007] Kernel panic - not syncing: Fatal exception in interrupt
[ 695.675822] Kernel Offset: 0xf400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 695.682892] Rebooting in 60 seconds..
ps:/ For reproducing this issue needs active traffic.
Let me check in stable, I think something related to nftables in rolling.
Hello @raphielscape, can you try manually add rules instead of CLI commands? I don’t know why, but it happens only on mangle chain
sudo iptables -I FORWARD -i wg0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
I have generated a bug report.
Even ethernet is susceptible to this.
Yes, it didn’t panic with forward chain, as I stated that this didn’t happens with iptables and only with set tcp-mss pmtu CLI command
Hi @raphielscape, thanks for testing, you can track progress in our development portal.
https://phabricator.vyos.net/T2868