Kernel Panic with set-mss pmtu

I’m currently in VyOS build 1.3-rolling-202008311805, and I’m trying to use set-mss pmtu but it keep kernel panicking once I set the policy route onto my interfaces, the kernel panic didn’t happen with iptables command and only happen with set tcp-mss pmtu

Regards,
Raphiel

Hi @raphielscape, can you describe the reproduction steps more detailed?

Yes, I’m trying to set set-mss pmtu to Wireguard interface

set policy route MSS-CLAMP rule 10 protocol tcp
set policy route MSS-CLAMP rule 10 set tcp-mss pmtu
set policy route MSS-CLAMP rule 10 tcp flags SYN
set interfaces wireguard wg0 policy route MSS-CLAMP

After I do commit, it went kernel panics immediately

EDIT : It’s also panics when I set it to ethernet interface, tried to apply the same policy route to eth0.5 it also panics, and then tried to set it to eth0 it also still panics

Hello @raphielscape, thanks for the description.
I can’t reproduce this issue on Qemu KVM hypervisor with VyOS 1.3-rolling-202009030118 version.
Which hypervisor using in your case? Or this is bare-metal?

I’m using a bare-metal, I’m able to reproduce this behavior on three machines

Can you try to reproduce on the latest rolling images?

Yes, I’m still able to reproduce it in 1.3-rolling-202009030118

Got it, thanks

[  695.375390] BUG: kernel NULL pointer dereference, address: 0000000000000008
[  695.380568] #PF: supervisor read access in kernel mode
[  695.384282] #PF: error_code(0x0000) - not-present page
[  695.387973] PGD 0 P4D 0 
[  695.389995] Oops: 0000 [#1] SMP PTI
[  695.392685] CPU: 0 PID: 12 Comm: kworker/0:1 Not tainted 5.8.5-amd64-vyos #1
[  695.397655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org
[  695.405985] Workqueue: wg-crypt-wg01 wg_packet_decrypt_worker [wireguard]
[  695.410832] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[  695.415232] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 28 d0 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 8 
[  695.427246] RSP: 0018:ffffaba4000038c8 EFLAGS: 00010246
[  695.430213] RAX: 000000000000058c RBX: ffff929348bb708e RCX: 0000000000000001
[  695.433802] RDX: 0000000000000000 RSI: ffff9293493ffa18 RDI: 0000000000000000
[  695.437290] RBP: ffffaba400003978 R08: ffff92935fc23808 R09: 0000000000000014
[  695.440992] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[  695.444938] R13: ffff92935ea17200 R14: 0000000000000028 R15: ffffaba400003bb0
[  695.448912] FS:  0000000000000000(0000) GS:ffff92935fc00000(0000) knlGS:0000000000000000
[  695.453332] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  695.456468] CR2: 0000000000000008 CR3: 000000000622a000 CR4: 00000000000006f0
[  695.460371] Call Trace:
[  695.462005]  <IRQ>
[  695.463479]  tcpmss_tg4+0x2c/0xa0 [xt_TCPMSS]
[  695.466070]  nft_target_eval_xt+0x30/0x50 [nft_compat]
[  695.469145]  nft_do_chain+0x149/0x4c0 [nf_tables]
[  695.472396]  ? check_preempt_curr+0x75/0x90
[  695.474955]  ? try_to_wake_up+0x199/0x3c0
[  695.477344]  ? __queue_work+0xd9/0x310
[  695.479803]  ? wg_packet_receive+0x3cf/0x6a0 [wireguard]
[  695.482731]  ? send6+0x3a0/0x3a0 [wireguard]
[  695.485229]  ? send6+0x3a0/0x3a0 [wireguard]
[  695.487807]  ? wg_receive+0x1d/0x30 [wireguard]
[  695.490499]  ? udp_queue_rcv_one_skb+0x1d4/0x460
[  695.493214]  ? udp_unicast_rcv_skb.isra.66+0x6f/0x80
[  695.496097]  ? __udp4_lib_rcv+0x553/0xb70
[  695.498243]  nft_do_chain_ipv4+0x61/0x80 [nf_tables]
[  695.500824]  nf_hook_slow+0x3f/0xc0
[  695.503371]  ? ip_local_deliver_finish+0x3f/0x50
[  695.506556]  nf_hook_slow_list+0x89/0x130
[  695.509450]  ip_sublist_rcv+0x1fb/0x210
[  695.511909]  ? ip_rcv_finish_core.isra.22+0x400/0x400
[  695.514818]  ip_list_rcv+0x132/0x156
[  695.517005]  __netif_receive_skb_list_core+0x296/0x2c0
[  695.520005]  netif_receive_skb_list_internal+0x1a1/0x2c0
[  695.523094]  ? dev_gro_receive+0x61e/0x690
[  695.525468]  gro_normal_list.part.162+0x14/0x30
[  695.528079]  napi_complete_done+0x62/0x170
[  695.530534]  wg_packet_rx_poll+0x60c/0xa10 [wireguard]
[  695.533392]  ? virtnet_poll+0x2e0/0x330 [virtio_net]
[  695.536166]  net_rx_action+0xf6/0x2e0
[  695.538267]  __do_softirq+0xd2/0x227
[  695.540351]  asm_call_on_stack+0x12/0x20
[  695.542529]  </IRQ>
[  695.543966]  do_softirq_own_stack+0x34/0x40
[  695.546559]  do_softirq.part.19+0x3c/0x40
[  695.548906]  __local_bh_enable_ip+0x46/0x50
[  695.551332]  process_one_work+0x189/0x2e0
[  695.553736]  ? create_worker+0x190/0x190
[  695.556203]  worker_thread+0x2b/0x380
[  695.558301]  ? create_worker+0x190/0x190
[  695.560557]  kthread+0x10c/0x130
[  695.562536]  ? kthread_park+0x80/0x80
[  695.564888]  ret_from_fork+0x22/0x30
[  695.567047] Modules linked in: ip_set xt_TCPMSS xt_comment wireguard libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ipi
[  695.607586] CR2: 0000000000000008
[  695.609627] ---[ end trace 4a212d01f48208e2 ]---
[  695.612307] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS]
[  695.615765] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 28 d0 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 8 
[  695.625532] RSP: 0018:ffffaba4000038c8 EFLAGS: 00010246
[  695.628539] RAX: 000000000000058c RBX: ffff929348bb708e RCX: 0000000000000001
[  695.632722] RDX: 0000000000000000 RSI: ffff9293493ffa18 RDI: 0000000000000000
[  695.637411] RBP: ffffaba400003978 R08: ffff92935fc23808 R09: 0000000000000014
[  695.642314] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028
[  695.648322] R13: ffff92935ea17200 R14: 0000000000000028 R15: ffffaba400003bb0
[  695.653263] FS:  0000000000000000(0000) GS:ffff92935fc00000(0000) knlGS:0000000000000000
[  695.659701] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  695.664956] CR2: 0000000000000008 CR3: 000000000622a000 CR4: 00000000000006f0
[  695.671007] Kernel panic - not syncing: Fatal exception in interrupt
[  695.675822] Kernel Offset: 0xf400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  695.682892] Rebooting in 60 seconds..

ps:/ For reproducing this issue needs active traffic.
Let me check in stable, I think something related to nftables in rolling.

1 Like

Hello @raphielscape, can you try manually add rules instead of CLI commands? I don’t know why, but it happens only on mangle chain

sudo iptables -I FORWARD -i wg0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

I have generated a bug report.
Even ethernet is susceptible to this.

https://phabricator.vyos.net/T2868

Yes, it didn’t panic with forward chain, as I stated that this didn’t happens with iptables and only with set tcp-mss pmtu CLI command

Hi @raphielscape, thanks for testing, you can track progress in our development portal.
https://phabricator.vyos.net/T2868