Disable Quick Assist?

General questions
, , ,
1

Hi. We just discovered this kernel panic bug when we moved the WAN port on our router to new hardware and the new machine immediately kernel panicked.

So, now the question is, can we disable the Quick Assist driver in 1.4, so that I can use VTI IPSec connections without kernel panicking?

We had to set up a connection on short notice two years back and had a Dell PowerEdge R530 lying around. Now we’re trying to upgrade the hardware to a SuperMicro SYS-1019D-FRN8TP, to be able to upgrade the WAN from 1gbit to 10gbit.

We temporarily moved some tunnels to OpenVPN, to be able to use the new hardware, but I’d really rather like to have those tunnels be IPsec.

Here is the output of “show version” on the new hardware:

Version:          VyOS 1.4-rolling-202104202252
Release Train:    sagitta

Built by:         [email protected]
Built on:         Wed 21 Apr 2021 01:17 UTC
Build UUID:       72a94b35-9559-46e1-999c-e72e28a0e281
Build Commit ID:  a934e73e2738e5

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Supermicro
Hardware model:   SYS-1019D-FRN8TP
Hardware S/N:     A354290X1301173
Hardware UUID:    3b72d000-468b-11ea-8000-3cecef44e298

Copyright:        VyOS maintainers and contributors
2

Believe this is a known issue on 1.4 per:

https://phabricator.vyos.net/T3484

You need to disable qat in your configuration.

3

Hi @ngoehring and thanks for the quick reply.

How do I disable QAT? It looks like QAT is already disabled, according to this:

asgeir@fw-thor-new:~$ show system acceleration qat status 
         system acceleration qat is not configured
asgeir@fw-thor-new:~$ show system acceleration qat 
b5:00.0 Co-processor [0b40]: Intel Corporation C62x Chipset QuickAssist Technology [8086:37c8] (rev 04)
b6:00.0 Co-processor [0b40]: Intel Corporation C62x Chipset QuickAssist Technology [8086:37c8] (rev 04)
4

That’s a great question and one I don’t know the answer to. Have you considered using Vyos 1.3 RC4? It does not suffer from this issue as the kernel version is different.

5

If 1.3 is a no go, I’m sure someone smarter than me will reply sometime soon. Wish I could be of more help.

6

Try to check configuration related qat.

show configuration commands | match qat

7

Nothing:

asgeir@fw-thor-new:~$ show configuration commands | match qat
asgeir@fw-thor-new:~$
8

Apropos of this, does anybody know how I downgrade a running system? Does add system image <url> work to downgrade or only to upgrade? (I’m guessing the config migration scripts that run at boot don’t like seeing versions from newer versions)

9

With this command " add system image " , you can go to the lower version also.

10

Ok, great! Thanks for the answer. I’ll try to schedule downtime as soon as possible and try downgrading to 1.3

11

Hello @asgeirbjarnason. I guess all modules should be blacklisted. Could you show sudo dmesg after the system completely boot?

vyos@vyos#  cat /etc/modprobe.d/intel-qat-blacklist.conf
blacklist intel_qat
blacklist qat_c3xxx
blacklist c6xx_dev0
blacklist d15xx_dev0
blacklist dh895xcc_dev0
12

Hi @Dmitry. I’m not in a position to reboot it now, but /proc/modules shows a module from the intel-qat-blacklist.conf file.

/proc/modules:

asgeir@fw-thor-new:~$ cat /proc/modules | grep qat
qat_c62x 20480 0 - Live 0x0000000000000000 (O)
intel_qat 303104 1 qat_c62x, Live 0x0000000000000000 (O)
dh_generic 16384 1 intel_qat, Live 0x0000000000000000
uio 20480 1 intel_qat, Live 0x0000000000000000
authenc 16384 1 intel_qat, Live 0x0000000000000000

The blacklist file:

asgeir@fw-thor-new:~$ cat /etc/modprobe.d/intel-qat-blacklist.conf
blacklist intel_qat
blacklist qat_c3xxx
blacklist c6xx_dev0
blacklist d15xx_dev0
blacklist dh895xcc_dev0
13

Hello @asgeirbjarnason
I tested the device with QAT support in our lab, and modules do not load when you have disabled QAT acceleration in VyOS CLI

vyos@R2-QAT:~$ show version | match Version
Version:          VyOS 1.4-rolling-202106062324
vyos@R2-QAT:~$ show configuration commands | match qat
vyos@R2-QAT:~$ sudo lsmod | grep qat
vyos@R2-QAT:~$  show system cpu 

CPU Vendor:       GenuineIntel
Model:            Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
Total CPUs:       0-7
Sockets:          1
Cores:            8
Threads:          1
Current MHz:      2197.813
Minimum MHz:      800.0000
Maximum MHz:      2200.0000

vyos@R2-QAT:~$ cat /proc/modules | grep qat
vyos@R2-QAT:~$
14

Uhh… I tried downgrading to 1.3 RC4 but I can still see intel_qat in /proc/modules:

asgeir@fw-thor:~$ cat /proc/modules | grep qat
qat_c62x 16384 2 - Live 0x0000000000000000 (O)
intel_qat 278528 217 qat_c62x, Live 0x0000000000000000 (O)
dh_generic 16384 1 intel_qat, Live 0x0000000000000000
uio 20480 1 intel_qat, Live 0x0000000000000000
authenc 16384 1 intel_qat, Live 0x0000000000000000

Show version:

asgeir@fw-thor:~$ show version

Version:          VyOS 1.3.0-rc4
Release Train:    equuleus

Built by:         Sentrium S.L.
Built on:         Mon 19 Apr 2021 08:28 UTC
Build UUID:       8d9996d2-511e-4dea-be4f-cd4515c404f3
Build Commit ID:  2aac286ccfe594

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Supermicro
Hardware model:   SYS-1019D-FRN8TP
Hardware S/N:     A354290X1301173
Hardware UUID:    3b72d000-468b-11ea-8000-3cecef44e298

Copyright:        VyOS maintainers and contributors

(oh, and there is no mention of QAT in the config)

asgeir@fw-thor:~$ show configuration | grep qat
asgeir@fw-thor:~$
15

Hello @asgeirbjarnason , could you grep -e qat from dmesg?

16

Sorry, sudo dmesg | grep -e qat returns nothing. The earliest message in dmesg is:

[29517.546456] [WAN-to-DMZ-2-D] IN=eth7 OUT=bond0.400 MAC=3c:ec:ef:44:da:9b:a0:93:51:20:4b:19:08:00 SRC=178.19.50.46 DST=31.209.136.134 LEN=40 TOS=0x00 PREC=0x00 TTL=122 ID=24982 DF PROTO=TCP SPT=5421 DPT=443 WINDOW=0 RES=0x00 ACK RST URGP=0

Looks like I have too much firewall logging turned on which is purging the boot messages from dmesg