Kernel logs are not written to log file

Bugs
, , ,
1

Hi,

I have tried all the versions of vyos both Crux and rolling release.

My problem is kernel logs where firewall log, interface activity happens are never written back to /var/log/messages.

I have checked that kernel messages are coming to /proc/kmsg. But never got written to messages file.

It feels like something is buffering those messages.

The only way to get those messages written to log file is if I restart the syslog service. Then all of a sudden those kernel message appear in /var/log/messages.

Does anybody knows why this queueing is occurring and how to fix it.

Regards

2

Hi @sajiby3k did you check journalctl output?

3

@Dmitry

Hi,

I can see the firewall logs in

dmesg
[ 405.171284] igb 0000:02:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
0 PREC=0x00 TTL=128 ID=58138 PROTO=UDP SPT=138 DPT=138 LEN=194
[ 737.053899] [A-Guest-Local-default-D]IN=br175 OUT= PHYSIN=eth1 MAC=00:1a:8c:40:02:cd:84:a9:3e:49:4c:f7:08:00 SRC=172.29.175.102 DST=172.29.165.1 LEN=60 TOS=0x00 PREC=0x00 TTL=128 ID=47008 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=30

In journalctl - I see this -

journalctl | grep A-Guest-Local-default-D
Mar 10 19:17:44 rqvyrtrx firewall-cfg[1424]: Running: iptables -t filter -A A-Guest-Local -m comment --comment “A-Guest-Local-10000 default-action drop” -j LOG --log-prefix “[A-Guest-Local-default-D]”

But those dmesg logs never got written to /var/log/messages.

Now if i do a - systemctl restart rsyslog, out of a sudden those logs appear in /var/log/messages.

I have below in vyos configuration -

set system syslog global facility all level ‘debug’
set system syslog global facility protocols level ‘debug’

Can you help me what I need to look for in “journalctl” that logging is not happening?

Let me know if you need the whole jpurnalctl file?

4

I need to know how I can reproduce this locally. Do you have any ideas?

5

Hi,

@Dmitry

Now comes the strange part if I try the exact same configuration in a vyos VM everything works fine.

But when I install vyos in real physical hardware the logging problem arises.

If you are interested in getting access to the physical box, let me know how you want to access it.

I hope I can give you access to the physical box.

6

Hi, I think I don’t need access to the physical box. Check please HDD or other storage on this box.
And you can try run ping flood and enable firewall logging on the VM.

7

@Dmitry

This is likely related to ⚓ T1938 syslog doesn't start automatically that I just reopened.

I think it has something to do with an ordering problem. I only noticed it after a reboot when my Grafana firewall log pipeline was blank. Restarting the rsyslogd service is the quick fix.

8

Hi @kroy

Thanks for opening an issue.

Restarting the rsyslog only shows previous buffered firewall logs. After syslog restart, new log messages are buffered again. To get new logs I need to restart syslog again.

Just to let you guys know. And strange is, it happens on physical vyos box, not when vyos is running in a VM.

9

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.