Syslog sending just stops randomly

General questions
1

I have a syslog collector for all of my systems. Vyos is setup to send all logs there using the “system syslog” configuration commands. At first it works great. Randomly, maybe every 8 hours or so, it will just stop sending. Nothing else is wrong with the system. I can still see all the logs on the local system. They’re just not sent. If I do “systemctl restart rsyslog”, they immediately start again. Here is my storage systems graph of logs from my vyos box:

vyos-syslog
vyos-syslog3128×414 36.8 KB

Notice at about 9:30 (end of the graph), they just stop. I can find nothing in any log regarding the stoppage.

Any ideas?

$ show version
Version:          VyOS 1.4-rolling-202301192311
Release train:    current

Built by:         <myemail>
Built on:         Thu 19 Jan 2023 23:11 UTC
Build UUID:       d24abd58-1d1d-4405-9a5b-965de4453617
Build commit ID:  35c958cbde618b

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Protectli
Hardware model:   FW4B
Hardware S/N:     123456789
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors
2

What do logs show at this time?

3

Unfortunately my logs since the last stoppage have rotated out. But, the last log that was sent was this:

[1095072.069310] [WAN-LOCAL-default-D]IN=eth0 OUT= MAC=<macids> SRC=<remoteip> DST=<myip> LEN=40 TOS=0x08 PREC=0x40 TTL=48 ID=12379 PROTO=TCP SPT=12707 DPT=23 WINDOW=9139 RES=0x00 SYN URGP=0

It was just a normal log flow up until then.
At the time I looked at the logs in the file on the host and there was nothing about rsyslog or any other logging issue. Just continued firewall logs and a few other about automated jobs that run regularly on my system.

Then, I did a restart of rsyslog, and this was the first message sent again:

Stopping System Logging Service...

So it kicked in as soon as I started the restart. Even before restart finished. A few lines later I received the message that rsyslog was restarted. Everything then resumed as normal.

In this case it was 30 minutes between stop and start. But that’s only because I restarted it. If I had not, it would have gone on for hours with no logs being sent.