I have a syslog collector for all of my systems. Vyos is setup to send all logs there using the “system syslog” configuration commands. At first it works great. Randomly, maybe every 8 hours or so, it will just stop sending. Nothing else is wrong with the system. I can still see all the logs on the local system. They’re just not sent. If I do “systemctl restart rsyslog”, they immediately start again. Here is my storage systems graph of logs from my vyos box:
Notice at about 9:30 (end of the graph), they just stop. I can find nothing in any log regarding the stoppage.
$ show version
Version: VyOS 1.4-rolling-202301192311
Release train: current
Built by: <myemail>
Built on: Thu 19 Jan 2023 23:11 UTC
Build UUID: d24abd58-1d1d-4405-9a5b-965de4453617
Build commit ID: 35c958cbde618b
Boot via: installed image
System type: bare metal
Hardware vendor: Protectli
Hardware model: FW4B
Hardware S/N: 123456789
Hardware UUID: Unknown
Copyright: VyOS maintainers and contributors
What do logs show at this time?
Unfortunately my logs since the last stoppage have rotated out. But, the last log that was sent was this:
[1095072.069310] [WAN-LOCAL-default-D]IN=eth0 OUT= MAC=<macids> SRC=<remoteip> DST=<myip> LEN=40 TOS=0x08 PREC=0x40 TTL=48 ID=12379 PROTO=TCP SPT=12707 DPT=23 WINDOW=9139 RES=0x00 SYN URGP=0
It was just a normal log flow up until then.
At the time I looked at the logs in the file on the host and there was nothing about rsyslog or any other logging issue. Just continued firewall logs and a few other about automated jobs that run regularly on my system.
Then, I did a restart of rsyslog, and this was the first message sent again:
Stopping System Logging Service...
So it kicked in as soon as I started the restart. Even before restart finished. A few lines later I received the message that rsyslog was restarted. Everything then resumed as normal.
In this case it was 30 minutes between stop and start. But that’s only because I restarted it. If I had not, it would have gone on for hours with no logs being sent.