Global rsyslog in rolling release keeps failing after a while

Bugs
,
1

In the latest rolling release, the global channel (/var/log/messages) will stop working after a while.
systemctl status rsyslog.service will show this error: gateway rsyslogd[380789]: file size limit cmd for file '/var/log/messages' did no resolve situation [v8.2302.0]. I believe this is due to the commit: syslog: T2778: migrate to get_config_dict() · vyos/vyos-1x@1dc79ce · GitHub especially at this line: https://github.com/vyos/vyos-1x/blob/4056974666cd1368c500a0afb75b3b8b1417ad89/data/templates/rsyslog/rsyslog.conf.j2#L14

The global channel expect a logrotate rule at /etc/logrotate.d/vyos-rsyslog but that file is currently an empty file with a line of comment which cause rsyslog just stop working because it doesn’t know how to rotate that file.

2

I have also noticed that rsyslog.service won’t restart if /var/log/messages is too large.

3

Will be fixed in any rolling release starting 2023-07-17

1 Like
4

thank you for the quick fix!

5

@c-po after the latest change, I am getting the following:

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/system-syslog.py", line 131, in <module>
    c = get_config()
        ^^^^^^^^^^^^
  File "/usr/libexec/vyos/conf_mode/system-syslog.py", line 87, in get_config
    syslog[tmp_config][tmp]['facility'][facility])
    ~~~~~~^^^^^^^^^^^^
TypeError: unhashable type: 'dict'

This happens as soon as I try to commit the following:

set system syslog host *redacted ip* facility kern level 'warning'
set system syslog host *redacted ip* protocol 'tcp'
set system syslog host *redacted ip* port '6001'
set system syslog host *redacted ip* format 'octet-counted'

Is there something that I’m missing somehow?

6

That is now fixed in current, and will be in the next nightly build. Thanks for the details, @bjw-s !

1 Like
7

@c-po @jestabro Sadly the issue of logrotate is not fixed.
The error message of rsyslog.service shows:

Jul 18 08:48:51 gateway rsyslogd[6560]: program '/usr/sbin/logrotate' (pid 6911) exited with status 1 [v8.2302.0]
Jul 18 08:48:51 gateway rsyslogd[6560]: file size limit cmd for file '/var/log/messages' did no resolve situation  [v8.2302.0]

For some reason the generated /etc/rsyslog.d/00-vyos.conf is this:

# We always log to /var/log/messages
$outchannel global,/var/log/messages,262144,/usr/sbin/logrotate
*.info;local7.debug :omfile:$global

instead of this like it used to be:

# We always log to /var/log/messages
$outchannel global,/var/log/messages,262144,/usr/sbin/logrotate /etc/logrotate.d/vyos-rsyslog
*.info;local7.debug :omfile:$global

To reproduce this easily, I think you should just make your most active firewall rule like WAN outgoing to enable-default-log and this issue will immediately happen after the /var/log/messages hit 256KB

8

Thanks @budimanjojo , I will take a closer look …

9

Thanks again, @budimanjojo , as you pointed out, there was a missing argument in the revised template file; that fix has been pushed to current. Note that this should resolve the logrotate issue, however one will see an unrelated, non-fatal, error message in journald regarding task ⚓ T5366 syslog: remove outdated reference to sysvinit rsyslog file in logrotate template. The latter is a trivial fix, pending some review.

1 Like