rsyslog.service: Main process exited, code=exited, status=1/FAILURE

[gkeishin]

On latest yacto 2.6 build
root@witherspoon:~# cat /etc/os-release
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="v2.4-247"
VERSION_ID="v2.4-247-g1a4b7ee"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) v2.4-247"
BUILD_ID="v2.4"

Remote logging server crashed out

Jan 09 07:42:07 witherspoon systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 670.
Jan 09 07:42:07 witherspoon systemd[1]: Stopped System Logging Service.
Jan 09 07:42:07 witherspoon systemd[1]: Starting System Logging Service...
Jan 09 07:42:07 witherspoon systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Jan 09 07:42:07 witherspoon systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Jan 09 07:42:07 witherspoon systemd[1]: Failed to start System Logging Service.
Jan 09 07:42:08 witherspoon systemd[1]: rsyslog.service: Service RestartSec=1s expired, scheduling restart.
Jan 09 07:42:08 witherspoon systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 671.
Jan 09 07:42:08 witherspoon systemd[1]: Stopped System Logging Service.
Jan 09 07:42:08 witherspoon systemd[1]: Starting System Logging Service...
Jan 09 07:42:08 witherspoon systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Jan 09 07:42:08 witherspoon systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Jan 09 07:42:08 witherspoon systemd[1]: Failed to start System Logging Service.

root@witherspoon:~# systemctl status rsyslog.service | cat
● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/rsyslog.service.d
           └─rsyslog-override.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2019-01-09 07:40:31 UTC; 246ms ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
  Process: 2487 ExecStart=/usr/sbin/rsyslogd -n -iNONE (code=exited, status=1/FAILURE)
 Main PID: 2487 (code=exited, status=1/FAILURE)
root@witherspoon:~#

[gkeishin]

This is flooding the journald , so we would need it to be cleaner sooner.

[dkodihal]

Found the problem. A behavioral change has been introduced in rsyslog, due to which we can no longer run it with "no actions enabled". This is how we used to run rsyslog before : it doesn't do anything when it starts up (no actions), but it can be configured (actions will be added) to perform remote logging. In previous revisions, you could start rsyslogd without any actions, in the recent revisions (that come with Yocto 2.6), you can no longer do that.

I think the solution is to start rsyslogd when remote logging is configured. I'm working on a fix.

This is the commit in rsyslog which changed this behavior :

rsyslog/rsyslog@09e3d06

[mdmillerii]

This is how we used to run rsyslog before : it doesn't do anything when it starts up (no actions), but it can be configured (actions will be added) to perform remote logging.

This seems counter to the justification in the upstream change log:

This patch make rsyslog properly terminate after the error message. It
is a change in behavior, but there really is no reason why a defunct
instance should be kept running.

If adding actions at runtime is in the upstream code, then we should submit a patch upstream to allow startng with no actions required. Keeping the other half of the error makes sense (aparently an action that is broken).

That being said, not starting a daemon that isn't doing useful work also sounds like a feature for resource consumption.

[dkodihal]

I think I didn't explain myself clearly in my last comment : we don't add actions at runtime, rather we update the rsyslog config file and restart the service. So the latest upstream seems to do the right thing : it prevents running an rsyslod instance that has no actions.

So we shouldn't be starting the rsyslogd daemon when we have no actions for the daemon.

[mdmillerii]

Ok. Be sure to consider removing the last action.

Maybe by having the config exist or not based on having an action and using ConditionPathExists?