Killing process XYZ (systemctl) with signal SIGKILL.

[KernelOops]

Hey everyone,

My CentOS 8 servers show the following journal messages:

Code: Select all

systemd[1]: user@1000.service: Killing process 50292 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 50359 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 50424 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 50498 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 50566 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 50632 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 50712 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 50904 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51013 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51104 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51185 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51266 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51348 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51459 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51532 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51606 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51678 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 51744 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 52003 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 52078 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 52238 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 52401 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 52532 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 52598 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 53016 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 53083 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 53147 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 53234 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 53308 (systemctl) with signal SIGKILL.
systemd[1]: user@1000.service: Killing process 53394 (systemctl) with signal SIGKILL.

Anyone knows what they are all about? I am guessing that these are cron jobs being killed, but it makes no sense why systemd would kill normal cron jobs.

Any help would be appreciated. Thank you!

[LaurentBarme]

Hi,

I have the same issue that seems to be related to cron job but also to an login session.

Strangely it concerns only a virtual server.

Can these messages be safely ignored ?

Thanks for any clue...

[KernelOops]

Sorry, I still haven't found an explanation.

[LaurentBarme]

Me neither. Although I have found a pattern:
(there is a cron job scheduled at 16:02)

Code: Select all

3150 2020-11-04T16:02:01.779444+01:00 sdd systemd[1]: Created slice User Slice of UID 0.

3151 2020-11-04T16:02:01.779766+01:00 sdd systemd[1]: Started /run/user/0 mount wrapper.
3152 2020-11-04T16:02:01.779833+01:00 sdd systemd[1]: Starting User Manager for UID 0...
3153 2020-11-04T16:02:01.785426+01:00 sdd systemd[1]: Started Session 88 of user root.
3154 2020-11-04T16:02:01.891213+01:00 sdd systemd[46330]: Reached target Paths.
3155 2020-11-04T16:02:01.898475+01:00 sdd systemd[46330]: Starting D-Bus User Message Bus Socket.
3156 2020-11-04T16:02:01.898669+01:00 sdd systemd[46330]: Reached target Timers.
3157 2020-11-04T16:02:01.914632+01:00 sdd systemd[46330]: Listening on D-Bus User Message Bus Socket.
3158 2020-11-04T16:02:01.914794+01:00 sdd systemd[46330]: Reached target Sockets.
3159 2020-11-04T16:02:01.914852+01:00 sdd systemd[46330]: Reached target Basic System.
3160 2020-11-04T16:02:01.914989+01:00 sdd systemd[46330]: Reached target Default.
3161 2020-11-04T16:02:01.915065+01:00 sdd systemd[46330]: Startup finished in 45ms.
3162 2020-11-04T16:02:01.915163+01:00 sdd systemd[1]: Started User Manager for UID 0.
3163 2020-11-04T16:02:02.004123+01:00 sdd systemd[1]: user-runtime-dir@0.service: Unit not needed anymore. Stopping.
3164 2020-11-04T16:02:02.004366+01:00 sdd systemd[1]: Stopping User Manager for UID 0...
3165 2020-11-04T16:02:02.004433+01:00 sdd systemd[46330]: Stopped target Default.
3166 2020-11-04T16:02:02.004519+01:00 sdd systemd[46330]: Stopped target Basic System.
3167 2020-11-04T16:02:02.004587+01:00 sdd systemd[46330]: Stopped target Paths.
3168 2020-11-04T16:02:02.004648+01:00 sdd systemd[46330]: Stopped target Sockets.
3169 2020-11-04T16:02:02.004706+01:00 sdd systemd[46330]: Closed D-Bus User Message Bus Socket.
3170 2020-11-04T16:02:02.004784+01:00 sdd systemd[46330]: Stopped target Timers.
3171 2020-11-04T16:02:02.004847+01:00 sdd systemd[46330]: Reached target Shutdown.

3172 2020-11-04T16:02:02.004926+01:00 sdd systemd[46330]: Starting Exit the Session...
3173 2020-11-04T16:02:02.042748+01:00 sdd systemd[1]: user@0.service: Killing process 46346 (systemctl) with signal SIGKILL.

3174 2020-11-04T16:02:02.043059+01:00 sdd systemd[1]: user-runtime-dir@0.service: Unit not needed anymore. Stopping.
3175 2020-11-04T16:02:02.064204+01:00 sdd systemd[1]: Stopped User Manager for UID 0.
3176 2020-11-04T16:02:02.064351+01:00 sdd systemd[1]: user-runtime-dir@0.service: Unit not needed anymore. Stopping.
3177 2020-11-04T16:02:02.064394+01:00 sdd systemd[1]: Stopping /run/user/0 mount wrapper...
3178 2020-11-04T16:02:02.064432+01:00 sdd systemd[1]: Removed slice User Slice of UID 0.
3179 2020-11-04T16:02:02.103825+01:00 sdd systemd[1]: Stopped /run/user/0 mount wrapper.

It takes less than 0.32 s;
Beside, it happens only on a VPS with very little RAM (I have other servers on CentOS 8 without this issue).

The best explanation I had is that on a short cron job, there is a systemctl command that don't stop fast enough at the end of the session and thus is killed by systemd.

What does this systemctl do in this particular context remains a mystery.

[KernelOops]

I see similar results, for example here is an hourly cron job that runs /usr/sbin/aide:

Code: Select all

Nov 03 21:00:01 server systemd[1433]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Nov 03 21:00:01 server systemd[1433]: Reached target Timers.
Nov 03 21:00:01 server systemd[1433]: Starting D-Bus User Message Bus Socket.
Nov 03 21:00:01 server systemd[1433]: Reached target Paths.
Nov 03 21:00:01 server systemd[1433]: Listening on D-Bus User Message Bus Socket.
Nov 03 21:00:01 server systemd[1433]: Reached target Sockets.
Nov 03 21:00:01 server systemd[1433]: Reached target Basic System.
Nov 03 21:00:01 server systemd[1433]: Reached target Default.
Nov 03 21:00:01 server systemd[1433]: Startup finished in 121ms.
Nov 03 21:00:01 server CROND[1445]: (root) CMD (/usr/sbin/aide --check)
Nov 03 21:00:03 server systemd[1433]: Stopped target Default.
Nov 03 21:00:03 server systemd[1433]: Stopped target Basic System.
Nov 03 21:00:03 server systemd[1433]: Stopped target Timers.
Nov 03 21:00:03 server systemd[1433]: Stopped target Sockets.
Nov 03 21:00:03 server systemd[1433]: Closed D-Bus User Message Bus Socket.
Nov 03 21:00:03 server systemd[1433]: Stopped target Paths.
Nov 03 21:00:03 server systemd[1433]: Reached target Shutdown.
Nov 03 21:00:03 server systemd[1433]: Starting Exit the Session...
Nov 03 21:00:03 server systemd[1439]: pam_unix(systemd-user:session): session closed for user root
Nov 03 21:00:03 server systemd[1]: user@0.service: Killing process 1453 (systemctl) with signal SIGKILL.

Its still not clear to me, what process is being killed...

[KernelOops]

ok, I found the problem... apparently systemd kills cron jobs due to a PAM security restriction. I have no idea what that restriction is, but I remember something about it in the distant past... hmmm

At least I can ignore systemd since that is not the cause of the problem and focus on PAM.

[KernelOops]

Found the problem...

Interesting issue. The sever has various security restrictions, one of them, is to set various limits in /etc/login.defs, two of those limits caused all my problems. The account password expiration, caused some accounts to have an expired password, which in turn prevents them from using cron, thus all cron jobs are being started, but killed at the same time. Weird...

Anyway, once I found that out, it was easy to modify /etc/login.defs back to original values, then manually reset password expiration.

The values that caused my problems in /etc/login.defs, here they are in their default values:

Code: Select all

PASS_MAX_DAYS   99999
PASS_MIN_DAYS   0

Reset accounts with chage:

Code: Select all

chage --mindays 0 --maxdays 99999 someuser

Things are back to normal