no shutdown logs at /var/log/messages

Issues related to applications and software problems and general support
Post Reply
ocs
Posts: 4
Joined: 2020/11/17 07:27:13

no shutdown logs at /var/log/messages

Post by ocs » 2020/11/17 07:53:26

Hello everybody
I just realized that my new centos 8 server does not record anything at /var/log/messages when a reboot or halt command is executed (or any other form of shutdown)
I manually executed "logger mymessage" and then "reboot", after "mymessage" comes the typical "kernel: Command line: BOOT_IMAGE..."
I have a systemd script that runs before shutdown.target or reboot.target.
This scripts executes logger and ads a message to the system log
this script also creates a "log" file and sends it by email.
the file is created at shutdown. (It includes a "date" command)
I can see the message of my unit-file in the screen while the computer is shutting down.
So my custom script is executed at shutdown.
But:
the logger command does not write anything in the /var/log/messages file
The script contains the line "postfix flush" which should immediately send any email in queue, however the email is not send until the server is restarted.

Of course if I manually execute:
systemctl start run-before-shutdown.service
everything works fine (a message is recorded in /var/log/messages and the email is sent immediately)

Could anybody provide insights on what is happening and how to solve it ? (record shutdown messages by services in the syslog and send email)
thank you very much
ocs

User avatar
TrevorH
Forum Moderator
Posts: 29915
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: no shutdown logs at /var/log/messages

Post by TrevorH » 2020/11/17 09:21:53

Does your service depend on rsyslog?
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

ocs
Posts: 4
Joined: 2020/11/17 07:27:13

Re: no shutdown logs at /var/log/messages

Post by ocs » 2020/11/17 15:29:38

Dear TrevorH,
thank you for your interest.

rsyslog is enabled, but I have not changed anything in its configuration. I did not changed anything (that I know) when I installed the server nor I manually enabled the service.
Everything is prety much at default.

Acually I have never done anything with rsyslog, neither in CentOS8 nor in 7. On my CentOS7 servers I don't have this problem, when I look at /var/log/messages I can see the normal expected messages (example: "systemd: Stopping OpenSSH server daemon...") and the following command:

grep -l "systemd: Stopp" messages*
returns results in a CentOS7
while the same, returns nothing in a CentOS8

Just to check, I executed the following commands:
logger before
tail /var/log/messages

(it showed "before")
systemctl stop rsyslog
logger after
tail /var/log/messages

(it showed "before" but not "after")
systemctl start rsyslog
tail /var/log/messages

(now it shows both "before" and "after" )

So probably the thing is that rsyslog.service is stoped before my script or any other is executed and I guess that it is the same case with postfix.service. Due to the fact that postfix.service is stoped before my script is executed, my email with the shutdown notification is not sent.
So if things work more or less like it use to be with initscripts, there should be a way to set up the order in which the "stop" commands are issued when a shutdown is executed.
However I found it a bit odd (even weird) that rsyslog is stoped at the beginning so no message issued by the services that are shutting down are recorded in the /var/log/messages file. An event that does not happen in CentOS7.

here is my unit-file:
# cat run-before-shutdown.service
[Unit]
Description=Run my custom task at shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target

[Service]
Type=oneshot
ExecStart=/opt/SYSTEM/bin/shutdown_script.sh
TimeoutStartSec=0

[Install]
WantedBy=shutdown.target



thank you very much
OCS

User avatar
TrevorH
Forum Moderator
Posts: 29915
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: no shutdown logs at /var/log/messages

Post by TrevorH » 2020/11/17 16:40:11

If you don't order your unit to depend on rsyslog then rsyslog could easily be shutdown before you can send the messages to it. It's your unit file that needs to change to do that.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

ocs
Posts: 4
Joined: 2020/11/17 07:27:13

Re: no shutdown logs at /var/log/messages

Post by ocs » 2020/11/18 16:58:37

Again thank you very much Trevor.
I think I solved the problem and I'm very convinced that this is a bug in CentOS 8.

First let me clarify the motives of my posts here.
* 1st and most important:
When I look at /var/log/messages in a CentOS8 I see no trace of a shutdown, there is nothing. However there is a lot of shutdown messages in a CentOS7 machine. Right before the "kernel: Command line: BOOT_IMAGE=..." line

I believe that this is not correct.

I have compared the unit file for rsyslog in both CentOS7 and CentOS8 (/usr/lib/systemd/system/rsyslog.service) and I found the following line in CentOS8 which is not in CentOS7: (line 19)
LimitNOFILE=16384
So I commented it and executed a reboot.
Now there is plenty of messages from the daemons stoping at reboot which they were not there before. Including the one generated by my script

* My 2nd motive and not so important, was to create a script that send an email and records a message in the syslog when the computer shutsdown (and as extra, when it starts).
For that I generated the following unit file (I basically copied it from here https://unix.stackexchange.com/question ... -or-reboot ) :
[Unit]
Description=Sending Alert Emails on System shutdown.
Requires=network.target NetworkManager.service postfix.service rsyslog.service
;Wants=network.target network-online.target postfix.service rsyslog.service
After=network.target NetworkManager.service postfix.service rsyslog.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/opt/SYSTEM/bin/BootMail.sh start
ExecStop=/opt/SYSTEM/bin/BootMail.sh stop
KillMode=none
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target


It worked perfectly in CentOS7 but it did not in CentOS8 ( no message in /var/log/messages and the email was sent, but only when the computer was restarting, not at shutdown).
But, when I commented the line 19 (LimitNOFILE=16384) in /usr/lib/systemd/system/rsyslog.service everything worked perfectlly. I'm receiving the email at shutdown and later, after a minute, one at start.

So I really don't know what line 19 does, but for the comments in the file (Lines 17 and 18):
# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.

I guess that it is still a bit "experimental"

In my modest opinion this should be corrected.

But I'll be very thankful if someone else could check all of this in their CentOS 8 machines

Thank you very much

OCS

Post Reply

Return to “CentOS 8 - General Support”