[Solved] CentOS8Stream: A start job is running for Flush Journal to Persistent Storage delays boot for 1m30s

[AndrewG10i]

Hello everyone,

I have noticed on all my CentOS8 Stream deployments that once I update CentOS8 Stream to the latest version (via "dnf update") - it starts to boot much slower, and particular reason that it waits 1min 30sec with the following message:

A start job is running for Flush Journal to Persistent Storage (XXs / 1min 30s)

img1.jpg (17.74 KiB) Viewed 7112 times

This started to come with updates during last ~2-5months.
Any suggestions how to get this resolved?

Thanks in advance!

[TrevorH]

Persistent journal is not the default on CentOS 8. Did you enable it by creating a /var/log/journal directory?

[AndrewG10i]

Persistent journal is not the default on CentOS 8. Did you enable it by creating a /var/log/journal directory?

Hello and thanks for replying! Nope, I didn't create it manually on my own and I am not using it directly. All those mentioned deployments - are usually "Minimal install" deployments.

Could it be created by deploying some software as dependency? Is it a way to know which software uses it or I may simply disable it?

EDIT1:
I have checked and confirmed that there is no "/var/log/journal" directory was present on the machine where this delay on boot occurs. So I have created this folder, and you know what - it started booting normally, without delay. Also noticed that files listed below appeared in the folder after boot:

img2.jpg (17.61 KiB) Viewed 7063 times

So basically this resolves my initial question. TrevorH, thank you so much for hint with the folder! Just curious do I need to leave it as is or I may disable this service (and if disable - then how)? ...I am not really aware what this service is about...

[jlehtone]

The systemd's journald is a replacement for rsyslogd. Read man systemd-journald.service

The "flush" is actually called by helper service systemd-journal-flush.service,
but that should do something only when /var/log/journal exists.

If you did not have that directory, then the service unit should have completed immediately
due to condition.

What do you get with

Code: Select all

systemctl status systemd-journal-flush.service

[AndrewG10i]

...
What do you get with

Code: Select all

systemctl status systemd-journal-flush.service

So it starts looking quite interesting: as I mentioned - I have several CentOS8 dev VMs (all "Minimal install") so I took another one which was last updated in Feb 2023:

img3_beforeUpdate.jpg (234.34 KiB) Viewed 7030 times

And I confirmed that:

• no "/var/log/journal" directory is present
• "systemctl status systemd-journal-flush.service" shows that all good
• and OS boots normally (without delay)

Then I performed "dnf update" command and you know what - the issue came as well: it started showing that message and delay boot for 1m30s.
Also "systemctl status systemd-journal-flush.service" now gives error:

img4_afterUpdate.jpg (77.1 KiB) Viewed 7030 times

But once I created "/var/log/journal" directory - that resolved the issue. Seems like some issue happens with the "systemd-journal-flush.service" during/after upgrade.

[TrevorH]

CentOS 8 ceased to exist at the end of 2021 as Red Hat killed it. If you still have CentSO Linux 8 instances running then you should migrate them to something else that is still supported. If you are using CentOS Stream 8 then be aware that it is a permanent beta product that often breaks.

For information on filing bugs against CentOS Linux or CentOS Stream please see https://wiki.centos.org/ReportBugs

[AndrewG10i]

CentOS 8 ceased to exist at the end of 2021 as Red Hat killed it. If you still have CentSO Linux 8 instances running then you should migrate them to something else that is still supported. If you are using CentOS Stream 8 then be aware that it is a permanent beta product that often breaks.

For information on filing bugs against CentOS Linux or CentOS Stream please see https://wiki.centos.org/ReportBugs

Yeah, I do remember that unfortunate fact , thanks for reminding. CentOS mostly used in dev environment, while still deciding which distro to switch to...

[jlehtone]

The latest el8 systemd is version 239-68.el8_7.4.
The latest stream8 systemd is version 239-73.el8, dated 2023-03-02.

If a service behaves differently, then something is different.
You can look at the actual definition of the unit:

Code: Select all

cat /usr/lib/systemd/system/systemd-journal-flush.service

If you have both pre- and post-update versions, you can tell whether the file has changed in the update.
If yes, how? If not, then something else in systemd is the reason.


PS. You are posting screenshots. Bitmap images. Can't you copy-paste text? It is easier to read and edit.

[AndrewG10i]

@jlehtone, many thanks for suggestion and your time! Anyway, considering: 1) reminder from @TrevorH that CentOS is dead; 2) also that immediate annoying issue was workaround-ed; 3) and that this case actually triggered me to start switching to Rocky Linux (already downloading distros to start working with them!) - I think we may stop spending our time on investigating this issue with CentOS further! Thank you all again for quick response!

P.S. Re posting images - yeah, sorry about that , just wanted to make it more visually clear and show the original output.

[BShT]

if you have a good backup, there are scripts that convert CentOS 8 to other distros without the need for a clean install.

I've done this with many VMs migrating to Oracle Linux.

The only tip is to configure the repos to vault and update CentOS to the limit before migrating.