hung_task_timeout_secs and blocked for more than 120 seconds

[supmethods]

I am getting the error message 'hung_task_timeout_secs' and 'blocked for more than 120 seconds'. So the process systemd-sleep is stuck in the D state which means it's waiting for some resource. Does this mean another process is utilise that resource and it is unavailable for the systemd-sleep?

Also I noticed the process systemd-sleep is running. Is this process triggered by someone pressing the power or sleep/hibernation configuration for inactivity? The server never went to sleep in the past but it appeared to be sleeping on this occasion. I looked at the server logs and I noticed alternating PSU failures: PSU1 fails and comes back online, then PSU2 fails and comes back online. Failure of the PSUs happened two times. Could this be the reason why the systemd-sleep process is being triggered?

I had to do a hard reset when the hung_task_timeout_secs messages appeared as the system was not responsive. It means I've lost valuable information on the reason why the system hung. Is there a way to capture this information in the future if it does happen again?

[jmadeira]

We are having a similar issue with Almalinux 8.6. The servers were recently upgraded to Kernel version 4.18.0-372.19.1.el8_6.x86_64. One thing to note is are running Trend Micro on the servers and Trend does not support the latest kernel version yet so we are rolling back to version 4.18.0-372.16.1.el8_6.x86_64 to see if that helps with the issue.

[TrevorH]

This problem was posted in the CentOS 7 section and you are reporting a similar problem on el8. It's unlikely to be the same problem.

When you get this error, it's because your disk subsystem isn't responding in time to i/o requests - it's taken more than 2 minutes so far and this is just wrong and indicative of a more serious problem with your i/o subsystem.

[supmethods]

In this case, systemd-sleep is waiting for I/O and have not been allocated resource for over 2 mins. Does this mean that something is taking up all the resources and preventing systemd-sleep from accessing the disk? Could it be hardware failure or inadequate power as the PSUs were failing intermittently?

Also why is systemd-sleep process running? Is this process triggered by someone pressing the power or sleep/hibernation configuration for inactivity?

[jmadeira]

To provide some context, we have several hundred virtual linux servers running on a cluster backed by the same storage system. Only servers that received the latest yum updates were effected. We believe there is a correlation between our server Anti-virus solution and one of the new packages installed. We temporarily disabled the AV services on a test system and so far the symptoms have not re-appeared. We engaged our AV vendor and there engineers are investigating.

[supmethods]

There's no anti-virus on the system and no updates were installed either. It just randomly happened and hasn't reappeared after the reboot.

[TrevorH]

Are you on CentOS 7 or is this whole thread in the wrong section?

[supmethods]

It is CentOS 7, I am just explaining that it's not an anti-virus issue as there is no anti-virus installed.