hung_task_timeout_secs and blocked for more than 120 seconds

General support questions
Post Reply
supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

hung_task_timeout_secs and blocked for more than 120 seconds

Post by supmethods » 2022/08/08 02:37:15

I am getting the error message 'hung_task_timeout_secs' and 'blocked for more than 120 seconds'. So the process systemd-sleep is stuck in the D state which means it's waiting for some resource. Does this mean another process is utilise that resource and it is unavailable for the systemd-sleep?

Also I noticed the process systemd-sleep is running. Is this process triggered by someone pressing the power or sleep/hibernation configuration for inactivity? The server never went to sleep in the past but it appeared to be sleeping on this occasion. I looked at the server logs and I noticed alternating PSU failures: PSU1 fails and comes back online, then PSU2 fails and comes back online. Failure of the PSUs happened two times. Could this be the reason why the systemd-sleep process is being triggered?

I had to do a hard reset when the hung_task_timeout_secs messages appeared as the system was not responsive. It means I've lost valuable information on the reason why the system hung. Is there a way to capture this information in the future if it does happen again?
Attachments
hung_tasks.gif
hung_tasks.gif (162.36 KiB) Viewed 3935 times

jmadeira
Posts: 2
Joined: 2022/08/15 13:58:19

Re: hung_task_timeout_secs and blocked for more than 120 seconds

Post by jmadeira » 2022/08/15 14:16:56

We are having a similar issue with Almalinux 8.6. The servers were recently upgraded to Kernel version 4.18.0-372.19.1.el8_6.x86_64. One thing to note is are running Trend Micro on the servers and Trend does not support the latest kernel version yet so we are rolling back to version 4.18.0-372.16.1.el8_6.x86_64 to see if that helps with the issue.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: hung_task_timeout_secs and blocked for more than 120 seconds

Post by TrevorH » 2022/08/15 14:25:51

This problem was posted in the CentOS 7 section and you are reporting a similar problem on el8. It's unlikely to be the same problem.

When you get this error, it's because your disk subsystem isn't responding in time to i/o requests - it's taken more than 2 minutes so far and this is just wrong and indicative of a more serious problem with your i/o subsystem.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Re: hung_task_timeout_secs and blocked for more than 120 seconds

Post by supmethods » 2022/08/16 10:32:35

In this case, systemd-sleep is waiting for I/O and have not been allocated resource for over 2 mins. Does this mean that something is taking up all the resources and preventing systemd-sleep from accessing the disk? Could it be hardware failure or inadequate power as the PSUs were failing intermittently?

Also why is systemd-sleep process running? Is this process triggered by someone pressing the power or sleep/hibernation configuration for inactivity?

jmadeira
Posts: 2
Joined: 2022/08/15 13:58:19

Re: hung_task_timeout_secs and blocked for more than 120 seconds

Post by jmadeira » 2022/08/16 14:30:08

To provide some context, we have several hundred virtual linux servers running on a cluster backed by the same storage system. Only servers that received the latest yum updates were effected. We believe there is a correlation between our server Anti-virus solution and one of the new packages installed. We temporarily disabled the AV services on a test system and so far the symptoms have not re-appeared. We engaged our AV vendor and there engineers are investigating.

supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Re: hung_task_timeout_secs and blocked for more than 120 seconds

Post by supmethods » 2022/08/24 14:15:24

There's no anti-virus on the system and no updates were installed either. It just randomly happened and hasn't reappeared after the reboot.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: hung_task_timeout_secs and blocked for more than 120 seconds

Post by TrevorH » 2022/08/24 15:00:04

Are you on CentOS 7 or is this whole thread in the wrong section?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

supmethods
Posts: 55
Joined: 2011/07/15 08:24:11

Re: hung_task_timeout_secs and blocked for more than 120 seconds

Post by supmethods » 2022/09/19 01:46:57

It is CentOS 7, I am just explaining that it's not an anti-virus issue as there is no anti-virus installed.

Post Reply