CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

General support questions
digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/28 08:33:38

Hi,

Following your advice I have created a tmpfs storage as a ramdisk, with sufficiend space, mounted at /var and redirected my application to use that storage, but I am still getting the soft lockup errors.

Any ideas?

Thank you in advance
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/28 09:37:30

What is the output of the sar command (previous posts)? Can you provide the output of top ?

digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/28 13:40:36

Hi,

Here is the output of 'sar'

Code: Select all

[root@localhost ~]# sar
Linux 3.10.0-327.36.2.el7.x86_64 (localhost.localdomain)        11/28/2016      _x86_64_        (8 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all      4.41      0.00      4.13     19.49      0.00     71.97
12:20:01 AM     all      4.76      0.00      4.80     25.06      0.00     65.38
12:30:01 AM     all      4.18      0.00      4.20     12.18      0.00     79.44
12:40:01 AM     all      5.13      0.00      5.73     29.25      0.00     59.89
12:50:02 AM     all      5.02      0.00      5.74     22.64      0.00     66.60
01:00:01 AM     all      4.52      0.00      5.28     25.57      0.00     64.63
01:10:01 AM     all      4.82      0.00      5.11     25.32      0.00     64.75
01:20:01 AM     all      4.15      0.00      4.14     18.39      0.00     73.31
01:30:04 AM     all      3.93      0.00      4.01     19.57      0.00     72.49
01:40:02 AM     all      4.50      0.00      4.76     20.31      0.00     70.43
01:50:01 AM     all      4.97      0.00      5.60     36.62      0.00     52.80
02:00:01 AM     all      3.52      0.00      3.68     12.65      0.00     80.15
02:10:01 AM     all      4.30      0.00      4.29     28.63      0.00     62.78
02:20:02 AM     all      4.93      0.00      5.99     26.40      0.00     62.68
02:30:01 AM     all      4.50      0.00      4.39     21.52      0.00     69.59
02:40:03 AM     all      4.72      0.00      5.49     23.00      0.00     66.79
02:50:01 AM     all      4.58      0.00      4.93     20.83      0.00     69.67
03:00:01 AM     all      4.16      0.00      4.44     16.10      0.00     75.29
03:10:01 AM     all      4.05      0.00      4.00     20.06      0.00     71.89
03:20:01 AM     all      4.58      0.00      5.16     26.44      0.00     63.82
03:30:02 AM     all      4.50      0.00      4.82     25.18      0.00     65.50
03:40:02 AM     all      4.06      0.00      3.93     32.84      0.00     59.17

03:40:02 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:50:01 AM     all      3.55      0.00      3.61     13.33      0.00     79.51
04:00:10 AM     all      4.53      0.00      5.27     23.44      0.00     66.76
04:10:01 AM     all      4.66      0.00      4.86     27.88      0.00     62.60
04:20:01 AM     all      4.43      0.00      4.91     17.62      0.00     73.04
04:30:02 AM     all      4.46      0.00      4.82     20.88      0.00     69.85
04:40:03 AM     all      5.16      0.00      5.83     24.01      0.00     64.99
04:50:09 AM     all      4.62      0.00      5.02     23.45      0.00     66.91
05:00:01 AM     all      3.39      0.00      3.05     18.12      0.00     75.44
05:10:01 AM     all      4.28      0.00      4.25     25.93      0.00     65.54
05:20:01 AM     all      4.24      0.00      4.11     19.23      0.00     72.41
05:30:03 AM     all      3.86      0.00      4.24     21.40      0.00     70.50
05:40:05 AM     all      3.44      0.00      3.30     23.19      0.00     70.06
05:50:01 AM     all      3.95      0.00      3.37     21.47      0.00     71.22
06:00:03 AM     all      4.67      0.00      5.14     21.25      0.00     68.94
06:10:01 AM     all      4.57      0.00      4.73     25.41      0.00     65.29
06:20:03 AM     all      4.65      0.00      4.77     21.53      0.00     69.05
06:30:06 AM     all      4.46      0.00      4.96     20.19      0.00     70.39
06:40:02 AM     all      5.12      0.00      5.82     26.47      0.00     62.59
06:50:03 AM     all      3.76      0.00      3.67     20.75      0.00     71.82
07:00:02 AM     all      4.77      0.00      5.57     21.55      0.00     68.12
07:10:01 AM     all      4.94      0.00      5.03     22.61      0.00     67.41
07:20:01 AM     all      4.47      0.00      4.85     25.05      0.00     65.63

07:20:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:30:01 AM     all      3.74      0.00      3.51     16.72      0.00     76.02
07:40:02 AM     all      4.90      0.00      5.56     24.30      0.00     65.24
07:50:01 AM     all      4.87      0.00      5.16     22.02      0.00     67.95
08:00:03 AM     all      5.30      0.00      6.72     25.41      0.00     62.57
08:10:03 AM     all      5.68      0.00      5.42     23.57      0.00     65.34
08:20:01 AM     all      5.37      0.00      4.48     17.48      0.00     72.67
08:30:01 AM     all      5.42      0.00      4.72     20.06      0.00     69.79
08:40:04 AM     all      4.74      0.00      4.31     15.50      0.00     75.45
Average:        all      4.47      0.00      4.67     22.06      0.00     68.81

08:45:15 AM       LINUX RESTART

08:50:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:00:02 AM     all      3.83      0.00      3.02      8.88      0.00     84.27
09:10:01 AM     all      4.25      0.00      3.47     10.83      0.00     81.46
09:20:02 AM     all      4.71      0.00      4.03     14.67      0.00     76.59
09:30:01 AM     all      4.85      0.00      4.45     19.01      0.00     71.68
09:40:01 AM     all      5.12      0.00      4.81     21.01      0.00     69.07
09:50:01 AM     all      5.83      0.00      5.54     19.78      0.00     68.85
10:00:02 AM     all      4.78      0.00      4.64     24.67      0.00     65.91
10:10:02 AM     all      5.34      0.00      5.47     42.44      0.00     46.75
10:20:01 AM     all      4.46      0.00      5.07     54.95      0.00     35.51
10:30:01 AM     all      5.46      0.00      5.94     26.65      0.00     61.95
10:40:01 AM     all      4.53      0.00      4.66     24.60      0.00     66.21
10:50:01 AM     all      5.62      0.00      5.52     25.20      0.00     63.66
11:00:01 AM     all      4.10      0.00      3.33     21.85      0.00     70.71
11:10:02 AM     all      5.72      0.00      4.88     24.13      0.00     65.28
11:20:01 AM     all      7.75      0.00      6.29     23.77      0.00     62.19
11:30:01 AM     all      6.46      0.00      5.37     19.03      0.00     69.14
11:40:02 AM     all      5.28      0.00      5.11     25.98      0.00     63.62
11:50:02 AM     all      5.00      0.00      5.97     21.54      0.00     67.48
12:00:02 PM     all      4.54      0.00      4.53     20.34      0.00     70.59
12:10:01 PM     all      4.58      0.00      4.75     23.46      0.00     67.20
12:20:01 PM     all      3.83      0.00      3.21     21.21      0.00     71.75
12:30:01 PM     all      5.36      0.00      5.35     23.72      0.00     65.57

12:30:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:40:03 PM     all      4.40      0.00      4.59     21.24      0.00     69.77
12:50:01 PM     all      4.88      0.00      5.64     24.27      0.00     65.21
01:00:01 PM     all      5.07      0.00      5.60     27.54      0.00     61.80
01:10:01 PM     all      4.68      0.00      5.32     25.21      0.00     64.79
01:20:01 PM     all      4.75      0.00      5.01     19.32      0.00     70.92
01:30:01 PM     all      3.99      0.00      4.14     18.43      0.00     73.44
01:40:01 PM     all      3.36      0.00      3.14     17.94      0.00     75.56
01:50:01 PM     all      4.42      0.00      4.58     17.77      0.00     73.24
02:00:01 PM     all      4.78      0.00      5.22     20.26      0.00     69.75
02:10:01 PM     all      4.31      0.00      4.19     22.17      0.00     69.33
02:20:02 PM     all      3.88      0.00      3.63     15.57      0.00     76.92
02:30:04 PM     all      4.45      0.00      4.17     13.03      0.00     78.34
02:40:02 PM     all      6.02      0.00      5.27     21.05      0.00     67.66
02:50:01 PM     all      6.69      0.00      6.09     26.99      0.00     60.23
03:00:02 PM     all      6.35      0.00      5.97     26.56      0.00     61.11
03:10:02 PM     all      5.60      0.00      5.04     23.01      0.00     66.34
03:20:01 PM     all      4.94      0.00      5.76     28.49      0.00     60.82
03:30:05 PM     all      4.12      0.00      4.15     16.85      0.00     74.89
Average:        all      4.89      0.00      4.74     22.18      0.00     68.18
and the output of 'top'

Code: Select all

[root@localhost ~]# top
top - 15:39:29 up  6:54,  1 user,  load average: 21.54, 17.72, 19.50
Tasks: 236 total,  13 running, 223 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.8 us,  3.8 sy,  0.0 ni, 68.4 id, 22.0 wa,  0.0 hi,  0.9 si,  0.0 st
KiB Mem : 16268088 total, 14379480 free,   813588 used,  1075020 buff/cache
KiB Swap:  8257532 total,  8257532 free,        0 used. 15154296 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   39 root      20   0       0      0      0 R 100.0  0.0 103:21.17 ksoftirqd/3
26821 apache    20   0  625492  25528   5648 R  97.4  0.2   0:36.37 httpd
26456 apache    20   0  627732  27784   5648 R  94.7  0.2   0:25.78 httpd
  479 apache    20   0  627568  26144   4340 R  81.6  0.2   0:02.00 httpd
19491 apache    20   0  627796  27916   5712 R  81.6  0.2   0:14.07 httpd
29567 root      20   0  157680   2164   1504 R  44.7  0.0   0:00.29 top
  469 root      20   0       0      0      0 S  18.4  0.0   4:44.20 xfsaild/dm-0
 2377 ajaxterm  20   0  181252   8792   1392 S   2.6  0.1   1:30.68 python
    1 root      20   0   43084   5572   2404 S   0.0  0.0   0:54.29 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.36 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:37.92 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S   0.0  0.0   0:29.96 migration/0
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0
   10 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1
   11 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2
Thanks
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/28 17:58:29

What worries me is the fact that you have 8 cores, no more than 4-5 used at 100% and still the load average is very high.

Can you provide the output from

Code: Select all

vmstat 2 10
The output can be a little bit longer.

Edit: Check this thread and if possible try his solution, it might help.
Link:Ubuntu over VmWare

Note: Replace step 6 with:

Code: Select all

grub2-mkconfig -o /etc/grub2.cfg
or

Code: Select all

grub2-mkconfig -o /etc/grub2-efi.cfg
Edit: changed from step 5 to step 6
Last edited by hunter86_bg on 2016/11/29 14:35:08, edited 1 time in total.

digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/11/29 07:06:35

Hi,

here is the output of vmstat

Code: Select all

[root@localhost ~]# vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6  4      0 13579408    948 1636440    0    0     3   227   49    4  5  5 69 22  0
 4  5      0 13533668    948 1650132    0    0    30  1038 3793 7606 14 15 39 32  0
 4  3      0 13580948    948 1643932    0    0     0  2816  980 1243  2  1 62 35  0
 3  3      0 13611832    948 1641224    0    0     0   350 1174 1480  5  1 50 44  0
 6  3      0 13612284    948 1640736    0    0     0   515  670  927  1  1 66 33  0
 6  2      0 13610932    948 1640608    0    0     0   408  604  800  2  1 69 28  0
 3  2      0 13615020    948 1635976    0    0     0   536  573  746  0  1 67 32  0
 3  4      0 13612168    948 1640512    0    0     0  3476  563  754  2  1 62 35  0
 6  3      0 13612020    948 1640532    0    0     0   330  471  583  1  1 61 38  0
 5  2      0 13612500    948 1640556    0    0     0   416  613  870  2  1 65 33  0
I will check out the thread you provided and will report back if the solution did the trick or not.

Thanks
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/11/29 14:33:36

Note that kernel "kernel-3.10.0-514.el7.x86_64" is available.You might consider updating it.

digea
Posts: 9
Joined: 2016/11/16 08:43:51

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by digea » 2016/12/06 12:00:16

Hello again,

I tried the solution you suggested here
hunter86_bg wrote:Edit: Check this thread and if possible try his solution, it might help.
Link:Ubuntu over VmWare

Note: Replace step 6 with:

Code: Select all

grub2-mkconfig -o /etc/grub2.cfg
or

Code: Select all

grub2-mkconfig -o /etc/grub2-efi.cfg
But unfortunately it didn't work.


Thanks in advance.
Stanislav Ermizidis
IT Administrator
DIGEA S.A
sermizidis@digea.gr

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by hunter86_bg » 2016/12/06 21:04:00

Did you update the kernel also?

While I was checking oVirt , I saw that the priority of the resources used by the VM can be modified.Is this possible in your Virtualization environment.
If so, try to increase that priority.

kludge
Posts: 5
Joined: 2017/03/06 18:58:38

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by kludge » 2017/03/06 19:03:40

We're seeing similar issues on Centos 7. No problem on 6.8, but upgrading to 7 with the 3.10.0-514.6.1.el7.x86_64 kernel and nvidia driver 375.39 gives us seemingly identical behaviour, running directly on real hardware. I think this is a new kernel bug, or else something related to an nvidia driver bug. When this is going on, the CPU used by the migration task goes up to 100%.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS 7 - kernel: BUG: soft lockup - CPU#3 stuck for 23s! [rcuos/1:19]

Post by TrevorH » 2017/03/06 21:00:23

This problem usually manifests itself on disks. To check what you're waiting for you need to catch it waiting then use the various tools to discover what it's waiting on.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply