kswapd0 using 100% of one CPU

General support questions
Post Reply
DrMikeDuke
Posts: 5
Joined: 2014/09/08 08:45:29

kswapd0 using 100% of one CPU

Post by DrMikeDuke » 2021/04/06 03:11:46

Hi all,

Hoping someone can assist me with understanding why kswapd0 is using 100% of one my CPUs
Additionally I have >90% memory in use (not cached?) despite not being able to identify any process(es) consuming this and low swap usage.
IOTOP doesn't show any heavy writing to disk and vmstat doesn't show swap budging + SI/SO numbers are 0.
Pretty sure I'm overlooking something, all of my other servers don't show this issue.

Worth mentioning it is sitting on HyperV, I don't believe memory ballooning is in use though but I do see messages like:
hv_balloon: Balloon request will be partially fulfilled. Balloon floor reached. - whatever that means

I have tried echo 1 | sudo tee /proc/sys/vm/drop_caches but it hasn't made any difference

Code: Select all

~]$ lsmod | grep -i hv_
hv_utils               25894  0
ptp                    23551  1 hv_utils
hv_balloon             22858  0
hv_netvsc              50802  0
hv_storvsc             22546  2
scsi_transport_fc      64007  1 hv_storvsc
hv_vmbus               97049  7 hv_balloon,hyperv_keyboard,hv_netvsc,hid_hyperv,hv_utils,hyperv_fb,hv_storvsc
Output of interest:

Code: Select all

Linux server 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.9.2009 (Core)
CPU Usage

Code: Select all

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   36 root      20   0       0      0      0 R 100.0  0.0   2429:01 kswapd0
29349 duke   20   0  161972   2124   1524 R   6.2  0.0   0:00.01 top
    1 root      20   0  128152   5528   3264 S   0.0  0.0   5:09.62 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:03.61 kthreadd
Memory[/b/]

Code: Select all

~]$ free -g
              total        used        free      shared  buff/cache   available
Mem:             18          17           1           0           0           1
Swap:             7           0           7

~] cat /proc/meminfo
MemTotal:       19909192 kB
MemFree:         1353144 kB
MemAvailable:    1350340 kB
Buffers:            4172 kB
Cached:           691180 kB
SwapCached:        17212 kB
Active:           594488 kB
Inactive:         384320 kB
Active(anon):     475396 kB
Inactive(anon):   258824 kB
Active(file):     119092 kB
Inactive(file):   125496 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       8060924 kB
SwapFree:        7847676 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:        277420 kB
Mapped:           102504 kB
Shmem:            450764 kB
Slab:              72160 kB
SReclaimable:      34140 kB
SUnreclaim:        38020 kB
KernelStack:        3072 kB
PageTables:        13740 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    18015520 kB
Committed_AS:    1907304 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      100956 kB
VmallocChunk:   34359633920 kB
Percpu:              528 kB
HardwareCorrupted:     0 kB
AnonHugePages:    137216 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      151488 kB
DirectMap2M:    20328448 kB

~]$ vmstat -S M 1 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  0    150   1470      4    826    0    0     1     3    5    2  1  3 96  0  0

~]$ vmstat -s
     19909192 K total memory
     17650488 K used memory
       597936 K active memory
       421612 K inactive memory
      1489336 K free memory
         4172 K buffer memory
       765196 K swap cache
      8060924 K total swap
       213248 K used swap
      7847676 K free swap
      3695681 non-nice user cpu ticks
        18005 nice user cpu ticks
     19217201 system cpu ticks
    654537823 idle cpu ticks
       136610 IO-wait cpu ticks
            0 IRQ cpu ticks
       544153 softirq cpu ticks
            0 stolen cpu ticks
      3605097 pages paged in
     21318908 pages paged out
       175936 pages swapped in
       205034 pages swapped out
   1623762510 interrupts
   4173318646 CPU context switches
   1614217579 boot time
       480427 forks

Code: Select all

~]$ ps aux --sort pmem
root       497  0.0  0.5 161844 106176 ?       Ss   Feb24   3:22 /usr/lib/systemd/systemd-journald
monitor  1022  7.5  0.4 1214956 87636 ?       Ssl  Feb24 4351:43 /usr/bin/monitor
root      1021  0.0  0.3 573196 62696 ?        Ssl  Feb24   5:07 /usr/sbin/rsyslogd -n
mysql     1529  0.0  0.3 1632036 79160 ?       Sl   Feb24  50:33 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
apache   17418  0.0  0.1 452504 20280 ?        S    16:35   0:02 /usr/sbin/httpd -DFOREGROUND
apache   15724  0.0  0.1 460704 28324 ?        S    Apr04   0:04 /usr/sbin/httpd -DFOREGROUND

~]$ top -b -o +RES | head -15 # Sorted by memory usage %
top - 22:07:09 up 35 days, 10 min,  1 user,  load average: 1.05, 1.06, 1.10
Tasks: 135 total,   2 running, 133 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 51.6 sy,  0.0 ni, 48.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 19909192 total,  1488924 free, 17650764 used,   769504 buff/cache
KiB Swap:  8060924 total,  7847676 free,   213248 used.  1526412 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  497 root      20   0  161844 106192 105952 S   0.0  0.5   3:22.10 systemd-journal
 1022 monitor  20   0 1214956  87636  12496 S   0.0  0.4   4351:58 monitor
 1529 mysql     20   0 1632036  79160   4412 S   0.0  0.4  50:33.66 mysqld
 1021 root      20   0  573196  62704  59532 S   0.0  0.3   5:07.77 rsyslogd
15724 apache    20   0  460704  28324   5980 S   0.0  0.1   0:04.49 httpd
17418 apache    20   0  452504  20280   5912 S   0.0  0.1   0:02.79 httpd
17415 apache    20   0  451960  19732   5896 S   0.0  0.1   0:03.46 httpd
31899 apache    20   0  451732  19528   5964 S   0.0  0.1   0:01.51 httpd

Any advice is much appreciated

Post Reply

Return to “CentOS 7 - General Support”