Hoping someone can assist me with understanding why kswapd0 is using 100% of one my CPUs
Additionally I have >90% memory in use (not cached?) despite not being able to identify any process(es) consuming this and low swap usage.
IOTOP doesn't show any heavy writing to disk and vmstat doesn't show swap budging + SI/SO numbers are 0.
Pretty sure I'm overlooking something, all of my other servers don't show this issue.
Worth mentioning it is sitting on HyperV, I don't believe memory ballooning is in use though but I do see messages like:
hv_balloon: Balloon request will be partially fulfilled. Balloon floor reached. - whatever that means
I have tried echo 1 | sudo tee /proc/sys/vm/drop_caches but it hasn't made any difference
Code: Select all
~]$ lsmod | grep -i hv_
hv_utils 25894 0
ptp 23551 1 hv_utils
hv_balloon 22858 0
hv_netvsc 50802 0
hv_storvsc 22546 2
scsi_transport_fc 64007 1 hv_storvsc
hv_vmbus 97049 7 hv_balloon,hyperv_keyboard,hv_netvsc,hid_hyperv,hv_utils,hyperv_fb,hv_storvsc
Code: Select all
Linux server 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.9.2009 (Core)
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36 root 20 0 0 0 0 R 100.0 0.0 2429:01 kswapd0
29349 duke 20 0 161972 2124 1524 R 6.2 0.0 0:00.01 top
1 root 20 0 128152 5528 3264 S 0.0 0.0 5:09.62 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:03.61 kthreadd
Code: Select all
~]$ free -g
total used free shared buff/cache available
Mem: 18 17 1 0 0 1
Swap: 7 0 7
~] cat /proc/meminfo
MemTotal: 19909192 kB
MemFree: 1353144 kB
MemAvailable: 1350340 kB
Buffers: 4172 kB
Cached: 691180 kB
SwapCached: 17212 kB
Active: 594488 kB
Inactive: 384320 kB
Active(anon): 475396 kB
Inactive(anon): 258824 kB
Active(file): 119092 kB
Inactive(file): 125496 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8060924 kB
SwapFree: 7847676 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 277420 kB
Mapped: 102504 kB
Shmem: 450764 kB
Slab: 72160 kB
SReclaimable: 34140 kB
SUnreclaim: 38020 kB
KernelStack: 3072 kB
PageTables: 13740 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18015520 kB
Committed_AS: 1907304 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 100956 kB
VmallocChunk: 34359633920 kB
Percpu: 528 kB
HardwareCorrupted: 0 kB
AnonHugePages: 137216 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 151488 kB
DirectMap2M: 20328448 kB
~]$ vmstat -S M 1 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
5 0 150 1470 4 826 0 0 1 3 5 2 1 3 96 0 0
~]$ vmstat -s
19909192 K total memory
17650488 K used memory
597936 K active memory
421612 K inactive memory
1489336 K free memory
4172 K buffer memory
765196 K swap cache
8060924 K total swap
213248 K used swap
7847676 K free swap
3695681 non-nice user cpu ticks
18005 nice user cpu ticks
19217201 system cpu ticks
654537823 idle cpu ticks
136610 IO-wait cpu ticks
0 IRQ cpu ticks
544153 softirq cpu ticks
0 stolen cpu ticks
3605097 pages paged in
21318908 pages paged out
175936 pages swapped in
205034 pages swapped out
1623762510 interrupts
4173318646 CPU context switches
1614217579 boot time
480427 forks
Code: Select all
~]$ ps aux --sort pmem
root 497 0.0 0.5 161844 106176 ? Ss Feb24 3:22 /usr/lib/systemd/systemd-journald
monitor 1022 7.5 0.4 1214956 87636 ? Ssl Feb24 4351:43 /usr/bin/monitor
root 1021 0.0 0.3 573196 62696 ? Ssl Feb24 5:07 /usr/sbin/rsyslogd -n
mysql 1529 0.0 0.3 1632036 79160 ? Sl Feb24 50:33 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
apache 17418 0.0 0.1 452504 20280 ? S 16:35 0:02 /usr/sbin/httpd -DFOREGROUND
apache 15724 0.0 0.1 460704 28324 ? S Apr04 0:04 /usr/sbin/httpd -DFOREGROUND
~]$ top -b -o +RES | head -15 # Sorted by memory usage %
top - 22:07:09 up 35 days, 10 min, 1 user, load average: 1.05, 1.06, 1.10
Tasks: 135 total, 2 running, 133 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 51.6 sy, 0.0 ni, 48.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 19909192 total, 1488924 free, 17650764 used, 769504 buff/cache
KiB Swap: 8060924 total, 7847676 free, 213248 used. 1526412 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
497 root 20 0 161844 106192 105952 S 0.0 0.5 3:22.10 systemd-journal
1022 monitor 20 0 1214956 87636 12496 S 0.0 0.4 4351:58 monitor
1529 mysql 20 0 1632036 79160 4412 S 0.0 0.4 50:33.66 mysqld
1021 root 20 0 573196 62704 59532 S 0.0 0.3 5:07.77 rsyslogd
15724 apache 20 0 460704 28324 5980 S 0.0 0.1 0:04.49 httpd
17418 apache 20 0 452504 20280 5912 S 0.0 0.1 0:02.79 httpd
17415 apache 20 0 451960 19732 5896 S 0.0 0.1 0:03.46 httpd
31899 apache 20 0 451732 19528 5964 S 0.0 0.1 0:01.51 httpd
Any advice is much appreciated