kswapd0 over 500% CPU Usage

Hi CWP!
I am having a big problem with my installation
I followed the wiki
set hostname
created a package
created a user
added a domain

and its all working but i cant figure out why kswapd0 is having so much usage.

first, it seems Firewall was not set up correctly those bunch of emails i received, I tried to ignore those false alarms
and kswapd0 high usage never showed up for a week or two.
change my database ports and ssh ports to negate all those spam logins(hell knows where they came from)

and now its back.
https://prnt.sc/rwmnoh

- i have swappiness set to 0
- i have a lot of RAM space
- 12 Intel(R) Xeon(R) E-2236 CPU @ 3.40GHz processors
- Disk Raid SSD 500GB

I already tried:
systemctl stop httpd
systemctl stop mysqld
sync && echo 2 > /proc/sys/vm/drop_caches
systemctl start mysqld
systemctl start httpd


and all i have is ONE LARAVEL SITE
the crontab process is not much.

i dont know what else could've gone wrong in my set up

this is my top:

top - 11:17:37 up  1:44,  2 users,  load average: 6.97, 7.00, 6.95
Tasks: 230 total,   2 running, 121 sleeping,   0 stopped,   0 zombie
%Cpu(s): 58.2 us,  0.3 sy,  0.0 ni, 41.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65787508 total, 59105236 free,  4870992 used,  1811280 buff/cache
KiB Swap:  5242876 total,  5242876 free,        0 used. 60170128 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3189 root      20   0 2442256   2076   1416 S 599.3  0.0 619:32.79 kswapd0
15223 amavis    20   0  691788 578364   7504 R 100.0  0.9   0:04.37 clamd
 9305 root      20   0  221780  17992   9628 S   0.7  0.0   0:18.57 php-fpm
11671 root      20   0  221696  17972   9676 S   0.7  0.0   0:16.77 php-fpm
11818 root      20   0  221816  18016   9684 S   0.7  0.0   0:18.10 php-fpm
 3963 beamium   20   0 1381572  13956   7668 S   0.3  0.0   0:03.50 beamium
14778 root      20   0  158936   8732   7388 S   0.3  0.0   0:00.52 sshd
    1 root      20   0  191680   5980   3964 S   0.0  0.0   0:03.23 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 rcu_par_gp
    6 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/0:0H-kb
    8 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/0
   10 root      20   0       0      0      0 I   0.0  0.0   0:03.01 rcu_sched
   11 root      20   0       0      0      0 I   0.0  0.0   0:00.00 rcu_bh
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/0
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/1
   15 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/1
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/1
   18 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/1:0H-kb
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/2
   20 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/2
   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/2
   23 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/2:0H-ev
   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/3
   25 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/3
   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/3
   28 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/3:0H-kb
   29 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/4
   30 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/4
   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/4
   33 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/4:0H-kb
   34 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/5
   35 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/5
   36 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/5
   38 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/5:0H-kb
   39 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/6
   40 root      rt   0       0      0      0 S   0.0  0.0   0:00.53 migration/6
   41 root      20   0       0      0      0 S   0.0  0.0   0:00.22 ksoftirqd/6
   43 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/6:0H-kb
   44 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/7
   45 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 migration/7
   46 root      20   0       0      0      0 S   0.0  0.0   0:00.06 ksoftirqd/7
   48 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/7:0H-kb
   49 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/8
   50 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 migration/8
   51 root      20   0       0      0      0 S   0.0  0.0   0:00.02 ksoftirqd/8
   53 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/8:0H-kb
   54 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/9


cat /proc/3189/maps

7f89bbe00000-7f89bc000000 rw-p 00000000 00:0c 23865                      /anon_hugepage (deleted)
7f89bc000000-7f89bc200000 rw-p 00000000 00:0c 308121                     /anon_hugepage (deleted)
7f89bc200000-7f89bc400000 rw-p 00000000 00:0c 302427                     /anon_hugepage (deleted)
7f89bc400000-7f89bc600000 rw-p 00000000 00:0c 23030                      /anon_hugepage (deleted)
7f89bc600000-7f89bc800000 rw-p 00000000 00:0c 304314                     /anon_hugepage (deleted)
7f89bc800000-7f89bca00000 rw-p 00000000 00:0c 303706                     /anon_hugepage (deleted)
7f89bca00000-7f89cca00000 rw-p 00000000 00:0c 306570                     /anon_hugepage (deleted)
7f89cca00000-7f8a4ea00000 rw-p 00000000 00:0c 306569                     /anon_hugepage (deleted)
7f8a4ea88000-7f8a4eaa8000 rwxp 00000000 00:00 0
7f8a4eaa8000-7f8a4eaaa000 ---p 00000000 00:00 0
7f8a4eaaa000-7f8a4eabf000 rw-p 00000000 00:00 0
7f8a4eabf000-7f8a4eadf000 rwxp 00000000 00:00 0
7f8a4eadf000-7f8a4eae1000 ---p 00000000 00:00 0
7f8a4eae1000-7f8a4eaf6000 rw-p 00000000 00:00 0
7f8a4eaf6000-7f8a4eb16000 rwxp 00000000 00:00 0
7f8a4eb16000-7f8a4eb18000 ---p 00000000 00:00 0
7f8a4eb18000-7f8a4eb2d000 rw-p 00000000 00:00 0
7f8a4eb2d000-7f8a4eb4d000 rwxp 00000000 00:00 0
7f8a4eb4d000-7f8a4eb4f000 ---p 00000000 00:00 0
7f8a4eb4f000-7f8a4eb64000 rw-p 00000000 00:00 0
7f8a4eb64000-7f8a4eb84000 rwxp 00000000 00:00 0
7f8a4eb84000-7f8a4eb86000 ---p 00000000 00:00 0
7f8a4eb86000-7f8a4eb9b000 rw-p 00000000 00:00 0
7f8a4eb9b000-7f8a4ebbb000 rwxp 00000000 00:00 0
7f8a4ebbb000-7f8a4ebbd000 ---p 00000000 00:00 0
7f8a4ebbd000-7f8a4ebd2000 rw-p 00000000 00:00 0
7f8a4ebd2000-7f8a4ebf2000 rwxp 00000000 00:00 0
7f8a4ebf2000-7f8a4ebf4000 ---p 00000000 00:00 0
7f8a4ebf4000-7f8a4f3f5000 rw-p 00000000 00:00 0
7f8a4f3f5000-7f8a4f3f7000 ---p 00000000 00:00 0
7f8a4f3f7000-7f8a4fbf8000 rw-p 00000000 00:00 0
7f8a4fbf8000-7f8a4fbfa000 ---p 00000000 00:00 0
7f8a4fbfa000-7f8a503fb000 rw-p 00000000 00:00 0
7f8a503fb000-7f8a503fd000 ---p 00000000 00:00 0
7f8a503fd000-7f8a50bfe000 rw-p 00000000 00:00 0
7f8a50bfe000-7f8a50c00000 ---p 00000000 00:00 0
7f8a50c00000-7f8a50c15000 rw-p 00000000 00:00 0
7f8a50c15000-7f8a50c16000 r--s 00000000 09:02 25298103                   /usr/share/zoneinfo/Europe/Berlin
7f8a50c16000-7f8a50c2c000 r--p 00000000 09:02 2621511                    /root/.configrc/a/kswapd0
7f8a50c2c000-7f8a50de8000 r-xp 00016000 09:02 2621511                    /root/.configrc/a/kswapd0
7f8a50de8000-7f8a50e56000 r--p 001d2000 09:02 2621511                    /root/.configrc/a/kswapd0
7f8a50e56000-7f8a50e61000 rw-p 0023f000 09:02 2621511                    /root/.configrc/a/kswapd0
7f8a50e61000-7f8a50e9f000 rw-p 00000000 00:00 0                          [heap]
7f8a50e9f000-7f8a50f66000 rw-p 00000000 00:00 0                          [heap]
7ffe10c96000-7ffe10cb7000 rw-p 00000000 00:00 0                          [stack]
7ffe10d9c000-7ffe10d9f000 r--p 00000000 00:00 0                          [vvar]
7ffe10d9f000-7ffe10da1000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


I dont know what to do anymore. google says im OOM but i have lots of RAM space more than enough to run one site

please help me

Loads of money to throw at a server and absolutely no clue as to how to manage it?
Just about all of my distributed servers could be amalgamated into that single one (if that was a good idea - it ain't)!

15223 amavis    20   0  691788 578364   7504 R 100.0  0.9   0:04.37 clamd

Quelle surprise.

i have swappiness set to 0

Bad Move

Laravel

Bloatware 

Saving me some typing and reinforcing what I've said on this forum before, but ignored as usual:

In other words, kswapd serves not only RAM extension aka "the swap", but also mmap and other memory paging related features in linux. – Nikita Kipriyanov Sep 14 '16 at 5:57

hey man.
thanks for giving time to reply

I do understand your take on this.

Loads of money to throw at a server and absolutely no clue as to how to manage it?

we are still in the process of deploying a number of our site.
so as a first step, we move just one site and see how it goes before moving all of the other sites in the same server.
as we move all of our sites and app in to the server, we expect to maximize its capabilities.
but we don't expect issues to rise at this point in time.

Bad Move

Yeah I agree. Im still trying to figure out the cause of this issue. doing a bit of google. most of them said to set swappines from 60 to 10. and if you dont expect any swap then have it zero.. at the moment, i dont expect OOM so we set swappiness to zero "in the mean time"

Laravel

Bloatware

errr i wont touch this topic.. it depends on the developer and implementation, i guess.

15223 amavis    20   0  691788 578364   7504 R 100.0  0.9   0:04.37 clamd

this process just come and go https://prnt.sc/rwo5uz so we dont deem this as a big deal "yet"

I haven't experienced this problem before so forgive me if its kinda oozy for ya.

again, thanks for the feedback!

If there are things I dont understand, please.. let me know.

Do I really have to spell it out in words of one syllable? (Trying desperately to be polite & not swear.)
Set vm.swappiness to 1
Period.

Set vm.swappiness to 1

dude.. would you believe me i say i tried setting from 10 to zero and rebooting each time just to see kswapd0 gone?
i did it again. set vm.swappiness to 1
echo 1 > /proc/sys/vm/swappiness here as well /etc/sysctl.conf
then reboot.

but nope. kswapd0 still the same High CPU.

I been dealing with this for Hell knows how long. 2 days? .. and i dont have any idea what the . is wrong man.

Ensure that you're using latest stable kernel.

Code: [Select]

yum -y updateKeep swappiness at 1.
Confirm that it's active..

Code: [Select]

sysctl -pFlush the caches.

Code: [Select]

echo 1 > /proc/sys/vm/drop_cachesCheck vm handling

Code: [Select]

cat /proc/sys/vm/swappiness
 cat /proc/sys/vm/vfs_cache_pressure
 cat /sys/kernel/mm/transparent_hugepage/enabled

This is looking more & more like a kernel issue, likely related to the large memory pool (with perhaps RAID SSD having an influence).

Code: [Select]

uname -aInvestigate running a different kernel series i.e. go from 4.xx.xx to 3.xx.xx or vice versa.

The alternative is to recompile the existing kernel, paying particular attention to which modules are loaded in the config. You will need to look closely at the particular CPU capabilities and the memory handling section. Not too difficult to do, once the development environment is setup but can be time-consuming to get a finely tuned custom kernel. You will also need access to a KVM for the server, if not a physical console. Perhaps ask your hosting provider (Hetzner/Contabo?) if anyone else is experiencing a similar issue with this particular server configuration.
A quicker option, if the current data is expendable, might be to install the server with Debian 10 and check its' functionality - will be a different kernel "channel". If working well, then install, for example Proxmox and run Centos/CWP in a VM. That's what I would do by default on such a machine, leaving some spare capacity for further VM.

Hey! Thanks for your patience!

seems my server is up-to-date

Code: [Select]

root@server:~# yum -y update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink                                                                                                                                                                        |  30 kB  00:00:00
 * epel: mirror.imt-systems.com
OVH-metrics/7/x86_64/signature                                                                                                                                                              |  819 B  00:00:00
OVH-metrics/7/x86_64/signature                                                                                                                                                              | 2.9 kB  00:00:00 !!!
OVH-rtm/7/x86_64/signature                                                                                                                                                                  |  819 B  00:00:00
OVH-rtm/7/x86_64/signature                                                                                                                                                                  | 2.9 kB  00:00:00 !!!
base                                                                                                                                                                                        | 3.6 kB  00:00:00
cwp                                                                                                                                                                                         | 2.9 kB  00:00:00
epel                                                                                                                                                                                        | 4.7 kB  00:00:00
extras                                                                                                                                                                                      | 2.9 kB  00:00:00
mariadb                                                                                                                                                                                     | 2.9 kB  00:00:00
ovhkernel                                                                                                                                                                                   | 2.9 kB  00:00:00
updates                                                                                                                                                                                     | 2.9 kB  00:00:00
(1/2): epel/x86_64/updateinfo                                                                                                                                                               | 1.0 MB  00:00:00
(2/2): epel/x86_64/primary_db                                                                                                                                                               | 6.8 MB  00:00:00
No packages marked for update

swappiness is active and set to 1

Code: [Select]

root@server:~# sysctl -p
vm.swappiness = 1


Code: [Select]

root@server:~# cat /proc/sys/vm/swappiness
1


Code: [Select]

root@server:~# cat /proc/sys/vm/vfs_cache_pressure
100


Code: [Select]

root@server:~# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never


I found out huge page is quite related with kswapd0 since its also existing in proc/maps
seems i dont have any Free space at all.

Code: [Select]

root@server:~# grep -i huge /proc/meminfo
AnonHugePages:   1204224 kB
ShmemHugePages:        0 kB
HugePages_Total:    1174
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2404352 kB
root@server:~#

I tried disabling transparent hugepage by echo never > /sys/kernel/mm/transparent_hugepage/enabled
but those numbers still up.
i tried increasing the HugePages_Total from 1174(default i think?) to 4096 so that ill have some HugePages_Free.
but kswapd0 still up at high CPU. just trying to do a trial and error.. seems not in anyway related..

i am now trying to switch kernel series and see how it goes.

hey! really thanks for the advise!

Cheers

This may help (didn't realise 5 was available, I stick with stable distro kernels*) and would be similar to switching to another series..
https://computingforgeeks.com/install-linux-kernel-5-on-centos-7/

*I built a custom kernel for Toshiba Click Mini and used a custom kernel for a SoYouStart (OVH) Arm storage box.