Control Web Panel
WebPanel => CentOS 6 Problems => Topic started by: Darkroom on September 23, 2017, 05:56:51 PM
-
Ok, I've been battling this issue for over a year and I think I've finally narrowed it down. My server just serving a couple low traffic sites (wordpress) would add about 1gb of 'used' ram per day. I switched to a different, same model Dell CS24-SC server, and the problem followed to the new server.When the server sits at home with no traffic, it's fine with traffic 1gb/day. Then I added a medium-ish traffic website and my memory usage went up to 2gb/day. And this is the memory minus cache, in cap and free -m it's showing up as in use. I never let it use all the ram, but I would reboot the server around 40/48GB of memory used (and there's only 12gb of files on the hole server!) and it would go back down and start ramping up again.
So I was poking around in /proc for some answers and found slab to be using a ton of memory:
cat /proc/meminfo
...
Slab: 33204524 kB
SReclaimable: 33168504 kB
Then slab top showed me
slabtop --s c -o
...
165121860 165121823 99% 0.19K 8256093 20 33024372K dentry
Whoah! 30GB of directory structure cache? That doesn't even make sense. I haven't changed anything in /proc/sys/vm/ aside from a sync; echo 2 > /proc/sys/vm/drop_caches (I know next time I'll stop sol before doing this) which take my memory usage back to 1GB. my /proc/sys/vm/vfs_cache_pressure its set at the 'normal' 100, swapiness is still 60 ¯\_(ツ)_/¯ no idea dentry is eating this much ram. At least now I can fix the problem without rebooting but it'd be nice if the problem didn't occur in the first place. Anyone have any suggestions?
Server info:
Apache version: Apache/2.4.27
PHP version: 5.6.27 [PHP Switcher]
MySQL version: 5.5.54
FTP version: 1.0.36
System Info
CPU Model: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz
CPU Details: 8 Core (2500 MHz)
Distro Name: CentOS release 6.9 (Final)
Kernel Version: 2.6.32-696.10.1.el6.x86_64
Platform: x86_64 [Dedicated]
Uptime: 22 days, 15:47
Server Time: Sat Sep 23 13:30:12 EDT 2017
CWP info
CWP version: 0.9.8.267
-
This kinda sounds like the same issue
https://serverfault.com/questions/561350/unusually-high-dentry-cache-usage
-
At first I thought that's not it I'm not using NFS. The more I read the more it made sense. Seems like NSS in curl/libcurl is the culprit. Unfortunately the most recent compatible ver of NSS for CentOS 6.9 is the one with the bug. in the mean time I did the:
[root]echo "export NSS_SDB_USE_CACHE=YES" >> /etc/sysconfig/httpd
[root]# service httpd restart
that's still recommended and:
[root]# echo 10000 > /proc/sys/vm/vfs_cache_pressure
which is helping to tame the problem but it hasn't gone away. Just adding slightly under 1gb/day in ram usage. On thing that would be good to know is if CPW uses curl/libcurl and where one could add the NSS_SDB_USE_CACHE variable when launching cop's apache.
Oh and look now, there's a new ver of NSS to update to even though I specifically tried humming an update last week! le sigh.
-
Well the update to NSS has helped a lot! Dentry is growing at less than 1gb per day. Still I would like to know where I could put:
echo "export NSS_SDB_USE_CACHE=YES" >> /etc/sysconfig/httpd
for the cap apache server. Anyone know?