Author Topic: dentry eats tons of ram.  (Read 9722 times)

0 Members and 1 Guest are viewing this topic.

Offline
*
dentry eats tons of ram.
« on: September 23, 2017, 05:56:51 PM »
Ok, I've been battling this issue for over a year and I think I've finally narrowed it down. My server just serving a couple low traffic sites (wordpress) would add about 1gb of 'used' ram per day. I switched to a different, same model Dell CS24-SC server, and the problem followed to the new server.When the server sits at home with no traffic, it's fine with traffic 1gb/day. Then I added a medium-ish traffic website and my memory usage went up to 2gb/day. And this is the memory minus cache, in cap and free -m it's showing up as in use.  I never let it use all the ram, but I would reboot the server around 40/48GB of memory used (and there's only 12gb of files on the hole server!) and it would go back down and start ramping up again.

So I was poking around in /proc for some answers and found slab to be using a ton of memory:
cat /proc/meminfo
...
Slab:           33204524 kB
SReclaimable:   33168504 kB

Then slab top showed me

slabtop --s c -o
...
165121860 165121823  99%    0.19K 8256093       20  33024372K dentry

Whoah! 30GB of directory structure cache? That doesn't even make sense. I haven't changed anything in /proc/sys/vm/ aside from a sync; echo 2 > /proc/sys/vm/drop_caches (I know next time I'll stop sol before doing this) which take my memory usage back to 1GB. my /proc/sys/vm/vfs_cache_pressure its set at the 'normal' 100, swapiness is still 60 ¯\_(ツ)_/¯ no idea dentry is eating this much ram. At least now I can fix the problem without rebooting but it'd be nice if the problem didn't occur in the first place. Anyone have any suggestions?

Server info:
Apache version: Apache/2.4.27
PHP version: 5.6.27 [PHP Switcher]
MySQL version: 5.5.54
FTP version: 1.0.36
System Info

CPU Model: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz
CPU Details: 8 Core (2500 MHz)
Distro Name: CentOS release 6.9 (Final)
Kernel Version: 2.6.32-696.10.1.el6.x86_64
Platform: x86_64 [Dedicated]
Uptime: 22 days, 15:47
Server Time: Sat Sep 23 13:30:12 EDT 2017

CWP info
CWP version: 0.9.8.267

Offline
*
Re: dentry eats tons of ram.
« Reply #1 on: September 25, 2017, 12:39:21 PM »
Web Design, Development & Web Hosting
https://6sense.com.au

Offline
*
Re: dentry eats tons of ram.
« Reply #2 on: October 02, 2017, 02:54:48 AM »
At first I thought that's not it I'm not using NFS. The more I read the more it made sense. Seems like NSS in curl/libcurl is the culprit. Unfortunately the most recent compatible ver of NSS for CentOS 6.9 is the one with the bug. in the mean time I did the:

[root]echo "export NSS_SDB_USE_CACHE=YES" >> /etc/sysconfig/httpd
[root]# service httpd restart

that's still recommended  and:

[root]# echo 10000 > /proc/sys/vm/vfs_cache_pressure

which is helping to tame the problem but it hasn't gone away. Just adding slightly under 1gb/day in ram usage. On thing that would be good to know is if CPW uses curl/libcurl and where one could add the NSS_SDB_USE_CACHE variable when launching cop's apache.

Oh and look now, there's a new ver of NSS to update to even though I specifically tried humming an update last week! le sigh.
« Last Edit: October 02, 2017, 03:04:53 AM by Darkroom »

Offline
*
Re: dentry eats tons of ram.
« Reply #3 on: October 05, 2017, 11:34:37 PM »
Well the update to NSS has helped a lot! Dentry is growing at less than 1gb per day.  Still I would like to know where I could put:
echo "export NSS_SDB_USE_CACHE=YES" >> /etc/sysconfig/httpd
for the cap apache server.  Anyone know?