Page 1 of 1

NFS client performance degrades

Posted: 2008/04/04 14:10:29
by cdillardhsp
Folks,
I would appreciate any help with this issue.

We are setting up a revamped hosting platform built on LAMP where we have two web/app servers and an NFS server to serve up the /htdocs files. We're running CentOS 5.1 on all three boxes. What happens is that from a reboot, the two web servers return pages very quickly, usually less than 300ms. After a couple of days of normal use the pages start being returned more slowly, say around 4 seconds.

Our mount options in fstab on the web servers are:
rw,noatime,nfsvers=3,tcp,intr,rsize=32768,wsize=32768

The export options on the NFS server are:
rw,no_root_squash,async

The servers are not swapping when we start seeing poor performance. We do notice that after a reboot, and before performance starts to degrade that the web servers immediately return page content. Once the performance starts to degrade we notice that when a link is clicked there is a delay before the server returns the page. Once the server starts the return of the page, it is quick but the delay is killing the response time.

Today I decided to write a file to the NFS mount on the client servers to see how this affected performance as I suspect that NFS is our issue since our Apache settings have been properly tuned. After writing a file to the NFS mount we immediately noticed that the page response times were slower by about 2.5 times that of previous requests prior to writing the file. We used time dd if=/dev/zero of=/mnt/htdocs/testfile bs=16k count=8192 to write a 134MB file to the NFS mount point on the web server.

time dd if=/dev/zero of=/mnt/htdocs/testfile bs=16k count=8192
8192+0 records in
8192+0 records out
134217728 bytes (134 MB) copied, 1.49009 seconds, 90.1 MB/s

The speed is good on the file write, but as I said, immediately after this the page response times were much slower. Only a reboot seems to fix this issue. We have tried stopping httpd and umounting then mounting the NFS export and starting httpd again but this does not result in fast page response times.

I'm hoping that someone has knowledge that will help us determine how to fix this issue. Thank you very much in advance!

Server Specs:

# Web Servers #
2 x Intel(R) Xeon(R) CPU E5420 @ 2.50GHz
1GB RAM
CentOS 5
2.6.18-53.1.14.el5xen

# NFS Server #
2 x AMD Opteron(tm) Processor 275 @ 1.8GHz
5GB RAM
CentOS 5
2.6.18-53.1.6.el5PAE

Re: NFS client performance degrades

Posted: 2008/04/04 14:16:26
by toracat
Please check out this forum post:

http://www.centos.org/modules/newbb/viewtopic.php?viewmode=thread&topic_id=13621&forum=41&post_id=45977