rsync over NFS 4.1 hangs periodically - why?

Issues related to configuring your network
Post Reply
ron7000
Posts: 162
Joined: 2019/01/15 20:00:28

rsync over NFS 4.1 hangs periodically - why?

Post by ron7000 » 2022/10/03 18:27:15

  • - Latest CentOS 7.9
    - I have two servers,at work, high end each has 512GB RAM and 4 xeon cpu's.
    - Connected over wired 1gbps network, using Cisco switch (sorry don't have those details at moment)
    - tuned-adm profile latency-network, and also throughput-performance; no affect problem continues...
    - ron.tar is 100GB
    - NFS server exports /bkup and the NFS-client mounts it as

    Code: Select all

    type nfs4 (rw,nosuid,noexec,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,,clientaddr=192.168.1.2,local_lock=none,addr=192.168.1.1,_netdev)
    - the /etc/sysconfig/nfs and /etc/nfs.conf are not altered.
    - rsync -P /data/ron.tar /bkup will burst at 400 MB/sec, run on average around 250 MB/sec, but sometimes drop to KB/sec and sometimes hang outright during the copy, like 5 minutes in after a sustained ~250MB/sec the displayed transfer speed of rsync that is displayed in the terminal window per second ceases for some amount of time. It eventually resumes (I think).
    - my ron.tar file does not exist under the destination /bkup until rynsc has fully completed and the terminal prompt returns.
    - the rsync output will say 100% and (xfr#1, to-chk=0/1) but the terminal prompt will not come back for many minutes along with ron.tar not being present under /bkup.

    What could be happening?

    Is there something in linux, or in NFS, that is saturating or something? What can I look into to try and understand why it's not getting a sustained X MB/sec and a reliable copy over NFS?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: rsync over NFS 4.1 hangs periodically - why?

Post by TrevorH » 2022/10/03 18:32:32

A complete guess but your rsync is probably filling up the file cache on the target system and once it gets full, it then has to write the data out to disk at which point there'd be a pause while it does so and frees up some RAM. You might look at setting things like vm.vfs_cache_pressure = 200 to try to make it write more as it receives the data rather than waiting until it has to. You could look at things like iotop on the target system as well as ordinary top to see how much of your RAM is being used for file cache (all of it I'd guess!).
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

BShT
Posts: 584
Joined: 2019/10/09 12:31:40

Re: rsync over NFS 4.1 hangs periodically - why?

Post by BShT » 2022/11/15 20:35:11

I have experience using rsync to samba/cifs and rsync to rsync, native copy is impressively faster

Post Reply