rsync over NFS 4.1 hangs periodically - why?

[ron7000]

• - Latest CentOS 7.9
  - I have two servers,at work, high end each has 512GB RAM and 4 xeon cpu's.
  - Connected over wired 1gbps network, using Cisco switch (sorry don't have those details at moment)
  - tuned-adm profile latency-network, and also throughput-performance; no affect problem continues...
  - ron.tar is 100GB
  - NFS server exports /bkup and the NFS-client mounts it as
  

  Code: Select all

  type nfs4 (rw,nosuid,noexec,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,,clientaddr=192.168.1.2,local_lock=none,addr=192.168.1.1,_netdev)

  - the /etc/sysconfig/nfs and /etc/nfs.conf are not altered.
  - rsync -P /data/ron.tar /bkup will burst at 400 MB/sec, run on average around 250 MB/sec, but sometimes drop to KB/sec and sometimes hang outright during the copy, like 5 minutes in after a sustained ~250MB/sec the displayed transfer speed of rsync that is displayed in the terminal window per second ceases for some amount of time. It eventually resumes (I think).
  - my ron.tar file does not exist under the destination /bkup until rynsc has fully completed and the terminal prompt returns.
  - the rsync output will say 100% and (xfr#1, to-chk=0/1) but the terminal prompt will not come back for many minutes along with ron.tar not being present under /bkup.
  
  What could be happening?
  
  Is there something in linux, or in NFS, that is saturating or something? What can I look into to try and understand why it's not getting a sustained X MB/sec and a reliable copy over NFS?

[TrevorH]

A complete guess but your rsync is probably filling up the file cache on the target system and once it gets full, it then has to write the data out to disk at which point there'd be a pause while it does so and frees up some RAM. You might look at setting things like vm.vfs_cache_pressure = 200 to try to make it write more as it receives the data rather than waiting until it has to. You could look at things like iotop on the target system as well as ordinary top to see how much of your RAM is being used for file cache (all of it I'd guess!).

[BShT]

I have experience using rsync to samba/cifs and rsync to rsync, native copy is impressively faster