10 Gbit slow (15-30MBps) running Hardware Blade

Issues related to configuring your network
Post Reply
munisb
Posts: 4
Joined: 2020/11/03 12:51:16

10 Gbit slow (15-30MBps) running Hardware Blade

Post by munisb » 2020/11/03 13:05:07

Hi,

Trying to figure this one out.

Got few CentOS VMs along with Windows in an HP Blade System
BladeSystem c7000 Enclosure G2
Model ProLiant BL460c Gen9
Server hardware type BL460c Gen9 3
Serial number 2M250701JJ

The CentOS VMs have a very slow transfer rate (15-30 MBps) whereas the transfer rate on the Windows is about 700MBps (acceptable).

Code: Select all

# sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 57.7648 s, 18.6 MB/s

Code: Select all

CentOS Linux release 7.5.1804 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

[code]
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.5.1804 (Core)
CentOS Linux release 7.5.1804 (Core)

# sudo lspci | grep Eth
06:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
06:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
06:00.2 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
06:00.3 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
06:00.4 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
06:00.5 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
06:00.6 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
06:00.7 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)

# ethtool eno51
Settings for eno51:
        Supported ports: [ Backplane ]
        Supported link modes:   10000baseKR/Full
                                20000baseKR2/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseKR/Full
                                20000baseKR2/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes


# modinfo be2net
filename:       /lib/modules/3.10.0-862.14.4.el7.x86_64/kernel/drivers/net/ethernet/emulex/benet/be2net.ko.xz
license:        GPL
author:         Emulex Corporation
description:    Emulex OneConnect NIC Driver 11.4.0.0r
version:        11.4.0.0r
retpoline:      Y
rhelversion:    7.5
srcversion:     57400C27D786514939F3803
alias:          pci:v000010DFd00000728sv*sd*bc*sc*i*
alias:          pci:v000010DFd00000720sv*sd*bc*sc*i*
alias:          pci:v000010DFd0000E228sv*sd*bc*sc*i*
alias:          pci:v000010DFd0000E220sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000710sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000700sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000221sv*sd*bc*sc*i*
alias:          pci:v000019A2d00000211sv*sd*bc*sc*i*
depends:
intree:         Y
vermagic:       3.10.0-862.14.4.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        E4:A1:B6:8F:46:8A:CA:5C:22:84:50:53:18:FD:9D:AD:72:4B:13:03
sig_hashalgo:   sha256
parm:           num_vfs:Number of PCI VFs to initialize (uint)
parm:           rx_frag_size:Size of a fragment that holds rcvd data. (ushort)

I am guessing it might be related to drivers on CentOS. Will appreciate any help.

tia
Last edited by munisb on 2020/11/11 03:25:23, edited 1 time in total.

User avatar
TrevorH
Forum Moderator
Posts: 29902
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 10 Gbit slow (15-30MBps) running on VM (ESXi)

Post by TrevorH » 2020/11/03 18:50:32

Start by updating to something a bit newer. You are 2.5 years out of date with 7.5. Current version is 7.8 and 7.9 is already in CR (yum --enablerepo=cr update).

Also post the output from lspci -nn | grep -i net from within the CentOS guest.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

munisb
Posts: 4
Joined: 2020/11/03 12:51:16

Re: 10 Gbit slow (15-30MBps) running on VM (ESXi)

Post by munisb » 2020/11/10 16:50:51

Hi Trevor,

Thanks for the reply - I updated the host with latest LT kernel and patched the system. Also tried with newer set of NIC drivers - but it didn't help much.


[root@uspaidxsplkpd01 MBadar]# lspci -nn | grep -i net
06:00.0 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)
06:00.1 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)
06:00.2 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)
06:00.3 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)
06:00.4 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)
06:00.5 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)
06:00.6 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)
06:00.7 Ethernet controller [0200]: Emulex Corporation OneConnect NIC (Skyhawk) [10df:0720] (rev 10)

BShT
Posts: 369
Joined: 2019/10/09 12:31:40

Re: 10 Gbit slow (15-30MBps) running on VM (ESXi)

Post by BShT » 2020/11/10 18:23:51

this Emulex is not a physical NIC?

User avatar
TrevorH
Forum Moderator
Posts: 29902
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 10 Gbit slow (15-30MBps) running on VM (ESXi)

Post by TrevorH » 2020/11/10 22:45:12

That looks suspiciously like output from a physical machine not the VM you were talking about? Or is VMWare emulating Emulex cards now?
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

munisb
Posts: 4
Joined: 2020/11/03 12:51:16

Re: 10 Gbit slow (15-30MBps) running on VM (ESXi)

Post by munisb » 2020/11/11 00:16:41

sorry - you are right; its a blade (physical)

I just discovered an anomaly

I get expected output when copying files FROM NFS to local disk using rsync

Code: Select all

[root@hostname 01]# pwd
/nfs/01


[root@hostname 01]# rsync --info=progress2 tempfile /home/munisb/temp1
  1,073,741,824 100%  440.99MB/s    0:00:02 (xfr#1, to-chk=0/1)
  
 [root@hostname 01]# rsync --info=progress2 /nfs/01/tempfile /home/munisb/temp2
  1,073,741,824 100%  438.16MB/s    0:00:02 (xfr#1, to-chk=0/1)
Now, if i try to copy file TO NFS from local disk - initially rSync says it copied the file at the expect 400+ MBps but then it halts for several seconds only to return to the prompt saying the below speed. I checked if the file was written to the destination - how it was not until rsync returned back to prompt.

Code: Select all

[root@hostname 01]# pwd
/nfs/01

[root@hostname 01]# rsync --info=progress2 /home/munisb/temp1 temp1
  1,073,741,824 100%   23.65MB/s    0:00:43 (xfr#1, to-chk=0/1)
 
 [root@hostname 01]# rsync --info=progress2 /home/munisb/temp1 /nfs/01/temp2
  1,073,741,824 100%   19.21MB/s    0:00:53 (xfr#1, to-chk=0/1)
another interesting test was to rsync on local disk

Code: Select all

[root@hostname 01]# pwd
/nfs/01

[root@hostname 01]# rsync --info=progress2 /home/munisb/temp1 /home/munisb/temp2
  1,073,741,824 100%  426.30MB/s    0:00:02 (xfr#1, to-chk=0/1)

[root@uspaidxsplkpd01 01]# rsync --info=progress2 temp2 temp3
  1,073,741,824 100%   12.07MB/s    0:01:24 (xfr#1, to-chk=0/1)

ran top - still have over 90% of CPU and barely any memory used (32 GB in total)

Code: Select all

# cat sysctl.conf
net.ipv4.tcp_window_scaling = 1

# Maximum receive socket buffer size
net.core.rmem_max = 134217728

# Maximum send socket buffer size
net.core.wmem_max = 134217728

# Minimum, initial and max TCP Receive buffer size in Bytes
net.ipv4.tcp_rmem = 187500 750000 134217728

# Minimum, initial and max buffer space allocated
net.ipv4.tcp_wmem = 187500 750000 134217728

# Maximum number of packets queued on the input side
net.core.netdev_max_backlog = 300000

# Auto tuning
net.ipv4.tcp_moderate_rcvbuf =1

# Don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1

# The Hamilton TCP (HighSpeed-TCP) algorithm is a packet loss based congestion control and is more aggressive pushing up to max bandwidth (total BDP) and favors hosts with lower TTL / VARTTL.
net.ipv4.tcp_congestion_control=htcp

# If you are using jumbo frames set this to avoid MTU black holes.
net.ipv4.tcp_mtu_probing = 1

munisb
Posts: 4
Joined: 2020/11/03 12:51:16

Re: 10 Gbit slow (15-30MBps) running Hardware Blade

Post by munisb » 2020/11/11 13:04:56

nvm, rsync appears to cache or keep a file in the temporary dir; so I guess back to step 1

User avatar
TrevorH
Forum Moderator
Posts: 29902
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 10 Gbit slow (15-30MBps) running Hardware Blade

Post by TrevorH » 2020/11/11 16:58:33

Writes to NFS filesystems are cached in RAM on the host doing the writing.

Try using iperf3 to do some tests.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

Post Reply

Return to “CentOS 7 - Networking Support”