Network bonding problem in Centos 7

[soushiant]

Hi. in my SuperMicro server with dual 1G ethernet ports i wanted to use port bonding (mode 802.3ad). The bonding is activated in the datacenter and they said that the ports are up but i can't connect to the server and it doesn't ping. The NetworkManager is "yum removed". When the datacenter disables the bonding on their router, then i can connect to the server normally with only 1G speed. The DNS is also set in etc/Resolv.conf . The output of /proc/net/bonding/nm-bond is :

The ifcfg for bonding:

Code: Select all

BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=1"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=188.0.0.1
PREFIX=29
IPADDR1=188.0.0.1
PREFIX1=29
IPADDR2=188.0.0.2
PREFIX2=32
IPADDR3=188.0.0.3
PREFIX3=32
GATEWAY=188.0.1.10
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Bond connection 1"
NM_CONTROLLED=no
UUID=54d2d0fa-64d1-455e-b0da-f9bd4435103a
DEVICE=nm-bond
ONBOOT=yes

the ifcfg for ethernet ports:

Code: Select all

HWADDR=0C:C4:7A:B4:87:FA
TYPE=Ethernet
NAME=Eth1
UUID=cf8a0c68-fa9d-4f72-8cb6-dcce1577f431
ONBOOT=yes
BOOTPROTO=none
MASTER=nm-bond
NM_CONTROLLED=no
SLAVE=yes
MASTER_UUID=54d2d0fa-64d1-455e-b0da-f9bd4435103a

Code: Select all

HWADDR=0C:C4:7A:B4:87:FB
TYPE=Ethernet
NAME=Eth2
UUID=eeb16789-f33f-466a-8f04-832f09600163
ONBOOT=yes
MASTER=nm-bond
BOOTPROTO=none
NM_CONTROLLED=no
SLAVE=yes
MASTER_UUID=54d2d0fa-64d1-455e-b0da-f9bd4435103a

[TrevorH]

Your bond looks fine from the output you posted though you did miss the second link's output from it (it's only got some of the output from eno1 in it and none of whatever the other link is) but it does say "Number of Ports : 2" which is usually a good indicator that it's all set up right.

What you do have though is an IPADDR=188.0.0.1/PREFIX=29 which means you can address 8 hosts in your subnet so it's from 188.0.0.1 to 188.0.0.7 yet your GATEWAY is 188.0.1.10 which is *way* outside the range it can reach.

You've also got no DEVICE= lines in your ifcg files, only NAME= which is cosmetic and used only by network-manager.

So the question is more: why is it working when the DC remove the bonding. It shouldn't as it's misconfigured and should never work.

[soushiant]

Well I've changed the IPs just for the forum display because i didn't wanted to be visible to public but i did it wrong! Consider the following ifcfg for bonding file:

Code: Select all

BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=1"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=188.0.0.195
PREFIX=32
IPADDR1=188.0.0.196
PREFIX1=32
IPADDR2=188.0.0.197
PREFIX2=32
IPADDR3=188.0.0.198
PREFIX3=32
GATEWAY=188.0.0.193
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Bond connection 1"
NM_CONTROLLED=no
UUID=54d2d0fa-64d1-455e-b0da-f9bd4435103a
DEVICE=nm-bond
ONBOOT=yes

According to proc/net/bonding the device names seems to be "eno1" and "eno2". But i don't know if missing device names caused the problem?
And by the way, the rest of proc/net/bonding/nm-bond image which shows data for Ethernet2 was like Eth1 but didn't fit in print screen from IPMI.

[TrevorH]

IPADDR=188.0.0.195
PREFIX=32
IPADDR1=188.0.0.196
PREFIX1=32
IPADDR2=188.0.0.197
PREFIX2=32
IPADDR3=188.0.0.198
PREFIX3=32
GATEWAY=188.0.0.193
[/quote

Still the same problem only now worse. Your PREFIX=32 which means it can only communicate with itself unless it goes via the GATEWAY and it can't reach the gateway because your PREFIX is too small to let it do so. Perhaps you meant to use PREFIX=29?

[soushiant]

With changing the PREFIX to 29 the bonding is now active both on the server and DC. But the problem is that the "Send" traffic goes only through Eth1 and the "Receive" traffic is coming only from Eth2! Therefore, the load is not balanced.

[TrevorH]

You are misunderstanding how bonding works. You will not get double the bandwidth. I suggest you install the kernel-doc package and then read /usr/share/doc/kernel-doc-3.10.0/Documentation/networking/bonding.txt which has a lot of detail about what it can and cannot do.

[soushiant]

When i download a sample file from the server, the traffic is not equally distributed between ethernet ports. It only comes through Eth1. As i know this is not normal for port channeling(bonding) in mode 4.

[TrevorH]

It is normal and explained in the doc I pointed you to.

[soushiant]

Yes it says that a single TCP connection will not be fragmented between the ethernet slaves. So in case of multiple users downloading from the server, the traffic would be balanced.

[soushiant]

A main problem after successfully configuring the bonding was that xmit_hash_policy was not set in bond0.ifcfg file. The default value then sends the outgoing traffic in one port (Although there are many destination IPs to send) and receives the incoming traffic on the other port. Setting the policy to layer2+3 finally solved the balancing problem:

Code: Select all

xmit_hash_policy=layer2+3