CentOS

I have a (usually) headless server with four ethernet jacks.

Previously, I've only used one (enp3s0) assigned to the server's IP of 192.168.2.20

OK, so I figured that I would try using link aggregation with two of the ethernet ports (enp3s0 and enp5s0) connected to the switch with LACP to improve bandwidth from the server.

I set up a "backup" using a third port (enp6s0) with the IP of 192.168.2.21 and then followed the instructions entitled "Set up LACP bonding interface on CentOS 8" (https://www.snel.com/support/set-up-lac ... -centos-8/)

This instructs disabling and removing NetworkManager (which is what I did in CentOS 6) and specifying scripts in /etc/sysconfig/network-scripts.

I also configured the switch to use link aggregation for those two ethernet ports.

The problem is that now I have no network.

Code: Select all

[root@ares /]# ifconfig

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 375860  bytes 31101374 (29.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 375860  bytes 31101374 (29.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Code: Select all

[root@ares /]# ip link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:75:92:bc brd ff:ff:ff:ff:ff:ff
3: enp4s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:75:92:bd brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:75:92:be brd ff:ff:ff:ff:ff:ff
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:75:92:bf brd ff:ff:ff:ff:ff:ff

I can bring up the interfaces (link-aggregated enp3s0 and enp5s0, and also the secondary/backup enp6s0):

Code: Select all

[root@ares /]# ip link set enp3s0 up

This brings up each ethernet link but doesn't restore the network connection.

Here are the configuration scripts:

Code: Select all

[root@ares /etc/sysconfig/network-scripts/]# cat ifcfg-bond0
DEVICE=bond0
Type=Bond
NAME=bond0
BONDING_MASTER=yes
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.2.20
PREFIX=24
GATEWAY=192.168.2.1
NM_CONTROLLED=no
BONDING_OPTS="mode=4 miimon=100 lacp_rate=1"


[root@ares /etc/sysconfig/network-scripts/]# cat ifcfg-enp3s0
TYPE="Ethernet"
UUID="69f913d9-7dc2-477f-a440-1515fc39e180"
DEVICE=enp3s0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
NM_CONTROLLED=no
MASTER=bond0
SLAVE=yes
IPV6INIT=no

[root@ares /etc/sysconfig/network-scripts/]# cat ifcfg-enp5s0
TYPE=Ethernet
BOOTPROTO=none
UUID=c0e558f1-cc68-4609-a9fa-2463ba3c802d
DEVICE=enp5s0
ONBOOT=yes
TYPE=Ethernet
NM_CONTROLLED=no
IPV6INIT=no
MASTER=bond0
SLAVE=yes

[root@ares /etc/sysconfig/network-scripts/]# cat ifcfg-enp6s0
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp6s0
UUID=ec4b757c-b0ae-430b-a00d-1d3a36c9ed9a
DEVICE=enp6s0
ONBOOT=yes
NM_CONTROLLED=no
IPADDR=192.168.2.21
PREFIX=24
GATEWAY=192.168.2.1

I can copy back the old network scripts that I saved before making the changes and see if that fixes the problem, but I was wondering if there was something simple I'm missing here to make this work. (My sense is that removing NetworkManager isn't the ideal solution for CentOS 8, but at least it's a learning experience.)

Michael

I seem to have solved my problem.

I was able to re-enable my backup ethernet connection:

Code: Select all

ip addr add 192.168.2.21/24 dev enp6s0
ip route add default via 192.168.2.1 dev enp6s0
dnf install NetworkManager NetworkManager-tui NetworkManager-team

Next I found procedures for establishing network bonding using nmcli (abbreviated version below):

Code: Select all

nmcli con add type team con-name bond0 ifname bond0 config '{"runner": {"name": "lacp"}}'
nmcli con show bond0
nmcli con show
nmcli con mod bond0 ipv4.addresses 192.168.2.20/24
nmcli con mod bond0 ipv4.gateway 192.168.2.1
nmcli con mod bond0 ipv4.dns 8.8.8.8
nmcli con mod bond0 ipv4.method manual
nmcli con mod bond0 connection.autoconnect yes
nmcli con add type team-slave con-name bond0-slave0 ifname enp3s0 master bond0
nmcli con add type team-slave con-name bond0-slave1 ifname enp5s0 master bond0
nmcli con show
nmcli connection down bond0 && nmcli connection up bond0
ip address show dev bond0

It's working now!

Michael

Some notes:
* It is possible to modify multiple attributes on one "con mod" command. In fact, most of them could be set within the "con add" command
* If you don't want IPv6, then "ipv6.method ignore" would be appropriate
* Firewalld zone of connection is the "connection.zone" attribute
* Team (introduced in RHEL 7) is a different implementation of aggregation than the Bond
* You probably did clean out the 192.168.2.21 already; multiple IPs on same subnet is trouble
* You need multiple clients, who do not hash the same, in order to get that extra bandwidth

@jlehtone, thanks for the info! I spent the morning reading up on teaming vs bonding. Honestly, with the lacp protocol chosen, they sound similar for what I need.

Yes, I did remove the second IP address from the backup ethernet port once I was able to get the primary (LACP) one working.

I made a mess for a while, but did learn a lot in the process.

Michael

CentOS

Attempted use of link aggregation and now have lost ethernet connection

Attempted use of link aggregation and now have lost ethernet connection

Re: Attempted use of link aggregation and now have lost ethernet connection

Re: Attempted use of link aggregation and now have lost ethernet connection

Re: Attempted use of link aggregation and now have lost ethernet connection