Packet Dropped on bridge over Bonding interface

Issues related to configuring your network
Post Reply
abhi60444
Posts: 3
Joined: 2020/05/27 14:19:24

Packet Dropped on bridge over Bonding interface

Post by abhi60444 » 2020/05/29 12:08:46

Hi All,

We are facing packet drop issue when we are using bridge interface over bonding but when we use bonding interface only it's works fine but in this case we are not able to run our kvm based virtual machines so,please help us to debug this issue .

Bridge interface(br1)->Bonding interface(bond0)->p5p1,p5p2

Below are the details for references .

Operating system:- CentOS Linux release 7.8.2003 (Core)
kernel version:- 3.10.0-1127.8.2.el7.x86_64
Kvm Version:- qemu-kvm-1.5.3-173.el7_8.3.x86_64

ifconfig bond0

bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
ether b0:26:28:44:d4:c0 txqueuelen 1000 (Ethernet)
RX packets 466196 bytes 33494292 (31.9 MiB)
RX errors 0 dropped 22686 overruns 0 frame 0
TX packets 17353 bytes 3430215 (3.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

cat ifcfg-br1

NAME=br1
DEVICE=br1
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge
BONDING_OPTS="mode=1 miimon=100"
IPADDR=192.168.1.54
NETMASK=255.255.255.0
GATEWAY=192.168.1.10
NM_CONTROLLED=no
USERCTL=no

cat ifcfg-bond0

NAME=bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
BRIDGE=br1
BONDING_OPTS="mode=1 miimon=100"
NM_CONTROLLED=no
USERCTL=no

cat ifcfg-p5p1

TYPE=Ethernet
BOOTPROTO=none
NAME=p5p1
DEVICE=p5p1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no
USERCTL=no

cat ifcfg-p5p2

TYPE=Ethernet
BOOTPROTO=none
NAME=p5p1
DEVICE=p5p1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no
USERCTL=no

ethtool -S p5p1 network statics :-

ethtool -S p5p1
NIC statistics:
[0]: rx_ucast_packets: 3932
[0]: rx_mcast_packets: 24422
[0]: rx_bcast_packets: 130611
[0]: rx_discards: 0
[0]: rx_drops: 0
[0]: rx_ucast_bytes: 776493
[0]: rx_mcast_bytes: 2763654
[0]: rx_bcast_bytes: 7836660
[0]: tx_ucast_packets: 1147
[0]: tx_mcast_packets: 0
[0]: tx_bcast_packets: 0
[0]: tx_discards: 0
[0]: tx_drops: 0
[0]: tx_ucast_bytes: 265109
[0]: tx_mcast_bytes: 0
[0]: tx_bcast_bytes: 0
[0]: tpa_packets: 303
[0]: tpa_bytes: 424098
[0]: tpa_events: 62
[0]: tpa_aborts: 2155
[0]: rx_l4_csum_errors: 0
[0]: missed_irqs: 0
[1]: rx_ucast_packets: 5847
[1]: rx_mcast_packets: 12863
[1]: rx_bcast_packets: 0
[1]: rx_discards: 0
[1]: rx_drops: 0
[1]: rx_ucast_bytes: 917201
[1]: rx_mcast_bytes: 777288
[1]: rx_bcast_bytes: 0
[1]: tx_ucast_packets: 2541
[1]: tx_mcast_packets: 0
[1]: tx_bcast_packets: 0
[1]: tx_discards: 0
[1]: tx_drops: 0
[1]: tx_ucast_bytes: 611266
[1]: tx_mcast_bytes: 0
[1]: tx_bcast_bytes: 0
[1]: tpa_packets: 273
[1]: tpa_bytes: 383576
[1]: tpa_events: 55
[1]: tpa_aborts: 1509
[1]: rx_l4_csum_errors: 0
[1]: missed_irqs: 0
[2]: rx_ucast_packets: 1981
[2]: rx_mcast_packets: 12907
[2]: rx_bcast_packets: 0
[2]: rx_discards: 0
[2]: rx_drops: 0
[2]: rx_ucast_bytes: 554826
[2]: rx_mcast_bytes: 786660
[2]: rx_bcast_bytes: 0
[2]: tx_ucast_packets: 1828
[2]: tx_mcast_packets: 3
[2]: tx_bcast_packets: 0
[2]: tx_discards: 0
[2]: tx_drops: 0
[2]: tx_ucast_bytes: 464774
[2]: tx_mcast_bytes: 210
[2]: tx_bcast_bytes: 0
[2]: tpa_packets: 273
[2]: tpa_bytes: 381710
[2]: tpa_events: 56
[2]: tpa_aborts: 1157
[2]: rx_l4_csum_errors: 0
[2]: missed_irqs: 0
[3]: rx_ucast_packets: 920
[3]: rx_mcast_packets: 265
[3]: rx_bcast_packets: 0
[3]: rx_discards: 0
[3]: rx_drops: 0
[3]: rx_ucast_bytes: 441937
[3]: rx_mcast_bytes: 24282
[3]: rx_bcast_bytes: 0
[3]: tx_ucast_packets: 1241
[3]: tx_mcast_packets: 0
[3]: tx_bcast_packets: 11
[3]: tx_discards: 0
[3]: tx_drops: 0
[3]: tx_ucast_bytes: 273977
[3]: tx_mcast_bytes: 0
[3]: tx_bcast_bytes: 462
[3]: tpa_packets: 253
[3]: tpa_bytes: 355886
[3]: tpa_events: 53
[3]: tpa_aborts: 496
[3]: rx_l4_csum_errors: 0
[3]: missed_irqs: 0
[4]: rx_ucast_packets: 882
[4]: rx_mcast_packets: 76970
[4]: rx_bcast_packets: 0
[4]: rx_discards: 0
[4]: rx_drops: 0
[4]: rx_ucast_bytes: 462424
[4]: rx_mcast_bytes: 4651724
[4]: rx_bcast_bytes: 0
[4]: tx_ucast_packets: 561
[4]: tx_mcast_packets: 0
[4]: tx_bcast_packets: 0
[4]: tx_discards: 0
[4]: tx_drops: 0
[4]: tx_ucast_bytes: 116328
[4]: tx_mcast_bytes: 0
[4]: tx_bcast_bytes: 0
[4]: tpa_packets: 275
[4]: tpa_bytes: 387047
[4]: tpa_events: 56
[4]: tpa_aborts: 406
[4]: rx_l4_csum_errors: 0
[4]: missed_irqs: 0
[5]: rx_ucast_packets: 1107
[5]: rx_mcast_packets: 60
[5]: rx_bcast_packets: 0
[5]: rx_discards: 0
[5]: rx_drops: 0
[5]: rx_ucast_bytes: 499499
[5]: rx_mcast_bytes: 12780
[5]: rx_bcast_bytes: 0
[5]: tx_ucast_packets: 6743
[5]: tx_mcast_packets: 3
[5]: tx_bcast_packets: 0
[5]: tx_discards: 0
[5]: tx_drops: 0
[5]: tx_ucast_bytes: 840807
[5]: tx_mcast_bytes: 266
[5]: tx_bcast_bytes: 0
[5]: tpa_packets: 275
[5]: tpa_bytes: 387047
[5]: tpa_events: 56
[5]: tpa_aborts: 490
[5]: rx_l4_csum_errors: 0
[5]: missed_irqs: 0
[6]: rx_ucast_packets: 969
[6]: rx_mcast_packets: 68
[6]: rx_bcast_packets: 0
[6]: rx_discards: 0
[6]: rx_drops: 0
[6]: rx_ucast_bytes: 422366
[6]: rx_mcast_bytes: 13956
[6]: rx_bcast_bytes: 0
[6]: tx_ucast_packets: 562
[6]: tx_mcast_packets: 0
[6]: tx_bcast_packets: 0
[6]: tx_discards: 0
[6]: tx_drops: 0
[6]: tx_ucast_bytes: 240340
[6]: tx_mcast_bytes: 0
[6]: tx_bcast_bytes: 0
[6]: tpa_packets: 237
[6]: tpa_bytes: 334799
[6]: tpa_events: 48
[6]: tpa_aborts: 476
[6]: rx_l4_csum_errors: 0
[6]: missed_irqs: 0
[7]: rx_ucast_packets: 991
[7]: rx_mcast_packets: 12871
[7]: rx_bcast_packets: 0
[7]: rx_discards: 0
[7]: rx_drops: 0
[7]: rx_ucast_bytes: 483460
[7]: rx_mcast_bytes: 778992
[7]: rx_bcast_bytes: 0
[7]: tx_ucast_packets: 301
[7]: tx_mcast_packets: 2
[7]: tx_bcast_packets: 0
[7]: tx_discards: 0
[7]: tx_drops: 0
[7]: tx_ucast_bytes: 123655
[7]: tx_mcast_bytes: 180
[7]: tx_bcast_bytes: 0
[7]: tpa_packets: 287
[7]: tpa_bytes: 400216
[7]: tpa_events: 60
[7]: tpa_aborts: 541
[7]: rx_l4_csum_errors: 0
[7]: missed_irqs: 0
rx_total_discard_pkts: 0
tx_total_discard_pkts: 0
rx_64b_frames: 401613
rx_65b_127b_frames: 39495
rx_128b_255b_frames: 19690
rx_256b_511b_frames: 288
rx_512b_1023b_frames: 175
rx_1024b_1518b_frames: 2014
rx_good_vlan_frames: 0
rx_1519b_2047b_frames: 0
rx_2048b_4095b_frames: 0
rx_4096b_9216b_frames: 0
rx_9217b_16383b_frames: 0
rx_total_frames: 463275
rx_ucast_frames: 21125
rx_mcast_frames: 228462
rx_bcast_frames: 213688
rx_fcs_err_frames: 0
rx_ctrl_frames: 0
rx_pause_frames: 0
rx_pfc_frames: 0
rx_align_err_frames: 0
rx_ovrsz_frames: 0
rx_jbr_frames: 0
rx_mtu_err_frames: 0
rx_tagged_frames: 0
rx_double_tagged_frames: 0
rx_good_frames: 463275
rx_pfc_ena_frames_pri0: 0
rx_pfc_ena_frames_pri1: 0
rx_pfc_ena_frames_pri2: 0
rx_pfc_ena_frames_pri3: 0
rx_pfc_ena_frames_pri4: 0
rx_pfc_ena_frames_pri5: 0
rx_pfc_ena_frames_pri6: 0
rx_pfc_ena_frames_pri7: 0
rx_undrsz_frames: 0
rx_eee_lpi_events: 0
rx_eee_lpi_duration: 0
rx_bytes: 35541292
rx_runt_bytes: 0
rx_runt_frames: 0
rx_stat_discard: 790
rx_stat_err: 0
tx_64b_frames: 116
tx_65b_127b_frames: 10856
tx_128b_255b_frames: 3848
tx_256b_511b_frames: 668
tx_512b_1023b_frames: 157
tx_1024b_1518b_frames: 743
tx_good_vlan_frames: 0
tx_1519b_2047b_frames: 0
tx_2048b_4095b_frames: 0
tx_4096b_9216b_frames: 0
tx_9217b_16383b_frames: 0
tx_good_frames: 16388
tx_total_frames: 16388
tx_ucast_frames: 14925
tx_mcast_frames: 1452
tx_bcast_frames: 11
tx_pause_frames: 0
tx_pfc_frames: 0
tx_jabber_frames: 0
tx_fcs_err_frames: 0
tx_err: 0
tx_fifo_underruns: 0
tx_pfc_ena_frames_pri0: 0
tx_pfc_ena_frames_pri1: 0
tx_pfc_ena_frames_pri2: 0
tx_pfc_ena_frames_pri3: 0
tx_pfc_ena_frames_pri4: 0
tx_pfc_ena_frames_pri5: 0
tx_pfc_ena_frames_pri6: 0
tx_pfc_ena_frames_pri7: 0
tx_eee_lpi_events: 0
tx_eee_lpi_duration: 0
tx_total_collisions: 0
tx_bytes: 3255636
tx_xthol_frames: 0
tx_stat_discard: 0
tx_stat_error: 0
link_down_events: 55
continuous_pause_events: 0
resume_pause_events: 0
continuous_roce_pause_events: 0
resume_roce_pause_events: 0
rx_bytes_cos0: 0
rx_packets_cos0: 0
rx_bytes_cos1: 0
rx_packets_cos1: 0
rx_bytes_cos2: 0
rx_packets_cos2: 0
rx_bytes_cos3: 0
rx_packets_cos3: 0
rx_bytes_cos4: 0
rx_packets_cos4: 0
rx_bytes_cos5: 0
rx_packets_cos5: 0
rx_bytes_cos6: 0
rx_packets_cos6: 0
rx_bytes_cos7: 0
rx_packets_cos7: 0
pfc_pri0_rx_duration_us: 0
pfc_pri0_rx_transitions: 0
pfc_pri1_rx_duration_us: 0
pfc_pri1_rx_transitions: 0
pfc_pri2_rx_duration_us: 0
pfc_pri2_rx_transitions: 0
pfc_pri3_rx_duration_us: 0
pfc_pri3_rx_transitions: 0
pfc_pri4_rx_duration_us: 0
pfc_pri4_rx_transitions: 0
pfc_pri5_rx_duration_us: 0
pfc_pri5_rx_transitions: 0
pfc_pri6_rx_duration_us: 0
pfc_pri6_rx_transitions: 0
pfc_pri7_rx_duration_us: 0
pfc_pri7_rx_transitions: 0
tx_bytes_cos0: 0
tx_packets_cos0: 0
tx_bytes_cos1: 0
tx_packets_cos1: 0
tx_bytes_cos2: 0
tx_packets_cos2: 0
tx_bytes_cos3: 0
tx_packets_cos3: 0
tx_bytes_cos4: 0
tx_packets_cos4: 0
tx_bytes_cos5: 0
tx_packets_cos5: 0
tx_bytes_cos6: 0
tx_packets_cos6: 0
tx_bytes_cos7: 0
tx_packets_cos7: 0
pfc_pri0_tx_duration_us: 0
pfc_pri0_tx_transitions: 0
pfc_pri1_tx_duration_us: 0
pfc_pri1_tx_transitions: 0
pfc_pri2_tx_duration_us: 0
pfc_pri2_tx_transitions: 0
pfc_pri3_tx_duration_us: 0
pfc_pri3_tx_transitions: 0
pfc_pri4_tx_duration_us: 0
pfc_pri4_tx_transitions: 0
pfc_pri5_tx_duration_us: 0
pfc_pri5_tx_transitions: 0
pfc_pri6_tx_duration_us: 0
pfc_pri6_tx_transitions: 0
pfc_pri7_tx_duration_us: 0
pfc_pri7_tx_transitions: 0
rx_bytes_pri0: 0
rx_bytes_pri1: 0
rx_bytes_pri2: 0
rx_bytes_pri3: 0
rx_bytes_pri4: 0
rx_bytes_pri5: 0
rx_bytes_pri6: 0
rx_bytes_pri7: 0
rx_packets_pri0: 0
rx_packets_pri1: 0
rx_packets_pri2: 0
rx_packets_pri3: 0
rx_packets_pri4: 0
rx_packets_pri5: 0
rx_packets_pri6: 0
rx_packets_pri7: 0
tx_bytes_pri0: 0
tx_bytes_pri1: 0
tx_bytes_pri2: 0
tx_bytes_pri3: 0
tx_bytes_pri4: 0
tx_bytes_pri5: 0
tx_bytes_pri6: 0
tx_bytes_pri7: 0
tx_packets_pri0: 0
tx_packets_pri1: 0
tx_packets_pri2: 0
tx_packets_pri3: 0
tx_packets_pri4: 0
tx_packets_pri5: 0
tx_packets_pri6: 0
tx_packets_pri7: 0

netstat -i :-

Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0 1500 477591 0 23308 0 17767 0 0 0 BMmRU
br1 1500 276756 0 0 0 14511 0 0 0 BMRU
lo 65536 5781 0 0 0 5781 0 0 0 LRU
p5p1 1500 376919 0 11654 0 15049 0 0 0 BMsRU
p5p2 1500 217609 0 0 0 2718 0 0 0 BMsRU
virbr0 1500 0 0 0 0 0 0 0 0 BMU
Attachments
interface.png
interface.png (14.59 KiB) Viewed 237 times

User avatar
TrevorH
Forum Moderator
Posts: 28827
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Packet Dropped on bridge over Bonding interface

Post by TrevorH » 2020/05/29 15:29:46

I run a hundred or more VMs in this configuration and have no such problems. Are you sure your bond is set up and working correctly? What is the content of /proc/net/bonding/bondX?
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

abhi60444
Posts: 3
Joined: 2020/05/27 14:19:24

Re: Packet Dropped on bridge over Bonding interface

Post by abhi60444 » 2020/05/29 17:21:34

@TrevorH Please see /proc/net/bonding/bondX output .Also VMs which are running on physical servers are running fine but we see frequent packet drops there also .

cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: p5p1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: p5p1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b0:26:28:44:d4:c0
Slave queue ID: 0

Slave Interface: p5p2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b0:26:28:44:d4:c1
Slave queue ID: 0

abhi60444
Posts: 3
Joined: 2020/05/27 14:19:24

Re: Packet Dropped on bridge over Bonding interface

Post by abhi60444 » 2020/06/26 09:33:37

@ALL We have observed this issue and finally figure out that this issue is happening when we have Mac address of salve nic active on bonding interface
and fixed it by adding below parameter in ifcfg-bond configuration file .

fail_over_mac=1 :- When we add this in bonding configuration then bonding interface will
use NIC of Active network interface .

Post Reply

Return to “CentOS 7 - Networking Support”