Flapping interface

Issues related to configuring your network
Post Reply
mattps
Posts: 2
Joined: 2020/08/28 09:08:47

Flapping interface

Post by mattps » 2020/08/28 09:13:56

Hi,

6 Centos servers are used to run CAE simulations in a cluster and we have noticed recently that CAE jobs have been failing. Looking at dmesg logs of one server and I can see the networking interface flapping down, then up a a few seconds later. This happens randomly every 12 - 24 hours or so.

Aug 25 08:42:30 misspiggy kernel: bnx2x 0000:01:00.1: em2: NIC Link is Down
Aug 25 08:42:30 misspiggy NetworkManager[2834]: <info> (em2): carrier now OFF (device state 3)
Aug 25 08:42:30 misspiggy NetworkManager[2834]: <info> (em2): device state change: disconnected -> unavailable (reason 'carrier-chagned') [3 2 40]
Aug 25 08:42:30 misspiggy NetworkManager[2834]: <info> (em2): deactivating device (reason 'carrier-chagned') [40]
Aug 25 08:42:30 misspiggy NetworkManager[2834]: <info> Policy set 'System em3' (em3) as default for IPv4 routing and DNS.
Aug 25 08:42:30 misspiggy NetworkManager[2834]: <info> Policy set 'System em3' (em3) as default for IPv4 routing and DNS.
Aug 25 08:42:34 misspiggy kernel: bnx2x 0000:01:00.1: em2: NIC Link is Up, 10000 Mbps full duplex, Flow control: none


The 6 servers are connected to their own 10GbE switch.

I am a Windows admin and don't really have much experience with Linux which is why the configuration with these boxes haven't changed.

Can anyone suggest anything?

Thanks,

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Flapping interface

Post by TrevorH » 2020/08/28 15:44:34

It says the interface went down. You should check the switch logs and see if there is a problem that side. Also try a different switch port and a different cable. All the info here looks like a hardware problem.

Also, that NetworkManager output doesn't really look like you are running CentOS 6. Are you sure it is? If it is then it will be using a 2.6.32-xxx series of kernels so that will show up in uname -a. If it's not 6 then we can move this topic elsewhere.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

mattps
Posts: 2
Joined: 2020/08/28 09:08:47

Re: Flapping interface

Post by mattps » 2020/08/28 15:50:56

Thanks TrevorH,

I sure it's 6.8 (that's what was specified by our team to the installer). uname output is:

[root@bunsen ~]# uname -a
Linux bunsen 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

It's strange as I have continuous pings running between the hosts on this switch and they all show zero packet loss. Cables have been replaced and the switch has stock configuration on it.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Flapping interface

Post by TrevorH » 2020/08/28 16:48:00

That's already 4 years out of date.

At a minimum you should be on 6.10 plus all the latest fixes and CentOS 6 itself goes EOL in November so you need to get off it quick.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply