multiple NICs in same subnet - data tranfer via wrong NIC

Issues related to configuring your network
Post Reply
dah_lala
Posts: 12
Joined: 2014/06/23 17:57:28

multiple NICs in same subnet - data tranfer via wrong NIC

Post by dah_lala » 2022/02/25 19:24:00

Hi,
I am running 7-9.2009.1.el7.centos.x86_64 on several machines. Two machines have several network interfaces, whereof two (192.168.0.100 and 192.168.0.176) are in the same subnet (192.168.0.0/24).
I noticed that if I transfer data using the IP of one of these interfaces in the same network (both are 10G fiber), the data is transferred via the other interface:
NIC1: 192.168.0.100
NIC2: 192.168.0.176
e.g. scp 192.168.0.176:/data/large_file ./
The data is transferred via NIC1 instead of NIC2.
How can this be if I used the IP of NIC2?
In this concrete example NIC1 is used basically exclusively. Even if I do an iperf3 NIC speed test using the IP of NIC2, it transfers packages using NIC1. I can verify that by looking at the LEDs on the network adapter, my switchs (Unifi USW aggregation and USW-Pro-24-PoE), and nload.
I see this behaviour on two machines.
If I assign NIC2 to a virtual machine, the data of the guest system goes correctly through NIC2.
The problem also doesn't persist if the two NICs are in different subnets.

I want certain data to go through certain NICs that are in the same subnet, how can I enforce the transfer through a certain NIC when the NIC's IP is used?

Thank you!

[edit]
Both NICs receive their IP from a DHCP server.

3: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:90:fa:2a:4a:9e brd ff:ff:ff:ff:ff:ff
inet 192.168.0.100/24 brd 192.168.0.255 scope global noprefixroute dynamic ens4f0
valid_lft 69162sec preferred_lft 69162sec
inet6 fe80::dbd:2153:6372:5c25/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:90:fa:2a:4a:a2 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.176/24 brd 192.168.0.255 scope global noprefixroute dynamic ens4f1
valid_lft 69162sec preferred_lft 69162sec
inet6 fe80::e49c:bc96:1651:c106/64 scope link noprefixroute
valid_lft forever preferred_lft forever

[/edit]

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: multiple NICs in same subnet - data tranfer via wrong NIC

Post by jlehtone » 2022/02/25 22:12:04

Why do want to have two addresses on same subnet?

dah_lala
Posts: 12
Joined: 2014/06/23 17:57:28

Re: multiple NICs in same subnet - data tranfer via wrong NIC

Post by dah_lala » 2022/02/26 18:57:30

Because I want to have specific traffic going through a specific NIC without affecting services using the other NIC (e.g. NFS only via NIC1 and everything else via NIC2). But that also shouldn't matter ...
(Please don't reply with alternative solutions.) I assume there must be an answer to this and if there isn't, please could someone explain why it is doing what it is doing?
Thank you :-)

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: multiple NICs in same subnet - data tranfer via wrong NIC

Post by TrevorH » 2022/02/26 19:02:45

It's doing what it is because that's what your routing tells it to do.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: multiple NICs in same subnet - data tranfer via wrong NIC

Post by jlehtone » 2022/02/26 19:31:35

I can't say whether the routing that you desire is possible, but if it is, then it is with policy based routing. See https://access.redhat.com/documentation ... ive-routes

Frankly, there are quite trivial and certain to function alternatives, but that is not what you want to hear about.

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: multiple NICs in same subnet - data tranfer via wrong NIC

Post by Whoever » 2022/02/26 19:54:18

This reply is also on target for your question and doesn't mention nmcli (for those who hate NetworkManager on servers):
https://unix.stackexchange.com/question ... s-incoming

dah_lala
Posts: 12
Joined: 2014/06/23 17:57:28

Re: multiple NICs in same subnet - data tranfer via wrong NIC

Post by dah_lala » 2022/03/03 18:19:48

Interesting! Yeah, that explains a lot :-)
Thanks a lot! I will look into it.

Post Reply