Packet forwarding issue with Centos 8 system.

Issues related to configuring your network
Post Reply
hcook1
Posts: 5
Joined: 2020/12/08 00:12:00

Packet forwarding issue with Centos 8 system.

Post by hcook1 » 2020/12/08 00:50:43

HELP! I am completely stymied and I know it is something stupid I am missing...
I cannot get a newly installed Centos 8 to forward packets between two subnets.

Here is the configuration:

Centos 8 server with two ethernet adapters:

The first Adapter:
eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
inet 10.4.150.57/24 brd 10.4.150.255 scope global dynamic noprefixroute eno1
valid_lft 37899sec preferred_lft 37899sec
inet6 fe80::ca1f:66ff:fef8:bd8c/64 scope link
valid_lft forever preferred_lft forever

The second adapter:
eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
inet 10.4.160.1/24 brd 10.4.160.255 scope global noprefixroute eno2
valid_lft forever preferred_lft forever
inet6 fe80::ca1f:66ff:fef8:bd8d/64 scope link
valid_lft forever preferred_lft forever

I have enabled Kernel forwarding:
sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

I think I have the routing setup correctly:
ip route
default via 10.4.150.1 dev eno1 proto dhcp metric 100
10.4.150.0/24 dev eno1 proto kernel scope link src 10.4.150.57 metric 100
10.4.160.0/24 dev eno2 proto kernel scope link src 10.4.160.1 metric 101


Here is the issue:
When I ping from a system on the 10.4.160.xx network, I can ping both interfaces on the new Centos server but I cannot ping anything on the 10.4.150.xx side of the server:
[root@rh3 ~]# ping -c 3 10.4.160.1 <--- 10.4.160.1 is the gateway - Adapter 2 above
PING 10.4.160.1 (10.4.160.1) 56(84) bytes of data.
64 bytes from 10.4.160.1: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 10.4.160.1: icmp_seq=2 ttl=64 time=0.116 ms
64 bytes from 10.4.160.1: icmp_seq=3 ttl=64 time=0.118 ms

--- 10.4.160.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2081ms
rtt min/avg/max/mdev = 0.116/0.117/0.118/0.000 ms
[root@rh3 ~]#
[root@rh3 ~]# ping -c 3 10.4.150.57 <--- 10.4.150.57 is the client on the 10.4.150.xx network - Adapter 1 above
PING 10.4.150.57 (10.4.150.57) 56(84) bytes of data.
64 bytes from 10.4.150.57: icmp_seq=1 ttl=64 time=0.118 ms
64 bytes from 10.4.150.57: icmp_seq=2 ttl=64 time=0.125 ms
64 bytes from 10.4.150.57: icmp_seq=3 ttl=64 time=0.131 ms

--- 10.4.150.57 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2056ms
rtt min/avg/max/mdev = 0.118/0.124/0.131/0.005 ms
[root@rh3 ~]#
[root@rh3 ~]# ping -c 3 10.4.150.1 <--- 10.4.150.1 is the gateway on the 10.4.150.xx network
PING 10.4.150.1 (10.4.150.1) 56(84) bytes of data.
^C <--- No response for a period of time.
--- 10.4.150.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2025ms

I am on a secure subnet so I have disabled firewalld.
I am using NetworkManager and dnsmasq for DHCP (which appears to be working just fine).
From newly installed Centos 8 system system I can ping other nodes on the 10.4.160.xx network and can ping any system over the 10.5.150.xx subnet.

Help Please! What am I missing?

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Packet forwarding issue with Centos 8 system.

Post by jlehtone » 2020/12/08 13:15:50

Lets recap:
* "rh3" has 10.4.160.1 as the default route
* "rh3" can ping anyone in the 10.4.160.0/24 (including 10.4.160.1) as they are link-local neighbours
* "rh3" can ping 10.4.150.57, because rh3's route sends that packet to 10.4.160.1 and
the server (that has 10.4.150.57) has route: 10.4.160.0/24 dev eno2 src 10.4.160.1

When "rh3" pings 10.4.150.1, it again sends via 10.4.160.1. The 10.4.150.1 will receive packet that has SRC=IP-of-rh3 (10.4.160.y)
What is the routing table at 10.4.150.1?
It probably has (if it is a router)

Code: Select all

10.4.150.0/24 dev xxx src 10.4.150.1
a.b.c.d/e dev yyy src a.b.c...
default via a.b.c.f
However, does it have route 10.4.160.0/24 via 10.4.150.57 too?

hcook1
Posts: 5
Joined: 2020/12/08 00:12:00

Re: Packet forwarding issue with Centos 8 system.

Post by hcook1 » 2020/12/08 17:56:36

jlehtone,

Thanks you for the reply.

Let me clarify, the ip route output I included in the original post was from the new centos 8 server that is the issue. To make is easier, the hostname of this new server is dick. This is the server where I am trying to make the packet forwarding function work. I have two other servers in the same lab (of which 10.4.150.1 is one of them) that are doing the exact same packet forwarding function that I am trying to setup here and I am replicating everything (I think) on this new server.

Here is the relevant data again from dick (the new server):

[root@dick ~]# hostname
dick

[root@dick ~]ip route
default via 10.4.150.1 dev eno1 proto kernel metric 1
10.4.150.0/24 dev eno1 proto kernel src 10.4.150.57 metric 100
10.4.160.0/24 dev eno2 proto kernel scope link src 10.4.160.1 metric 101

[root@dick ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether c8:1f:66:f8:bd:8c brd ff:ff:ff:ff:ff:ff
inet 10.4.150.57/24 brd 10.4.150.255 scope global dynamic noprefixroute eno1
valid_lft 36325sec preferred_lft 36325sec
inet6 fe80::ca1f:66ff:fef8:bd8c/64 scope link
valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether c8:1f:66:f8:bd:8d brd ff:ff:ff:ff:ff:ff
inet 10.4.160.1/24 brd 10.4.160.255 scope global noprefixroute eno2
valid_lft forever preferred_lft forever
inet6 fe80::ca1f:66ff:fef8:bd8d/64 scope link
valid_lft forever preferred_lft forever


as far as your question regarding the routing table on 10.4.150.1, I do not believe that they come into play. I cannot ping any of the systems on 10.4.150.0/24 - and there are several.

in any case, I tried adding a route as you suggested: "However, does it have route 10.4.160.0/24 via 10.4.150.57 too" but the effect of that was to stop traffic from the new server (aka dick) to the 10.4.160.0/24 network. That is, pings to system on 10.4.160.0/24 hung and report all packets lost.

What am I missing...
--- Harold

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Packet forwarding issue with Centos 8 system.

Post by jlehtone » 2020/12/08 19:06:35

Lets try again. Four machines:
* rh3 is in subnet 10.4.160.0/24
* snafu is in subnet 10.4.150.0/24
* dick is in both
* Router is in subnet 10.4.150.0/24 and in WAN

rh3 wants to ping snafu. rh3 does not know where 10.4.150.0/24 is. It knows only that dick knows.
Therefore, rh3 sends the ping to dick (10.4.160.1)

Dick receives the ping and sees that its destination (snafu) is in 10.4.150.0/24.
Therefore dick tosses it out of the port (eno1)

Snafu receives a ping from rh3. Snafu knows its own subnet (10.4.150.0/24) and that Router (10.4.150.1) knows everybody else. Snafu does not know rh3 amd hence sends pong to Router.

Router receives pong from snafu. Pong should go to "rh3". Router does not know where 10.4.160.0/24 so it must send it to its "default route" ... in the WAN. (But good routers do not send to WAN what has private address, so it should drop the pong.)


If snafu had a static route 10.4.160.0/24 via 10.4.150.57, then it would not send pong to Router, but to dick (10.4.150.57, who can forward it to rh3.)

If Router had a static route 10.4.160.0/24 via 10.4.150.57, then it could send pong to dick (who can forward it to rh3.)

If you did add 10.4.160.0/24 via 10.4.150.57 to dick, which already had 10.4.160.0/24 dev eno2 proto kernel scope link src 10.4.160.1, then you just broke the machine that did not have any problem to begin with.

hcook1
Posts: 5
Joined: 2020/12/08 00:12:00

Re: Packet forwarding issue with Centos 8 system.

Post by hcook1 » 2020/12/09 18:38:22

jlehtone,

Thank you for your comments. While it does show improvement in the situation, it did not resolve it. Let me provide a few more details.

Code: Select all

                        <corp net>---<server A>---<server B>---<server dick>---<node rh3>
upstream address:                     10.1.3.27    10.4.1.27    10.4.150.57    10.4.160.35
Downstream address:                   10.4.1.1     10.4.150.1   10.4.160.1      N/A
Server B is the one that I think you were referring to as "snafu".

With the config files that I provided above, all three of the servers can ping all of peer nodes on their respective subnets and all three servers can ping nodes upstream and downstream from their local subnet. All of the nodes on the 10.4.1.0/24 and 10.4.150.0/24 subnets can ping everything EXCEPT nodes on the 10.4.160.0/24 network.

As I have described earlier, rh3 can ping peer nodes in the 10.4.160.0/24 subnet and can ping both interfaces on on dick (10.4.160.1 and 10.4.150.57) BUT cannot ping anything else.

So, when I added the route that you suggested: "If snafu had a static route 10.4.160.0/24 via 10.4.150.57, then it would not send pong to Router, but to dick (10.4.150.57, who can forward it to rh3.)", the situation is improved, but still ultimately fails:

[root@rh3 ~]# ping -c 3 10.4.160.1 <== GOOD
PING 10.4.160.1 (10.4.160.1) 56(84) bytes of data.
64 bytes from 10.4.160.1: icmp_seq=1 ttl=64 time=0.154 ms
64 bytes from 10.4.160.1: icmp_seq=2 ttl=64 time=0.124 ms
64 bytes from 10.4.160.1: icmp_seq=3 ttl=64 time=0.125 ms

--- 10.4.160.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.124/0.134/0.154/0.013 ms

[root@rh3 ~]# ping -c 3 10.4.150.57 <== GOOD
PING 10.4.150.57 (10.4.150.57) 56(84) bytes of data.
64 bytes from 10.4.150.57: icmp_seq=1 ttl=64 time=0.124 ms
64 bytes from 10.4.150.57: icmp_seq=2 ttl=64 time=0.118 ms
64 bytes from 10.4.150.57: icmp_seq=3 ttl=64 time=0.117 ms

--- 10.4.150.57 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2084ms
rtt min/avg/max/mdev = 0.117/0.119/0.124/0.003 ms

[root@rh3 ~]# ping -c 3 10.4.150.1 <== Improvement
PING 10.4.150.1 (10.4.150.1) 56(84) bytes of data.
64 bytes from 10.4.150.1: icmp_seq=1 ttl=63 time=0.222 ms
64 bytes from 10.4.150.1: icmp_seq=2 ttl=63 time=0.203 ms
64 bytes from 10.4.150.1: icmp_seq=3 ttl=63 time=0.204 ms

--- 10.4.150.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2083ms
rtt min/avg/max/mdev = 0.203/0.209/0.222/0.008 ms
[root@rh3 ~]# ping -c 3 10.4.1.1 <== FAILURE
PING 10.4.1.1 (10.4.1.1) 56(84) bytes of data.
^C
--- 10.4.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2059ms

The goal is to allow the nodes in the 10.4.160.0/24 subnet to get "out" to all of the other nodes in the network and ultimately the internet.

I am pretty sure that I am missing something on the server dick in the configuration files. H have NetworkManager running, dnsmasq running, firewalld disabled and kernel forwarding enabled.

Interesting to note, when I try to ping rh3 (10.4.160.35) from server B, this is the result that I am getting:

[Root@serverB ~] ping 10.4.160.35
PING 10.4.160.35 (10.4.160.35) 56(84) bytes of data.
From 10.1.3.1: icmp_seq=1 Redirect Host(New nexthop: 10.1.3.27)
From 10.1.3.1 icmp_seq=1 Redirect HostFrom 10.1.3.1: icmp_seq=2 Redirect Host(New nexthop: 10.1.3.27)
From 10.1.3.1 icmp_seq=2 Redirect HostFrom 10.1.3.1: icmp_seq=3 Redirect Host(New nexthop: 10.1.3.27)
From 10.1.3.1 icmp_seq=3 Redirect HostFrom 10.1.3.1: icmp_seq=4 Redirect Host(New nexthop: 10.1.3.27)
From 10.1.3.1 icmp_seq=4 Redirect HostFrom 10.1.3.1: icmp_seq=5 Redirect Host(New nexthop: 10.1.3.27)
From 10.1.3.1 icmp_seq=5 Redirect HostFrom 10.1.3.1: icmp_seq=6 Redirect Host(New nexthop: 10.1.3.27)
From 10.1.3.1 icmp_seq=6 Redirect Host^C
--- 10.4.160.35 ping statistics ---
6 packets transmitted, 0 received, +6 errors, 100% packet loss, time 5000ms

ping from server dick to rh3 is fine:

[root@dick etc]# ping -c 5 rh3
PING rh3 (10.4.160.35) 56(84) bytes of data.
64 bytes from rh3 (10.4.160.35): icmp_seq=1 ttl=64 time=0.125 ms
64 bytes from rh3 (10.4.160.35): icmp_seq=2 ttl=64 time=0.121 ms
64 bytes from rh3 (10.4.160.35): icmp_seq=3 ttl=64 time=0.122 ms
64 bytes from rh3 (10.4.160.35): icmp_seq=4 ttl=64 time=0.120 ms
64 bytes from rh3 (10.4.160.35): icmp_seq=5 ttl=64 time=0.118 ms

--- rh3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 95ms
rtt min/avg/max/mdev = 0.118/0.121/0.125/0.007 ms


What am I missing?

Thank you in advance for your help!
--- Harold

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Packet forwarding issue with Centos 8 system.

Post by jlehtone » 2020/12/09 21:09:49

hcook1 wrote:
2020/12/09 18:38:22

Code: Select all

                        <corp net>---<server A>---<server B>---<server dick>---<node rh3>
upstream address:                     10.1.3.27    10.4.1.27    10.4.150.57    10.4.160.35
Downstream address:                   10.4.1.1     10.4.150.1   10.4.160.1      N/A
[root@rh3 ~]# ping -c 3 10.4.1.1 <== FAILURE
PING 10.4.1.1 (10.4.1.1) 56(84) bytes of data.
^C
--- 10.4.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2059ms

I am pretty sure that I am missing something on the server dick in the configuration files. H have NetworkManager running, dnsmasq running, firewalld disabled and kernel forwarding enabled.

Interesting to note, when I try to ping rh3 (10.4.160.35) from server B, this is the result that I am getting:

[Root@serverB ~] ping 10.4.160.35
PING 10.4.160.35 (10.4.160.35) 56(84) bytes of data.
From 10.1.3.1: icmp_seq=1 Redirect Host(New nexthop: 10.1.3.27)
From 10.1.3.1 icmp_seq=1 Redirect HostFrom 10.1.3.1: icmp_seq=2 Redirect Host(New nexthop: 10.1.3.27)
There is no problem in the dick.

Run on serverB and serverA:

Code: Select all

ip route get 10.4.160.35
On serverB it should answer: via 10.4.150.57
On serverA it should answer: via 10.4.1.27

You can also run in rh3:

Code: Select all

traceroute 8.8.8.8
That should show first steps, but then fail when it does reach routers that do not know where to send the replies to.

hcook1
Posts: 5
Joined: 2020/12/08 00:12:00

Re: Packet forwarding issue with Centos 8 system.

Post by hcook1 » 2020/12/10 23:36:30

Awesome! Almost resolved.

By adding the correct routes to all of the servers, I can now ping into and out of 10.4.160.0/24. However, I still have one troubling issue.
traceroute from rh3 fails! here are two examples:

Code: Select all

[root@rh3 etc]# traceroute 10.4.150.57
traceroute to 10.4.150.57 (10.4.150.57), 30 hops max, 60 byte packets
 1  dick (10.4.150.57)  0.540 ms !X  0.505 ms !X  0.470 ms !X
 
[root@rh3 etc]# traceroute 10.4.1.1
traceroute to 10.4.1.1 (10.4.1.1), 30 hops max, 60 byte packets
 1  rh25 (10.4.160.1)  0.555 ms  0.518 ms  0.484 ms
 2  rh25 (10.4.160.1)  0.450 ms !X  0.416 ms !X  0.381 ms !X
Any ideas on what is causing the !X???

Note that traceroute from any of the servers works as expected.

Any thoughts?
--- Harold

hcook1
Posts: 5
Joined: 2020/12/08 00:12:00

Re: Packet forwarding issue with Centos 8 system.

Post by hcook1 » 2020/12/10 23:47:08

RESOLVED!!!!

I found that the firewall on dick had accidentally been re-enabled...

Disabling the firewall, now all of the functions I need can function!

Thank you for all of your help!

mkhomo
Posts: 4
Joined: 2021/03/24 06:45:22

Re: Packet forwarding issue with Centos 8 system.

Post by mkhomo » 2021/07/11 21:44:58

Got to this question for the same subject title problem.
I wonder if there's a resolution with firewalld still enabled (on host "dick" to protect against forwarded external interface subnet traffic)?

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Packet forwarding issue with Centos 8 system.

Post by jlehtone » 2021/07/11 23:17:28

Firewalld.service in CentOS 8 is not built for forwarding traffic. Not impossible though, but not feasible.

nftables.service and hand-written ruleset is the simpler solution.

Post Reply