Weird Network Behavior - Reachable from some hosts only

Issues related to configuring your network
Post Reply
backtolinux
Posts: 3
Joined: 2020/06/20 02:41:57

Weird Network Behavior - Reachable from some hosts only

Post by backtolinux » 2021/08/10 23:23:52

Hello,

I have been struggling with a weird thing I have never seen in my life. My ethernet (tg3 driver) is not reachable from certain hosts on a network while it is visible from some others.

Just to clarify: I have a larger network (192.168.0.0/24, Network A) and a smaller one (192.168.44.0/24, Network B). My secondary router (192.168.44.1, Router B) is connected to the larger network, and my primary router (192.168.0.1, Router A) is connected to the internet. Ideally, I would use my CentOS station at Network B, but for testing purposes I tried to put it on Network A too. Both routers have a DHCP rule to set my the station's ethernet the .2-ending IP (and the Wi-Fi to the .3-ending IP, also in both routers).

The DHCP is working fine in both cases, meaning it can reach at some degree both routers. However, after the IP is attributed no communications between router and station will happen, pinging Router B doesn't work when connected to Network B. Pinging the station from the router gets the same results, as if it was disconnected.

But perhaps when I run the test from another station, it reaches and the networking works pretty fine. I am able to connect to services like httpd, sshd, and others, etc. I thought it could be a firewall-related issue, but it's not, as I tested stopping the firewalld service. I thought it could be the router, so I tested on Network B and still no communication to the router, and still communicating with other station.

I tried activating Wi-Fi, so I could run a full update (I'm running the latest CentOS release, 8.4.2105) and test it. After changing default gateway Wi-Fi works perfect at both networks. However, I'd rather prefer using ethernet as I have a better ethernet device than my Wi-Fi.

Some information:
ifconfig cmd output:

Code: Select all

[root@blackcat ~]# ifconfig
enp1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.44.2  netmask 255.255.255.0  broadcast 192.168.44.255
        inet6 fe80::221a:6ff:fe55:1667  prefixlen 64  scopeid 0x20<link>
        ether 20:1a:06:55:16:67  txqueuelen 1000  (Ethernet)
        RX packets 3689  bytes 782775 (764.4 KiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 2682  bytes 281485 (274.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 3497  bytes 625089 (610.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3497  bytes 625089 (610.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.66.1  netmask 255.255.255.0  broadcast 192.168.66.255
        ether 52:54:00:fe:f0:6a  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.166.49  netmask 255.255.255.224  broadcast 192.168.166.63
        ether 52:54:00:b7:8e:90  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.44.3  netmask 255.255.255.0  broadcast 192.168.44.255
        inet6 fe80::b2cf:4e35:84a1:14c9  prefixlen 64  scopeid 0x20<link>
        ether 48:d2:24:75:ac:bc  txqueuelen 1000  (Ethernet)
        RX packets 3384  bytes 745553 (728.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4377  bytes 654258 (638.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
ifcfg-enp1s0f0:

Code: Select all

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp1s0f0
UUID=cfdf3faa-3cc2-43a6-989a-469f03b97018
DEVICE=enp1s0f0
ONBOOT=yes
PEERDNS=no
DNS1=192.168.44.2
DNS2=192.168.44.1
DNS3=192.168.0.1
ZONE=public
PS: DEFROUTE defined to 'no' after Wi-Fi was activated. It was originally set to 'yes'.

ifcfg-WLAN:

Code: Select all

ESSID="----------------------"
MODE=Managed
MAC_ADDRESS_RANDOMIZATION=default
TYPE=Wireless
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=TighterWLAN
UUID=9309b376-33f7-4040-970e-b520cd03ae7c
DEVICE=wlp2s0
ONBOOT=yes
KEY_MGMT=WPA-PSK
PS: SSID intentionally omitted.

nmcli con output:

Code: Select all

NAME         UUID                                  TYPE      DEVICE
WLAN       9309b376-33f7-4040-970e-b520cd03ae7c  wifi      wlp2s0
enp1s0f0     cfdf3faa-3cc2-43a6-989a-469f03b97018  ethernet  enp1s0f0
virbr0       665196f2-d0ee-4912-97a2-51077dc45cfc  bridge    virbr0
virbr1       9ecb0875-f26e-41ad-ba43-81ce004dbaed  bridge    virbr1
I appreciate any help with that issue.

Best regards,
Bruno Moreira-Guedes

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Weird Network Behavior - Reachable from some hosts only

Post by jlehtone » 2021/08/11 17:46:48

What do you get with these when you have wire connected?

Code: Select all

nmcli con s enp1s0f0
ip ro
Why does enp1s0f0 set name-servers? Doesn't the DHCP hand that info?


PS. Your A and B have same size, /24.

backtolinux
Posts: 3
Joined: 2020/06/20 02:41:57

Re: Weird Network Behavior - Reachable from some hosts only

Post by backtolinux » 2021/08/11 19:47:13

jlehtone wrote:
2021/08/11 17:46:48
What do you get with these when you have wire connected?

Code: Select all

nmcli con s enp1s0f0
ip ro
Here's the result of the nmcli con s enp1s0f0:

Code: Select all

connection.id:                          enp1s0f0
connection.uuid:                        cfdf3faa-3cc2-43a6-989a-469f03b97018
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              enp1s0f0
connection.autoconnect:                 yes
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               0 (default)
connection.auth-retries:                -1
connection.timestamp:                   1628710608
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        public
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
connection.llmnr:                       -1 (default)
connection.wait-device-timeout:         -1
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             --
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
ipv4.method:                            auto
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.addresses:                         --
ipv4.gateway:                           --
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.routing-rules:                     --
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   yes
ipv4.dhcp-client-id:                    --
ipv4.dhcp-iaid:                         --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.dhcp-hostname-flags:               0x0 (none)
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv4.dhcp-vendor-class-identifier:      --
ipv4.dhcp-reject-servers:               --
ipv6.method:                            ignore
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.routing-rules:                     --
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     yes
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.ra-timeout:                        0 (default)
ipv6.dhcp-duid:                         --
ipv6.dhcp-iaid:                         --
ipv6.dhcp-timeout:                      0 (default)
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.dhcp-hostname-flags:               0x0 (none)
ipv6.token:                             --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           enp1s0f0
GENERAL.UUID:                           cfdf3faa-3cc2-43a6-989a-469f03b97018
GENERAL.DEVICES:                        enp1s0f0
GENERAL.IP-IFACE:                       enp1s0f0
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        yes
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/3
GENERAL.ZONE:                           public
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         192.168.44.2/24
IP4.GATEWAY:                            192.168.44.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.44.1, mt = 100
IP4.ROUTE[2]:                           dst = 192.168.44.0/24, nh = 0.0.0.0, mt = 100
DHCP4.OPTION[1]:                        dhcp_lease_time = 4294967295
DHCP4.OPTION[2]:                        dhcp_server_identifier = 192.168.44.1
DHCP4.OPTION[3]:                        domain_name_servers = 192.168.44.1
DHCP4.OPTION[4]:                        ip_address = 192.168.44.2
DHCP4.OPTION[5]:                        requested_broadcast_address = 1
DHCP4.OPTION[6]:                        requested_domain_name = 1
DHCP4.OPTION[7]:                        requested_domain_name_servers = 1
DHCP4.OPTION[8]:                        requested_domain_search = 1
DHCP4.OPTION[9]:                        requested_host_name = 1
DHCP4.OPTION[10]:                       requested_interface_mtu = 1
DHCP4.OPTION[11]:                       requested_ms_classless_static_routes = 1
DHCP4.OPTION[12]:                       requested_nis_domain = 1
DHCP4.OPTION[13]:                       requested_nis_servers = 1
DHCP4.OPTION[14]:                       requested_ntp_servers = 1
DHCP4.OPTION[15]:                       requested_rfc3442_classless_static_routes = 1
DHCP4.OPTION[16]:                       requested_root_path = 1
DHCP4.OPTION[17]:                       requested_routers = 1
DHCP4.OPTION[18]:                       requested_static_routes = 1
DHCP4.OPTION[19]:                       requested_subnet_mask = 1
DHCP4.OPTION[20]:                       requested_time_offset = 1
DHCP4.OPTION[21]:                       requested_wpad = 1
DHCP4.OPTION[22]:                       routers = 192.168.44.1
DHCP4.OPTION[23]:                       subnet_mask = 255.255.255.0
IP6.ADDRESS[1]:                         fe80::221a:6ff:fe55:1667/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 256
IP6.ROUTE[2]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
And here's the result of the ip ro:

Code: Select all

default via 192.168.44.1 dev enp1s0f0 proto dhcp metric 100
192.168.44.0/24 dev enp1s0f0 proto kernel scope link src 192.168.44.2 metric 100
192.168.44.0/24 dev wlp2s0 proto kernel scope link src 192.168.44.3 metric 600
192.168.66.0/24 dev virbr0 proto kernel scope link src 192.168.66.1 linkdown
192.168.166.32/27 dev virbr1 proto kernel scope link src 192.168.166.49 linkdown
Is there anything I'm not seeing?
jlehtone wrote:
2021/08/11 17:46:48
Why does enp1s0f0 set name-servers? Doesn't the DHCP hand that info?
Not in the intended way, I have a local DNS for specific purposes. But I have commented the DNS lines right now, for testing.
jlehtone wrote:
2021/08/11 17:46:48
PS. Your A and B have same size, /24.
Thanks for pointing out, I will decrease B soon.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Weird Network Behavior - Reachable from some hosts only

Post by jlehtone » 2021/08/11 20:26:19

backtolinux wrote:
2021/08/11 19:47:13
Not in the intended way, I have a local DNS for specific purposes. But I have commented the DNS lines right now, for testing.
That explains why:

Code: Select all

DNS1=192.168.44.2
DNS2=192.168.44.1
DNS3=192.168.0.1
differs from:

Code: Select all

ipv4.dns:                               --
However, these two say the same:

Code: Select all

PEERDNS=no

ipv4.ignore-auto-dns:                   yes
Note how in the output of nmcli con s enp1s0f0 the lowercase attributes are stored configuration and uppercase are the active state.

Code: Select all

IP4.ADDRESS[1]:                         192.168.44.2/24
IP4.GATEWAY:                            192.168.44.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.44.1, mt = 100
IP4.ROUTE[2]:                           dst = 192.168.44.0/24, nh = 0.0.0.0, mt = 100
Your handmade DNS list has 192.168.44.2 as first. That is localhost. Could use 127.0.0.1. If you do run DNS server in this machine, does it listen on both addresses? Can it listen 192.168.44.2 explicitly, if it starts before NM has got the address from DHCP?
You do know that glibc resolver attempts to get answer from first server for every query before trying the next server?

The second handmade address, 192.168.44.1, is same as what you would get from DHCP:

Code: Select all

DHCP4.OPTION[3]:                        domain_name_servers = 192.168.44.1
If the inner router has 192.168.0.1 as its DNS server (which it probably does if it gets its "wan 192.168.0.x address" from the outer 192.168.0.1 router), then the DNS3 seems unnecessary.


The real worrybit is:

Code: Select all

default via 192.168.44.1 dev enp1s0f0 proto dhcp metric 100
192.168.44.0/24 dev enp1s0f0 proto kernel scope link src 192.168.44.2 metric 100
192.168.44.0/24 dev wlp2s0 proto kernel scope link src 192.168.44.3 metric 600
Do you have the Wifi enabled while testing wired?


An another test method is to listen to the interface to see if there is any traffic, particularly when you try to ping. For example:

Code: Select all

tcpdump -nn -vv -i enp1s0f0

backtolinux
Posts: 3
Joined: 2020/06/20 02:41:57

Re: Weird Network Behavior - Reachable from some hosts only

Post by backtolinux » 2021/08/12 00:03:15

First of all, thanks @jlehtone for your time and kind replies.

I have solved the problem after I tested again on Network A and concluded I have likely made a poor test at the first time (as it worked this time, without meaningful changes, so I think it's fair to assume I somehow failed to test it properly). After disconnecting Router B and using the wire to connect to the CentOS station I noticed my SSH connection was still alive, and the machine was still responding to pings at 192.168.44.2, even when the enp1s0f0 IP, obtained from DHCP, was set to 192.160.0.2.

I'm still trying to figure out what happened precisely, and unfortunately I didn't run a tcpdump at the moment, but the only possible explanation I can imagine is that, somehow, the WiFi card was responding for both .2 and .3 IPs (maybe something wrong with my router's ARP table? I'm not sure). I concluded it might have been due to something at Router B, I checked its settings and noticed there was two entries with the same hostname at the DHCP reservation list. I think this router might be a little bit 'messed up', I tried just removing the duplicate entry leaving only the .2 reserved. At first it didn't work, but it worked after I reset the router. Everything's working fine since then.
jlehtone wrote:
2021/08/11 20:26:19
Note how in the output of nmcli con s enp1s0f0 the lowercase attributes are stored configuration and uppercase are the active state.

Code: Select all

IP4.ADDRESS[1]:                         192.168.44.2/24
IP4.GATEWAY:                            192.168.44.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.44.1, mt = 100
IP4.ROUTE[2]:                           dst = 192.168.44.0/24, nh = 0.0.0.0, mt = 100
Thanks for the hint, I am quite 'outdated' as I used Linux long before NM became 'mandatory'.
jlehtone wrote:
2021/08/11 20:26:19
Your handmade DNS list has 192.168.44.2 as first. That is localhost. Could use 127.0.0.1. If you do run DNS server in this machine, does it listen on both addresses? Can it listen 192.168.44.2 explicitly, if it starts before NM has got the address from DHCP?
You do know that glibc resolver attempts to get answer from first server for every query before trying the next server?
Indeed I am aware. Besides relay, my named service runs some .local domains and I am often editing configs, so it's the first entry, but I use the others for moments I eventually break something.
jlehtone wrote:
2021/08/11 20:26:19
The second handmade address, 192.168.44.1, is same as what you would get from DHCP:

Code: Select all

DHCP4.OPTION[3]:                        domain_name_servers = 192.168.44.1
If the inner router has 192.168.0.1 as its DNS server (which it probably does if it gets its "wan 192.168.0.x address" from the outer 192.168.0.1 router), then the DNS3 seems unnecessary.
Indeed, that's the case, thanks for pointing out again, I removed DNS3 as it was really unnecessary!
jlehtone wrote:
2021/08/11 20:26:19
The real worrybit is:

Code: Select all

default via 192.168.44.1 dev enp1s0f0 proto dhcp metric 100
192.168.44.0/24 dev enp1s0f0 proto kernel scope link src 192.168.44.2 metric 100
192.168.44.0/24 dev wlp2s0 proto kernel scope link src 192.168.44.3 metric 600
Do you have the Wifi enabled while testing wired?
I thoght if I set only DEFROUTE=yes only on ethernet it should not be a problem, but now I'm questioning.
The WiFi was indeed part of the problem. I had this station disabled for a month, as I was very busy, so I can't remember 100% sure. As far as I remember, I just enabled WiFi to run some updates as the ethernet was not working, but now I question whether that's really true.
jlehtone wrote:
2021/08/11 20:26:19
An another test method is to listen to the interface to see if there is any traffic, particularly when you try to ping. For example:

Code: Select all

tcpdump -nn -vv -i enp1s0f0
Thanks again, excellent hint, I will do some tests when I have the courage to try to simulate the problem again (I still want to figure out).

Best regards!

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Weird Network Behavior - Reachable from some hosts only

Post by jlehtone » 2021/08/12 10:53:08

backtolinux wrote:
2021/08/12 00:03:15
Indeed I am aware. Besides relay, my named service runs some .local domains and I am often editing configs, so it's the first entry, but I use the others for moments I eventually break something.
Note about NM. It supports many resolvers. See "dns" in man NetworkManager.conf.
Resolvers other than glibc are "smarter".

I have used dns=dnsmasq. I have a new file:

Code: Select all

$ cat /etc/NetworkManager/conf.d/00-dns.conf 
# ansible #
[main]
dns=dnsmasq
That makes NM start instance of dnsmasq. It is possible to customize some of its config, but the
"fancy part" is that NM writes nameserver 127.0.0.1 into /etc/resolv.conf.
Which upstream DNS does the dnsmasq query? NM gives connections' DNS (static or DHCP)
to dnsmasq via DBus.

The dnsmasq is quite easy to configure. (Ok, misconfigurations are possible too.)

Post Reply