Guest machine can't see the network

[Iucharbius]

Hi,

I've got a TrueNAS guest VM in my CentOS 9 Stream that can't see the network. I'll put relevant details in below:

Here are my bridges:

[iucharbius@localhost ~]$ brctl show
bridge name bridge id STP enabled interfaces
br0 8000.1402ec8a6420 yes eno49
vnet0
docker0 8000.02423ebaf669 no veth18321c6
virbr0 8000.525400d0ca01 yes


Here is eno49.nmconnection:

[connection]
id=eno49
uuid=a4333c11-a122-4017-9ec4-8085f2aee4e0
type=ethernet
interface-name=eno49
master=br0
slave-type=bridge

[ethernet]

[bridge-port]


Here's eno49 and br0 from ifconfig:

eno49: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 14:02:ec:8a:64:20 txqueuelen 1000 (Ethernet)
RX packets 129036 bytes 16465615 (15.7 MiB)
RX errors 0 dropped 751 overruns 0 frame 0
TX packets 16848 bytes 1127609 (1.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.5.30 netmask 255.255.255.0 broadcast 192.168.5.255
inet6 fe80::ed95:8bee3561 prefixlen 64 scopeid 0x20<link>
ether 14:02:ec:8a:64:20 txqueuelen 1000 (Ethernet)
RX packets 115028 bytes 12838026 (12.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2178 bytes 310416 (303.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0


virsh net-edit for the vm's network adaptor ("bridged-network"):

<network>
<name>bridged-network</name>
<uuid>3a5ed8a5-fe11-4ec5-bd7e-0b3a9411dea7</uuid>
<forward mode='bridge'/>
<bridge name='br0'/>
</network>


Here's the NIC from virsh edit:

<interface type='network'>
<mac address='52:54:00:85:9c:67'/>
<source network='bridged-network'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>


Lastly, if I look at the NIC in the GUI, I see that it has been assigned an IP address (and I can find that IP address in the correct network on my router). Pings to that IP from the same network - or any other network - go unreturned. The weird thing is that it worked for a few days after I set it up, being done in by a server reboot.

Any idea where I can start looking? Is there anything that I can provide to help with diagnosing this?

Thanks,

-Bill

[jlehtone]

So it is in the same 192.168.5.0/24 subnet as the br0. Can the host ping the guest?

If you do listen on br0 (with tcpdump), do you see pings sent from outside? Pings sent from the guest?

You could disable STP from br0.

[Iucharbius]

It should be the same 192.168.5.0/24 as br0, but I'm not sure how to test it.

Okay, I've tried a tcpdump both before and after disabling STP from br0 and I'm getting nothing from the guest to the host. I spun up a Mint instance and it is behaving the same way.

The adaptor in the guests are not actually getting ipv4 addresses from DHCP. It's weird because I can see them as client devices in Unifi but they have no IP addresses there either.

I think I'm missing something pretty obvious, but I can't for the life of me figure out what it is.

[TrevorH]

Your VM xml seems to be missing

<source bridge='br0'/>

or things to that effect.

[Iucharbius]

Where would I put that? If I look at the xml for the connection with VIRSH NET-EDIT bridged-network, I see this:

<network>
<name>bridged-network</name>
<uuid>3a5ed8a5-fe11-4ec5-bd7e-0b3a9411dea7</uuid>
<forward mode='bridge'/>
<bridge name='br0'/>
</network>


Should there also be a "source bridge" tag?

[TrevorH]

This is a snippet from virsh edit $vm on one of mine

Code: Select all

    <interface type='bridge'>
      <mac address='52:54:00:xx:xx:xx'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

[Iucharbius]

That's very helpful - thank you. Do you only have an interface type='bridge' or is this in addition to having an interface type='network'?

[TrevorH]

No mention of 'network' in virsh dumpml $vm at all

[Iucharbius]

I've tried a few more things. On the host, I've blown up all of the network adapters, blown up the bridge, and re-created everything. I've removed the network connection from the guest and re-added it. I've then gone into the guest and tried to re-setup the network interfaces. The net result of this was that there was no change.

I rechecked the tcpdump (sudo tcpdump -i <adapter name> -v icmp) and pinged it from my laptop. Interesting results:

pinging 192.168.5.30 echoed but did not return anything in tcpdump in either eno49 (the interface for br0) or br0. It did, however, show up in the tcpdump for eno50 (which has no connection whatsoever to br0). This held true when I pinged the other four NICs in the machine (192.168.5.67 - 192.168.5.70).

Pinging 192.168.5.31 echoed and showed up in the tcpdump for eno50 as expected.

Why would eno50 be answering pings that went to an ipv4 address that is connected to another adapter?

If it helps, my routing table looks like this:

[iucharbius@kawadomek ~]$ ip r
default via 192.168.5.1 dev eno50 proto dhcp src 192.168.5.31 metric 100
default via 192.168.5.1 dev eno1 proto dhcp src 192.168.5.67 metric 102
default via 192.168.5.1 dev eno2 proto dhcp src 192.168.5.68 metric 103
default via 192.168.5.1 dev eno3 proto dhcp src 192.168.5.69 metric 104
default via 192.168.5.1 dev eno4 proto dhcp src 192.168.5.70 metric 105
default via 192.168.5.1 dev br0 proto dhcp src 192.168.5.30 metric 425
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.5.0/24 dev eno50 proto kernel scope link src 192.168.5.31 metric 100
192.168.5.0/24 dev eno1 proto kernel scope link src 192.168.5.67 metric 102
192.168.5.0/24 dev eno2 proto kernel scope link src 192.168.5.68 metric 103
192.168.5.0/24 dev eno3 proto kernel scope link src 192.168.5.69 metric 104
192.168.5.0/24 dev eno4 proto kernel scope link src 192.168.5.70 metric 105
192.168.5.0/24 dev br0 proto kernel scope link src 192.168.5.30 metric 425
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

[jlehtone]

That routing table looks like a there is a total mess in config. I would have expected:

Code: Select all

default via 192.168.5.1 dev br0 proto dhcp src 192.168.5.30 metric 425
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.5.0/24 dev br0 proto kernel scope link src 192.168.5.30 metric 425

Do you really have five interfaces on the same subnet?
More than one IP address on same subnet is not trivial and calls for policy routing.


I don't create any bridges with libvirt's virsh net-edit.
I do create bridge on the host with nmcli (see man nmcli-examples) before I even install libvirt.
Therefore, the bridge exists before libvirtd starts.

For a VM the virsh dumpxml output has then a section, like:

Code: Select all

    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
    </interface>