Virtualization routing problem [SOLVED]

[simon@bp]

I have a problem with networking/routing on KVM guests. This started after an update and reboot.

The first problem I noticed was that the virt-manager could not connect and virsh commands hung, so I decided to update the machine and reboot.

After this, the VMs had problems with networking. I can see the DHCP requests on the DHCP server, but the machines don't seem to be able to receive or configure the IP address (I don't know which).

If I configure an IP address and route manually, they can ping the default gateway, they can also ping their own IP address and that of the virtualization host, but no other machines in the LAN. Oddly, it looks like some of the windows machines were able to get an IP address, but a few minutes later are unable to renew it.

I am most puzzled by the fact that, if I manually configure an IP address, the VM can ping the gateway, but not other IP addresses on the LAN.

I just rebooted one of the Windows guests and I could ping it from the default gateway briefly..

The VMs are networked using br0 (a bridge device) which is a bond with 2 physical interfaces. If I try to connect the VM to either of the underlying physical devices, I get this:

Error starting domain: error creating macvtap interface macvtap0@eno1 (52:54:00:3e:00:6f): Device or resource busy.

I tried converting one of the VMs to use NAT and it seems to work, as long as firewalld is running, but this doesn't solve my problem: I need to use bridge networking for the guests.

On the host, I only have the default set of firewall rules, which appears to work on other systems.

[simon@bp]

Update:

From one of the VMs:

Code: Select all

# ip route
default via 192.168.254.1 dev eth0 
169.254.0.0/16 dev eth0 scope link metric 1002 
192.168.254.0/24 dev eth0 proto kernel scope link src 192.168.254.112 

# ping 192.168.254.1 # The gateway
PING 192.168.254.1 (192.168.254.1) 56(84) bytes of data.
64 bytes from 192.168.254.1: icmp_seq=1 ttl=64 time=1.14 ms
64 bytes from 192.168.254.1: icmp_seq=2 ttl=64 time=0.539 ms
^C
# ping 192.168.254.10 # another machine in the LAN. 
PING 192.168.254.10 (192.168.254.10) 56(84) bytes of data.
From 192.168.254.112 icmp_seq=1 Destination Host Unreachable
From 192.168.254.112 icmp_seq=2 Destination Host Unreachable
From 192.168.254.112 icmp_seq=3 Destination Host Unreachable
From 192.168.254.112 icmp_seq=4 Destination Host Unreachable

# ping 192.168.254.75 # br0 on the host. 
PING 192.168.254.75 (192.168.254.75) 56(84) bytes of data.
64 bytes from 192.168.254.75: icmp_seq=1 ttl=64 time=0.409 ms
64 bytes from 192.168.254.75: icmp_seq=2 ttl=64 time=0.130 ms

I wonder if one of the interfaces that make up the bond on the host has a hardware issue. ethtool shows both interfaces as up, with a link.

[simon@bp]

And now, a few minutes later, from the same VM that failed to ping the .10 address:

Code: Select all

# ping 192.168.254.10
PING 192.168.254.10 (192.168.254.10) 56(84) bytes of data.
64 bytes from 192.168.254.10: icmp_seq=1 ttl=64 time=0.722 ms
64 bytes from 192.168.254.10: icmp_seq=2 ttl=64 time=0.445 ms
64 bytes from 192.168.254.10: icmp_seq=3 ttl=64 time=0.443 ms
^C

[simon@bp]

Problem was solved by taking down one of the interfaces that make up the bond. I think the problem is related to the two interfaces in the bond being connected to different switches.

[jlehtone]

The status of a bond should be readable with
cat /proc/net/bonding/bond0
(assuming the device is named "bond0").

I've used only LACP-mode bonds -- all links to same switch -- so can't tell how bond on host and the switches should be configured (although links via separate devices should be a sensible HA solution).

[simon@bp]

The status of a bond should be readable with
cat /proc/net/bonding/bond0
(assuming the device is named "bond0").

I've used only LACP-mode bonds -- all links to same switch -- so can't tell how bond on host and the switches should be configured (although links via separate devices should be a sensible HA solution).

I looked at that file and everything looked normal (as far as I could tell, I'm not really an expert on this).

What I don't understand is why the bonding setup only affected the bridged virtual guests and not the host itself or the guest with a NAT setup.

This is what it looked like before taking down one of the interfaces that make up the bond (enp24s0f0).

Code: Select all

# more /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eno1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:ec:ef:5f:18:50
Slave queue ID: 0

Slave Interface: enp24s0f0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b4:96:91:8a:6c:5e
Slave queue ID: 0

[TrevorH]

What's the output from brctl show

[jlehtone]

The public IP of the host is on the bridge, is it?

What I don't understand is why the bonding setup only affected the bridged virtual guests and not the host itself or the guest with a NAT setup.

The "guest with NAT" means that the guest is on separate virtual subnet (usually "virbr0") and the host acts as router between that subnet and outside.
That is why "only the host" talks to outside, not the guest.

Why did host manage to use the "good link", while bridged guests got the "bad link"?
I don't know, as the "round robin" should use all links even when the host talks?