Page 1 of 2

VLANs with QEMU/KVM

Posted: 2020/05/17 14:48:26
by LinuxLand
I have a server with two NICs. One is dedicated for QEMU/KVM. I have a few guests I'd like on one of two VLANS (192.168.10.0/24 and 192.168.20/24). What are the options for configuring this?

I've tried creating virtual interfaces for each vlan on enp2s0 (enp2s0:10, enp2s0:20). I added a virtual bridge and made enp2s0 an interface on the virtual bridge vbridge00. I did not put an IP on enp2s0, just on the sub interfaces enp2s0:10 and enp2s0:20 *.10.3 and *.20.3 respectively. I made the physical switch port a trunk with allowed vlans 10,20. I can hit the *.10.3 address but not the *.20.3 address. Also, I cannot use the interfaces in KVM. They show up as macvtap interfaces and KVM complains that the resource is busy when I start the machine.

Most of the examples and configuration guides on redhat seem to focus on different scenarios and I'm not convinced they will work in this case.

I appreciate any insight you might have!

Thanks!

Re: VLANs with QEMU/KVM

Posted: 2020/05/17 16:09:47
by jlehtone
LinuxLand wrote:
2020/05/17 14:48:26
I have a server with two NICs. One is dedicated for QEMU/KVM.
Good. The server has one IP on one NIC. What to do with the second NIC?

Code: Select all

VM---bridge10---vlan10===enp2s0===switch
VM---bridge20---vlan20==/
=== tagged traffic
--- untagged traffic
I would:

Code: Select all

# Remove existing connections from second NIC
...
# Create bridge for VLAN 10
nmcli con add type bridge con-name bridge-br10 ifname br_vlan10 bridge.stp no ipv6.method ignore ipv4.method disabled
nmcli con show
# Connect bridge to VLAN 10
nmcli con add type vlan con-name vlan-10 dev enp2s0 id 10 connection.slave-type bridge connection.master *UUID* ipv6.method ignore ipv4.method disabled bridge.stp no

# Create bridge for VLAN 20
nmcli con add type bridge con-name bridge-br20 ifname br_vlan20 bridge.stp no ipv6.method ignore ipv4.method disabled
nmcli con show
# Connect bridge to VLAN 20
nmcli con add type vlan con-name vlan-20 dev enp2s0 id 20 connection.slave-type bridge connection.master *UUID* ipv6.method ignore ipv4.method disabled bridge.stp no

Re: VLANs with QEMU/KVM

Posted: 2020/05/25 12:04:53
by LinuxLand
Thanks for posting this. Conceptually this is pretty much what I imagined it should be. However, this configuration doesn't work.

I deleted all the configuration for enp2s0 and followed the commands adding uuid's where necessary. The *.10.* network works, the *.20.* doesn't. I suspect the 10 vlan is getting through because enp1s0 is configured with that address--though it is untagged on that physical port (access rather than trunk).

I did try adding IP's to the bridges like I have with the single vlan configuration working now--but that didn't do anything.

Is there anyway to set this up with ifcfg files instead of nm?

Re: VLANs with QEMU/KVM

Posted: 2020/05/25 15:08:06
by jlehtone
LinuxLand wrote:
2020/05/25 12:04:53
I suspect the 10 vlan is getting through because enp1s0 is configured with that address--though it is untagged on that physical port (access rather than trunk).
That should not be possible.

If you did set things as expected, then
  • The host has IP on enp1s0. If that wire carries untagged vlan 10, then that is what the host sees.
  • The host has no IP address on enp2s0, vlan-10, vlan-20, bridge-br10, or bridge-br20.
  • Tagged traffic of vlan 10 and vlan 20 comes and leaves enp2s0
  • VM attached to bridge-br10 sees untagged traffic of vlan 10
  • VM attached to bridge-br20 sees untagged traffic of vlan 20
You can watch what happens at enp2s0 in the host:

Code: Select all

tcpdump -i enp2s0 -nn -e vlan
You can show what ports are attached to which bridges on the host with:

Code: Select all

brctl show
You can show what the active network config is with:

Code: Select all

nmcli
nmcli d s
nmcli c s
ip ro

A fundamental difference between CLI and GUI is that user can write whatever on CLI, but on GUI the UI limits options to what is (hopefully) valid. Alas, almost every UI allows nonsense; valid syntax is no guarantee of logic sanity.

Editing configuration directly in files requires that user writes the "whatever" in syntax that the programs can use.
The nmcli (and other NetworkManager UI's) write the files in correct syntax. These tools do validate their input.

Re: VLANs with QEMU/KVM

Posted: 2020/05/26 01:02:26
by LinuxLand
Thank you for your response.

I think your configuration is correct, though it is not yet working in my setup. I did use tcpdump as you suggest and I can see the tagged vlan traffic on the interface. However, I can't see anything leaving that interface. I noticed a configuration issue on my router for the trunking piece which I corrected. I would still expect to communicate with my default gateway. Looking at arp tables shows no mac addresses learned on the interface that enp2s0 connects to. I think this may be a small configuration change that I'm missing in either the routing or switching piece.

For the record, enp2s0 connects to a Cisco 2960 switch with the configuration:

Code: Select all

interface FastEthernet0/1
 description hv01 enp2s0
 switchport trunk allowed vlan 1,10,20
 switchport mode trunk
end
The switch is connected to the router with the two gigabit interfaces configured as follows:

Code: Select all

!
interface GigabitEthernet0/1
 switchport trunk allowed vlan 1,10,20
 switchport mode trunk
end
and

Code: Select all

!
interface GigabitEthernet0/2
 switchport trunk allowed vlan 1,10,20
 switchport mode trunk
end
As for spanning tree only gi0/1 is forwarding gi0/2 is disabled.

On the router side of things, I have fa0 and fa1 trunked as follows:

Code: Select all

 !
interface FastEthernet0
 switchport trunk allowed vlan 1,2,10,20,1002-1005
 switchport mode trunk
 no ip address
end
and

Code: Select all

!
interface FastEthernet1
 switchport trunk allowed vlan 1,2,10,20,1002-1005
 switchport mode trunk
 no ip address
end

I think I'm going to simplify this configuration somehow and see if I can get some basics to work. First, I want basic switching. I should see some mac addresses on the switch at L2. Once I can do that I'll work on L3. I do see some potential issues there as well.

Thanks again for your help.

Re: VLANs with QEMU/KVM

Posted: 2020/05/26 06:54:35
by jlehtone
Bridge is a switch. Your network should now have:

Code: Select all

VM---switch(br)===switch(cisco)===router
We did set STP off for the br. It has nothing to send outwards, except the "I have MAC-of-VM" ARP replies (once VM is up).

Does the VM send DHCP requests? You have a DHCP server somewhere in the VLAN?

You can listen bridge and vlan interfaces with tcpdump. While enp2s0 should show tagged traffic of both VLANs, the bridge sees only untagged traffic of one subnet.


For the record, Linux kernel does support vlan filtering on software bridge ports and NetworkManager should support that.
https://developers.redhat.com/blog/2017 ... on-bridge/
See settings for bridge and bridge-port in:

Code: Select all

man nm-settings
I have failed to grok that and therefore presented the older "without VLAN filtering" setup.

Re: VLANs with QEMU/KVM

Posted: 2020/05/26 10:58:01
by LinuxLand
The network you suggested is exactly what I have.

As for setting STP off I have:

Code: Select all

# brctl show
bridge name	bridge id		STP enabled	interfaces
br_vlan10		8000.0025901aa637	no		enp2s0.10
br_vlan20		8000.0025901aa637	no		enp2s0.20
							vnet0
virbr0		8000.5254002ce12b	yes		virbr0-nic
And for clarity, the second command creating the vlan command

Code: Select all

# Connect bridge to VLAN 10
nmcli con add type vlan con-name vlan-10 dev enp2s0 id 10 connection.slave-type bridge connection.master *UUID* ipv6.method ignore ipv4.method disabled bridge.stp no
did not work for me.

Code: Select all

nmcli con add type bridge con-name bridge-br30 ifname br_vlan10 bridge.stp no ipv6.method ignore ipv4.method disabled
nmcli con show

nmcli con add type vlan con-name vlan-30 dev enp2s0 id 10 connection.slave-type bridge connection.master (UUID from bridge-br30) ipv6.method ignore ipv4.method disabled bridge.stp no
Error: invalid or not allowed setting 'ipv6': 'ipv6' not among [connection, vlan, 802-3-ethernet (ethernet), ethtool, bridge-port].

nmcli con add type vlan con-name vlan-30 dev enp2s0 id 10 connection.slave-type bridge connection.master (UUID from bridge-br30)

But that shouldn't matter, unless I'm missing something else here.

The guest vm does send DHCP requests out:

Code: Select all

06:36:15.472056 aa:aa:aa:aa:aa:dd > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 20, p 0, ethertype IPv4, 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:aa:aa:aa:aa:dd, length 300
I tried setting a manual IP as well to test and the guest can't communicate out.

As for using tcpdump to listen on bridge and vlan these show no traffic:

Code: Select all

tcpdump -i br_vlan20 -nn -e vlan

tcpdump -i enp2s0.20 -nn -e vlan
[\code]

Re: VLANs with QEMU/KVM

Posted: 2020/05/26 11:52:13
by jlehtone
LinuxLand wrote:
2020/05/26 10:58:01
The network you suggested is exactly what I have.

As for setting STP off I have:

Code: Select all

# brctl show
bridge name	bridge id		STP enabled	interfaces
br_vlan10		8000.0025901aa637	no		enp2s0.10
br_vlan20		8000.0025901aa637	no		enp2s0.20
							vnet0
virbr0		8000.5254002ce12b	yes		virbr0-nic
Three bridges. One VM (vnet0) that is attached to bridge br_vlan20.

The bridge "virbr0" (and "virbr0-nic") are created by libvirtd, because there is defined a network "default" that is set to autostart.
The host will route traffic between "default" and enp1s0, and masquerade so that VM members of default hide behind host's IP address.
If you will not use that network, then I recommend to disable its autostart so that it won't be recreated on every boot.
LinuxLand wrote:
2020/05/26 10:58:01
And for clarity, the second command creating the vlan command
did not work for me.

Code: Select all

nmcli con add type vlan con-name vlan-30 dev enp2s0 id 10 connection.slave-type bridge connection.master (UUID from bridge-br30) ipv6.method ignore ipv4.method disabled bridge.stp no
Error: invalid or not allowed setting 'ipv6': 'ipv6' not among [connection, vlan, 802-3-ethernet (ethernet), ethtool, bridge-port].
But that shouldn't matter, unless I'm missing something else here.
My bad. Normally, when creating a VLAN connection, it tries to get IPv4 and IPv6 config (automatic, DHCP).
The ipv6.method ignore ipv4.method disabled explicitly says "no" to that.
However, since we at the same time enslave the connection to a bridge with connection.slave-type bridge connection.master (UUID from bridge-br30) the nmcli is clever enough to remove the ipv4 and ipv6 settings entirely. Hence, the explicit "no" is not necessary.

LinuxLand wrote:
2020/05/26 10:58:01
As for using tcpdump to listen on bridge and vlan these show no traffic:

Code: Select all

tcpdump -i br_vlan20 -nn -e vlan
There are no VLANs on the bridge. No tagged traffic. Neither the VM nor the enp2s0.20 on the other port of the bridge do give tagged traffic to the bridge.

Code: Select all

tcpdump -i br_vlan20 -nn
Should show the aa:aa:aa:aa:aa:dd > ff:ff:ff:ff:ff:ff, 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP

Does the Cisco switch, the router, or the DHCP server receive these broadcasts? (I don't know Cisco. At all.)
If the (tagged) broadcast packet is shown at enp2s0, then it should show up at the other end of the wire too.

Re: VLANs with QEMU/KVM

Posted: 2020/05/26 15:20:52
by LinuxLand
The switch is not seeing any traffic. I confirmed this using the following:

Code: Select all

switch01#sh int fa0/1    
FastEthernet0/1 is up, line protocol is up (connected) 
  Hardware is Fast Ethernet, address is xxxx.xxxx.xxxx (bia xxxx.xxxx.xxxx)
  Description: hv01 enp2s0
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:07:37
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 2000 bits/sec, 3 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     1280 packets output, 180995 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
I cleared the counters, tried ifup/ifdown on the guest and it should send some Ethernet broadcasts for DHCP discovery. I see nothing passing. Now, I suppose I could look at this with some packet capture, but at a glance, I don't see any traffic to capture.

I wonder if there is some negotiation between for trunking that is not happening correctly. This is a pretty vanilla setup on the switch side. I really haven't configured much because I don't need it in my case.

I think I have some more troubleshooting to do on this. I put enp2s0 back to a regular interface with a manual IP and I don't see any traffic on the switchport in that configuration either. I suppose this could be a cable, a bad switchport, or a bad NIC on the server.

Re: VLANs with QEMU/KVM

Posted: 2020/05/27 13:48:06
by jlehtone
LinuxLand wrote:
2020/05/26 15:20:52
I think I have some more troubleshooting to do on this. I put enp2s0 back to a regular interface with a manual IP and I don't see any traffic on the switchport in that configuration either. I suppose this could be a cable, a bad switchport, or a bad NIC on the server.
You should be able to systematically test each component on the path, given physical access etc.

Don't put two IP's to same subnet on the server though. That is unnecessary trouble.

A dedicated NIC on dedicated, separate admin subnet would be great for a server, for then one can reconfig "production ports" without fear of cutting self off.