network bridge with libvrt not working

Issues related to configuring your network
hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: network bridge with libvrt not working

Post by hunter86_bg » 2017/05/21 10:50:09

Usually I do this setup on my Desktop through Virtual Machine Manager (and libvirtd respectively). When you create a new network - a new bridge is created... So easy and simple.

opticalc
Posts: 9
Joined: 2017/05/11 17:10:29

Re: network bridge with libvrt not working

Post by opticalc » 2017/05/21 16:24:45

my xml is:

Code: Select all

<network>
  <name>insideBridge</name>
  <bridge name="inside0" stp="on" delay="2" />
  <mac address='00:16:3E:00:00:00'/>
  <forward mode="route" dev="enp3s0"/>
  <ip address="192.168.10.253" netmask="255.255.255.0" />
</network>
ok yes, so libvrt starts dnsmasq as necessary, im not sure why it need it though, as im using a forward type of route, even though I am putting an IP on that bridge. also, I did notice that after I destroy then start the inside0 bridge, and hten after I use service network restart, that things just work then, and that dnsmasq is running then too. so i dont anymore think dnsmasq is breaking anything. that dnsmasq error is probably just something else indicative of the overall issue.

like for example, after a reboot, the inside0 bridge comes up but doesnt have that IP on it, but after the net-start/destroy it does have that ip on it, and then after that, when I do service network restart, I get an error that my physical int isnt a member of the bridge, (verified by using "brctl show" before using service network restart) but after service network restart it is. (i am supposed to have "BRIDGE=inside0" in my ifcfg-enp3s0 file, right? even though I use "<forward mode="route" dev="enp3s0"/>" in my XML file? right?)

it appears that maybe just libvrt isnt starting all its things up in the right order or giving enough time for things to establish, or not checking for required things before starting other things?

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: network bridge with libvrt not working

Post by jlehtone » 2017/05/22 06:36:59

opticalc wrote:ok yes, so libvrt starts dnsmasq as necessary, im not sure why it need it though, as im using a forward type of route, even though I am putting an IP on that bridge.
The insideBridge is a separate subnet. It must be different from the subnet, where the br3 is.

If the br3 has "static ip 192.168.10.253", then the inside0 cannot have 192.168.10.253, or any other address in that subnet.

This is exactly the same situation as the pfsense has. It has two interfaces (on br2 and br3), each with unique IP address, does it not? It does route between two subnets.


The insideBridge is an entirely separate subnet. Where is it's DHCP server? Nowhere so far. The dnsmasq started by libvirtd is the logical answer.
You have to have

Code: Select all

  <ip address='192.168.XXX.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.XXX.33' end='192.168.XXX.254'/>
    </dhcp>
  </ip>
In the definition of insideBridge, where XXX is not 10.

There is a libvirt syntax (which I have forgotten) to set the dnsmasq to hand out same "static" IP for a MAC (of a VM).

theibel
Posts: 3
Joined: 2017/05/23 22:58:00

Re: network bridge with libvrt not working

Post by theibel » 2017/05/23 23:12:44

With regards to ...
Option B. Bridging.
1. Define no networks on the libvirt. Create no VirLAN at all.
2. Create a bridge interface on the host. Manually. Enslave the enp3s0 into that bridge. Let the bridge get an IP from the DHCP of the PhyLAN.
3. For each VM, enslave their interface to the same bridge. This will make them members of the PhyLAN.
While I understand the frustrations of setting up routing, I have read in many places about a true bridge between host and guest, but have never seen a working example. Do you have a working example of a KVM running Centos 7 connected to a true bridge?

I've read libvirt, linux-kvm, redhat and seen blog after blog detailing how it can be done, with plenty of examples showing the bridge on the physical host, but in my testing, I've never been able to get the KVM linked to this bridge. In a weeks worth of searching, I've never found anyone that's done it either. They claim it can be done, but they fail to show any real working example.

Would you elaborate on this point? You clearly have an extensive background in networking, but I thought I did to until I tried to connect the guest to the host with a true bridge.

What would be the network config on the KVM side (guest) after the bridge is setup on the host (physical)?

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: network bridge with libvrt not working

Post by jlehtone » 2017/05/24 19:19:26

1. Somewhat hijacking a thread?

2. What is a "true bridge"?

3. If it is what I guess it to be, then opticalc has already two up and running.

4. Continuing from that, the guest has nothing special. It simply has an interface on a subnet.
The libvirt definition of the VM has something like:

Code: Select all

    <interface type='bridge'>
      <mac address='52:54:00:12:34:56'/>
      <source bridge='br4'/>
      <model type='virtio'/>
    </interface>

theibel
Posts: 3
Joined: 2017/05/23 22:58:00

Re: network bridge with libvrt not working

Post by theibel » 2017/05/24 22:05:59

Thank you for your reply.

1) The OP is trying to conquer a network problem, and if using a true bridge (not NAT'd or routed), then this might help him.
2) "True Bridge" - no routing or nat'ing
3) I didn't see the KVM config (network/xml) from the OP
4) I agree the KVM should simply have an IP on the same subnet as the host, which I do, but it fails to connect.

virsh dumpxml test
<interface type='bridge'>
<mac address='52:54:00:c2:5e:42'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:05:ea:45'/>
<source network='test'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>

virsh net-dumpxml test
<network connections='1'>
<name>test</name>
<uuid>8b390604-36d4-422c-8ee0-d55afe67e06d</uuid>
<forward mode='bridge'/>
<bridge name='br1'/>
</network>

Setting up the test KVM (eth0) with an IP on the same network as the physical LAN simply fails. No connection at all.

So far I've tried:

1) Creating a bridge (br0) to em1 (Dell servers) and using this bridge to build the KVMs
2) I've tried creating a bridge to the KVM from the physical host after the KVM was built. Of course, there's no connection because they can't see each other. The default NAT routing was undefined.
3) Even though I should only need an IP assigned to eth0 on the KVM, I've also tried creating a bridge back to the physical host (br0/em1) from the KVM
4) Without a bridge in the KVM xml, it's not possible to bridge from the KVM to the physical host.

At this point, I've read many pages of blogs that claim this stuff can be done, unfortunately I can't get it to work. I cannot find any confirmation that it does work.

Here's my scenario, and if you can help me that would be a big help to the community at large.

Physical Host: NIC1 Port 1 (em1) 192.168.1.120
Test KVM: eth0 192.168.1.130
Physical Gateway: 192.168.1.1
Physical Subnet Mask: /24
NAT: none (default network will be removed)

I have scripts that I use to spin up KVM servers, and several terabytes of storage for hosting them. If you can tell me the scenario that works, I can build it from scratch in a few minutes. What I need are the network configs (ifcfg-em1/ifcfg-br0) for both ends of the bridge (host <---> guest), and the network xml config for the KVM network.

My guess is this is a very serial process or libvirt has to keep the default NAT routing in place. For security reasons, I can't keep the default NAT routing in place.

If you're up for it, thank you for your help.

theibel
Posts: 3
Joined: 2017/05/23 22:58:00

Re: network bridge with libvrt not working

Post by theibel » 2017/05/25 03:45:21

I started from scratch, and with the help of my notes, I figured out how to get the "true bridge" to work. I will compile the step-by-step process tomorrow. I still have more tests to run.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: network bridge with libvrt not working

Post by jlehtone » 2017/05/25 09:40:53

the network configs (ifcfg-em1/ifcfg-br0)
Those are of interest, if one does not use NetworkManager. The NetworkManager is a default in 7, and thus a new era of magick incantations is upon us.

Start state:
* Physical host has em1, but no connections defined.
* There is no libvirt.
* The subnet has a "gateway" that runs DHCP server.

1.

Code: Select all

$ nmcli con add type bridge con-name TowerBridge ifname TowerBridge
$ nmcli con add type ethernet con-name br-slave-1 ifname em1 master TowerBridge
$ nmcli con modify TowerBridge bridge.stp no
2. Add/start libvirtd
3. Disable unnecessary network that was created by libvirt installation:

Code: Select all

virsh net-autostart --network default --disable
4. Create a VM, whose definition contains

Code: Select all

<interface type='bridge'>
  <source bridge='TowerBridge'/>
</interface>
By default, the TowerBridge uses DHCP client to get an IPv4 address for the physical host.
Likewise, whatever OS is installed into the VM, should use DHCP for network configuration.

viqivi
Posts: 1
Joined: 2020/01/06 07:47:21

Re: network bridge with libvrt not working

Post by viqivi » 2020/01/06 08:14:48

Hi there,

2 year old thread but I still get the issue with bridge mode.

I am using latest Centos7 with KVM & Virtualbox, everything is ok with NAT mode; The bridge network is basically works in the first 10 minutes, after then I could not ssh or ping the guest (Centos7 as well) (loggin by using virt-viewer -> ok; ping revived -> off virt-viewer session --> couldn't ping/ssh again)

Thing did try:
- Stop/disable NetworkManager
- Changed virtio interface to e1000 or rtl8139
- Disabled ipv6, turned on ip_forward

Same phenomenon with Virtualbox+Extension pack

UPDATED: Very strange but Centos7 host w Centos7 guest was not made it; After switched guest to Ubuntu, everything works fine as it should. Tks

alamahant
Posts: 3
Joined: 2020/01/13 13:47:35

Re: network bridge with libvrt not working

Post by alamahant » 2020/01/13 14:51:11

You do not need nat for a bridge.
You do not need to use

Code: Select all

virsh net-start
This is used for libvirt NAT networks, whereas a bridge will allow you to share the host's local network.
Please use 'nmcli' to create the bridge thus:

Code: Select all

nmcli con add type bridge autoconnect yes con-name br0 ifname br0 ip4 <ip/cidr> gw4 <your-router>
nmcli con mod br0 ipv4.dns <your-router-or-prefered-dns> ipv4.method manual
nmcli con add type bridge-slave autoconnect yes con-name br0-slave1 ifname <enp......> master br0
Additionally you need
iptables and ebtables
installed.
Furthermore allow ip-forwarding thus

Code: Select all

echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/ip_forward.conf && sysctl -p /etc/sysctl.d/ip_forward.conf
Then when creating VMs assign the "br0" to them.
;) ;)

Post Reply