creating my first KVM VM with network bridge is not working

Issues related to configuring your network
Post Reply
sreese
Posts: 2
Joined: 2020/01/17 17:05:10

creating my first KVM VM with network bridge is not working

Post by sreese » 2020/01/17 17:10:00

Hello guys,

I have a network bridge on my CentOS 7 box where eno1 is the slave of br0. The br0 gets its IP via DHCP. Now I called following commands to create the VM.

Code: Select all

genisoimage -output plex.iso -volid cidata -joliet -rock user-data meta-data
qemu-img create -f qcow2 -o backing_file=../qcows/CentOS-7-x86_64-GenericCloud.qcow2 plex.qcow2
virt-install --connect qemu:///system --import --name plex --ram 4096 --vcpus 2 --disk path=plex.qcow2 --disk path=plex.iso,device=cdrom --network bridge=br0,model=virtio --os-type=linux --os-variant=centos7.0 --noautoconsole --nographics
Unluckily the network inside the VM does not start and does not get IP via DHCP, why? I see following error.

Code: Select all

[FAILED] Failed to start LSB: Bring up/down networking.
This prints after a timeout.

Code: Select all

[     *] A start job is running for LSB: Bri.../down networking (45s / 5min 5s)
The guest VM's network service logs the following.

Code: Select all

sudo systemctl status network
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: timeout) since Fri 2020-01-17 18:52:11 UTC; 50s ago
     Docs: man:systemd-sysv-generator(8)

Jan 17 18:51:19 plex dhclient[855]: DHCPDISCOVER on eth0 to 255.255.255.255...5)
Jan 17 18:51:33 plex dhclient[855]: DHCPDISCOVER on eth0 to 255.255.255.255...5)
Jan 17 18:51:51 plex dhclient[855]: DHCPDISCOVER on eth0 to 255.255.255.255...5)
Jan 17 18:52:02 plex dhclient[855]: DHCPDISCOVER on eth0 to 255.255.255.255...5)
Jan 17 18:52:11 plex systemd[1]: network.service start operation timed out....g.
Jan 17 18:52:11 plex systemd[1]: Failed to start LSB: Bring up/down networking.
Jan 17 18:52:11 plex systemd[1]: Unit network.service entered failed state.
Jan 17 18:52:11 plex systemd[1]: network.service failed.
Jan 17 18:52:12 plex dhclient[855]: No DHCPOFFERS received.
Jan 17 18:52:12 plex network[726]: Determining IP information for eth0... f...d.
Hint: Some lines were ellipsized, use -l to show in full.
The eth0 on the guest looks like this.

Code: Select all

BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=52:54:00:5d:f6:5d
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
How I am setting up the bridge follows now.

List the state after CentOS installation on the host.

Code: Select all

nmcli con show --active
NAME    UUID                                  TYPE      DEVICE
eno1    6eaf6ce4-fd7b-428a-8a4f-0390f13d3930  ethernet  eno1
virbr0  5803036f-c5e9-4893-ad8c-b8f5cd42c0ed  bridge    virbr0
Create the bridge interface and add eno1 as slave.

Code: Select all

nmcli con add ifname br0 type bridge con-name br0
nmcli con add type bridge-slave ifname eno1 master br0
This adds 'bridge-slave-eno1', no idea why.

Disable spanning tree.

Code: Select all

nmcli con modify br0 bridge.stp no
At the end I get this.

Code: Select all

nmcli con show
NAME               UUID                                  TYPE      DEVICE
br0                c81fcfb4-9091-47f5-8c0b-503666401912  bridge    br0
eno1               6eaf6ce4-fd7b-428a-8a4f-0390f13d3930  ethernet  eno1
virbr0             5803036f-c5e9-4893-ad8c-b8f5cd42c0ed  bridge    virbr0
bridge-slave-eno1  310597e5-fa36-4fbb-a3aa-6c15b387af9a  ethernet  --
eno1 down and br0 up.

Code: Select all

nmcli con up br0
ncmli con down eno1
Everything looks OK to me (noob).

Code: Select all

ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.5  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::cd6:5b23:4367:fab9  prefixlen 64  scopeid 0x20<link>
        ether 1c:69:7a:0a:98:2f  txqueuelen 1000  (Ethernet)
        RX packets 113  bytes 8477 (8.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 39  bytes 7002 (6.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:8d:49:28:10  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 1c:69:7a:0a:98:2f  txqueuelen 1000  (Ethernet)
        RX packets 97697  bytes 134675728 (128.4 MiB)
        RX errors 0  dropped 4  overruns 0  frame 0
        TX packets 32598  bytes 2350477 (2.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xc0b00000-c0b20000

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:82:15:ce  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
Restart NetworkManager to be sure.

Code: Select all

sudo systemctl restart NetworkManager
Configured bridge in virtlib.

Code: Select all

cat > bridge.xml <<EOF
<network>
    <name>host-bridge</name>
    <forward mode="bridge"/>
    <bridge name="br0"/>
</network>
EOF
virsh net-define bridge.xml
virsh net-start host-bridge
virsh net-autostart host-bridge
Do I have to do more for virtlib to work?
Last edited by sreese on 2020/01/17 21:38:27, edited 1 time in total.
Cheers
Sherry

sreese
Posts: 2
Joined: 2020/01/17 17:05:10

Re: creating my first KVM VM with network bridge is not working

Post by sreese » 2020/01/17 19:07:44

It seems this was the freaking firewall.

Code: Select all

firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i br0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o br0 -j ACCEPT
firewall-cmd --reload
And now I get the IP. Creating the bridge in virtlib (with virsh commands) are probably not necessary.
Cheers
Sherry

s91066
Posts: 12
Joined: 2011/12/01 09:41:40

Re: creating my first KVM VM with network bridge is not working

Post by s91066 » 2020/12/03 07:03:13

Aren't those rules specifically allow all traffic?
I am a bit confused because I have exactly the same problem.
But I am almost convinced this is not about forwarding, but because (in my case) firewalld assigned the bridged to the zone and the VMs interfaces are not part of it (so, I need to either disable the interfaces binding, or to find a way to automatically add them in the zone)

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: creating my first KVM VM with network bridge is not working

Post by jlehtone » 2020/12/05 13:41:42

Do note that CentOS 7 has received updates after January 2020 (time of original posts) and hence initial issues could have been invalidated.


Bridge is a L2 switch. Kernel does not filter bridged traffic. Not by default. It is possible to enable that filtering, but you need to explicitly load kernel modules, etc.
Forwarded (aka routed) traffic is filtered (by firewall rules).

On bridged setup:

Code: Select all

LAN --- NIC --- Bridge --- VNIC -- VM
* The VM is member of external subnet/broadcast domain ("LAN")
* The OS in VM probably has firewall (for INPUT) on its interface (VNIC)
* Libvirtd could add firewall rules to filter at VNIC on the host side. (Again, this is not default)
* Host does not filter packets that traverse NIC and Bridge
* Host does not route traffic
* The Bridge is just a non-physical network switch that allows more machines to be hooked to the LAN

Since physical NIC of the host is one port of the Bridge, it is more logical that the Bridge is configured along other network of the host (by NetworkManager in default CentOS 7). The Bridge is thus up, whether your start libvirtd.service or not.
Libvirtd.service can configure a Bridge, but frankly that is less logical.
If the host has only one NIC, then the Bridge should really be configured outside of libvirt, because host's IP address should be on the Bridge.

The libvirt does by default define a virtual network named "default". With it the setup is:

Code: Select all

LAN --- NIC     default --- VNIC -- VM
* The default is a bridge. A virtual switch.
* Host has IP address on its NIC (member of LAN) and on the bridge ("default")
* Host routes traffic between LAN and default
* Host filters traffic between LAN and default
* Host masquerades (rule added by libvirt) traffic that goes to LAN to hide VMs that are in the host


You can see the actual rules created by firewalld, libvirtd, fail2ban, docker, etc with:

Code: Select all

sudo iptables -S
sudo iptables -t nat -S
sudo iptables -t mangle -S
You can see bridges with

Code: Select all

brctl show

s91066
Posts: 12
Joined: 2011/12/01 09:41:40

Re: creating my first KVM VM with network bridge is not working

Post by s91066 » 2020/12/20 16:09:18

Wonderful explanation, I haven't read such good explanation for quite some time. Many thanks for spending time to explain it!

As for the original issue, in my case it was simply a misunderstanding of the documentation. To my defense, the libvirt documentation mention this:

Code: Select all

The final step is to disable netfilter on the bridge:
However, this is mentioned in the network section and not in the NetworkManager, so, I wrongly assumed that it was something that is was relative to users with the network service and not NM.

When I applied the

Code: Select all

# cat >> /etc/sysctl.conf <<EOF
 net.bridge.bridge-nf-call-ip6tables = 0
 net.bridge.bridge-nf-call-iptables = 0
 net.bridge.bridge-nf-call-arptables = 0
 EOF
 # sysctl -p /etc/sysctl.conf
 
everything is OK.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: creating my first KVM VM with network bridge is not working

Post by jlehtone » 2020/12/20 21:26:24

Do you have /usr/lib/sysctl.d/00-system.conf?
It should belong to initscripts and it has those "disable bridge-nf*".

However, your kernel should also say this during boot:

Code: Select all

bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Here is some history.
Ages ago the bridged traffic was not filtered. RHL/Fedora Core/CentOS kernel did not even have a module to do it.
A third party did build a kernel module that does send bridged packets into netfilter. Installing and using that was not a default.

Later, the module was added to default RHEL kernel. However, Red Hat chose to add the "disable bridge-nf*" config
to keep system to behave as before and to keep traffic to bridged VM's as efficient as possible.
Even though kernel now could have filtered bridged traffic, it did not on CentOS.
Some other distros might have been "filtering by default" and libvirt has written the instructions for them.

Some five years ago kernel (also in CentOS 7) did change to use module "br_netfilter" for this and to not load it by default.
Since the module that would send bridged packets to netfilter should not load, there should be no use nor effect for the bridge-nf* config.

Post Reply