Host network bridge and kvm/libvirtd/qemu problems

Issues related to configuring your network
beargfr
Posts: 8
Joined: 2020/07/15 04:47:29

Host network bridge and kvm/libvirtd/qemu problems

Post by beargfr » 2021/01/31 17:34:27

I've been fighting with this for two solid days now and it's driving me insane.
Please someone have mercy on this poor soul.

What has me so mystified is that this all worked for many months - but now it's not, and I have no clue what happened or why.

Configuration: Centos 8 running libvirt 6.0.0.29 and qemu-kvm 15:4.2.0-35

Problem: bridge to host network has stopped passing traffic.
VM's configured to use this bridge cannot access the host network at all, no DHCP, no DNS, no nothing.
Please note again that this WAS working for months and I have no idea what happened to change/break it.

Code: Select all

# nmcli c show
NAME             UUID                                  TYPE      DEVICE          
virbr1           b670290d-efec-4ddb-b151-c504db171f5b  bridge    virbr1          
conn-Bridge0     96f05ab9-88ab-434d-978a-c9cf1bae1e98  bridge    ifc-Bridge0     
docker0          2bba55b3-782e-4931-a03c-ed23175d2dd7  bridge    docker0         
docker_gwbridge  50e3585a-f172-4ee5-9fbc-09b1cbbf41db  bridge    docker_gwbridge 
slave-Bridge0    ce06c20a-2b26-460a-9ac7-d09348ada2aa  ethernet  enp7s0          
enp7s0           bfac8de3-0d1e-4cf5-80c8-14bb46b6d0ec  ethernet  --   
I literally spent 12 hours on it yesterday, deleting and redefining the bridge more times than I can count using all the methods described here:
https://www.tecmint.com/create-network- ... -centos-8/

I tried rebooting, I tried restarting the NetworkManager service, I used different names, I stopped firewalld, everthing I could think of.
Then the very last thing I tried, which was when using nm-connection-editor to define it all *one more time*, I used the 'general' tab while defining both the conn-Bridge0 connection and the slave-Bridge0 connection to place them in the 'trusted' firewall group instead of the 'default' firewall group. After I did that, things worked. I thought I had it licked.

But then today - it's not working and not passing traffic - again. After spending 7 hours on it so far today, nothing I've tried is working - not even changing the firewall group for the two components, not even completely shutting down firewalld.

I've got no idea what the root cause of this problem is.

Code: Select all

# nmcli d show
GENERAL.DEVICE:                         ifc-Bridge0
GENERAL.TYPE:                           bridge
GENERAL.HWADDR:                         6C:F0:49:57:CD:C3
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     conn-Bridge0
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/3
IP4.ADDRESS[1]:                         192.168.2.114/24
IP4.GATEWAY:                            192.168.2.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.2.1, mt = 427
IP4.ROUTE[2]:                           dst = 192.168.2.0/24, nh = 0.0.0.0, mt = 427
IP4.DNS[1]:                             192.168.2.33
IP4.DNS[2]:                             192.168.2.35
IP4.DOMAIN[1]:                          <hidden>
IP6.ADDRESS[1]:                         fe80::6f86:2186:a786:319a/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 427
IP6.ROUTE[2]:                           dst = ff00::/8, nh = ::, mt = 256, table=255

GENERAL.DEVICE:                         enp7s0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         6C:F0:49:57:CD:C3
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     slave-Bridge0
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/5
WIRED-PROPERTIES.CARRIER:               on
IP4.GATEWAY:                            --
IP6.GATEWAY:                            --


# ip link show type bridge
5: virbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether e6:f2:d7:47:fc:f7 brd ff:ff:ff:ff:ff:ff
6: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:3e:f4:8b:9c brd ff:ff:ff:ff:ff:ff
7: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:8b:e6:13:4e brd ff:ff:ff:ff:ff:ff
17: ifc-Bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 6c:f0:49:57:cd:c3 brd ff:ff:ff:ff:ff:ff
    
 ip link show type bridge_slave
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ifc-Bridge0 state UP mode DEFAULT group default qlen 1000
    link/ether 6c:f0:49:57:cd:c3 brd ff:ff:ff:ff:ff:ff
10: veth794278c@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 26:79:60:e4:26:90 brd ff:ff:ff:ff:ff:ff link-netnsid 0
16: vethfffe0f3@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP mode DEFAULT group default 
    link/ether 3a:fd:f0:47:06:0f brd ff:ff:ff:ff:ff:ff link-netnsid 2

I'm really hoping it's something obvious/stupid that I'm just overlooking.


TIA
Bear

beargfr
Posts: 8
Joined: 2020/07/15 04:47:29

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by beargfr » 2021/02/01 19:03:43

Fixed it!!!

After spending 15-18 hours on it yesterday, and another 4 hours or so today, I found this link:
https://bbs.archlinux.org/viewtopic.php?id=223827

It turns out that 'something' (not me, probably some update) had futzed with my iptables settings.

Code: Select all

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
.
.
.
All it took to fix it was:

Code: Select all

# iptables -P FORWARD ACCEPT
[root@centsrv01 system]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
.
.
.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by jlehtone » 2021/02/01 21:42:15

Couple tiny notes:

The kernel of CentOS 8 has nftables. Therefore, firewall rules are adjusted with 'nft' tool.
The 'iptables' is just a wrapper that translates (99%) iptables syntax to nftables.
The 'iptables -L' does show only couple chains, while nftables can have many.
For example, the default service, firewalld, creates both (empty) "iptables chains" and its own.
To see entire ruleset, use: sudo nft list ruleset


If you have bridged setup, then the VM's are in the same subnet/broadcast domain
as physical network outside of the server. The bridge is just one more switch that extends
the subnet. Switches do not filter.

Bridged packets do not traverse the FORWARD filter chain. You can get them filtered, but
then you have to explicitly load a kernel module. That is not a default.

beargfr
Posts: 8
Joined: 2020/07/15 04:47:29

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by beargfr » 2021/02/01 22:44:43

I don't pretend to know squat about the topics you mentioned, that's one of the reasons I came here asking for help.

All I know is that it's a fact that

Code: Select all

iptables -P FORWARD ACCEPT
fixed the problem when nothing else worked.

What I want to know now, is how and where do I harden 'whatever that did' so that it doesn't get reset or lost the next time I boot this machine?

<rant>
One of the things that made trying to diagnose and fix this problem so $%^&* hard is the seemingly (to me) flippant and casual way that major parts of this world are discarded/deprecated for no apparent reason other than to provide some 'new' way of doing the same job. For instance, during my ordeal these past few days I kept searching and finding information about using "brctl" to dig into these sorts of problems, but apparently "brctl" has been deprecated/dropped in favor of "ip", part of the iproute2 package, but good luck on finding any kind of documentation that tells you things like "if you did it this way using brctl, then here's how you do the same thing using ip". nmcli/NetworkManager apparently have replaced 'something else', just like systemd/systemctl have supplanted init.d </rant>

Anyway, I've got decades of software and "system admin" type experience but only the tiniest fraction of that time, just in the last few months really, has been on any sort of linux platform, so the learning curve there for me is still pretty steep. I really would like to know, what's the "proper" way to accomplish whatever that iptables command did and harden it so that it won't go away on the next reboot?

And, related to that question, is there anywhere any kind of comprehensive reference information I can use as a resource? There's no shortage of anecdotal 'bite sized' fragrments that provide a few mostly incomplete examples of how to do a few specific tasks, but I want *all* the details about things - even the parts that I may not ever need - every option, every parameter, etc.

Thanks,
Bear

tunk
Posts: 1204
Joined: 2017/02/22 15:08:17

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by tunk » 2021/02/01 23:32:42

Firewalld is the default firewall, and issuing iptables commands
may not work when firewalld is running. I wouldn't really know as
I've disabled it and installed iptables-services:
https://www.digitalocean.com/community/ ... n-centos-7
(This is for C7, but I think it works on C8 as well.)

When first starting using linux, I used this as a starting point
for creating rules (in particular listing 3, Paranoid home user):
https://linuxgazette.net/103/odonovan.html

beargfr
Posts: 8
Joined: 2020/07/15 04:47:29

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by beargfr » 2021/02/02 02:01:42

Well, apparently I have more to figure out. "Something" has run on the system and reset iptables back to their previous, "broken" state such that I had to run the previously mentioned iptables command again to unlock it.

I'm guessing it's something that gets started periodically via cron (or has cron also been deprecated/replaced by something "new and better"? -- sheesh)

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by TrevorH » 2021/02/02 09:36:21

If you are running firewalld (which is default) then it runs a daemon that makes sure your firewall rules match what it thinks they should look like. It will change the rules back if it sees them altered which is why you should never use the iptables (or nft) commands to alter any firewall rules. The effect of altering the rules behind firewalld's back will be temporary and it will revert those changes when it feels like it.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by jlehtone » 2021/02/02 14:40:19

beargfr wrote:
2021/02/01 22:44:43
And, related to that question, is there anywhere any kind of comprehensive reference information I can use as a resource?
Do RHEL docs count? https://access.redhat.com/documentation ... networking
(The previous chapter is about firewalld.service, but nobody recommends it for "real work".)

Upstream nftables has its own material: https://wiki.nftables.org/wiki-nftables ... /Main_Page
Gentoo's examples were "bite sized", but easy to chew: https://wiki.gentoo.org/wiki/Nftables/Examples

I did hop from firewalld to nftables.service by first dumping firewalld's ruleset from kernel to a file in /etc/nftables/
and then appended include-line to /etc/sysconfig/nftables.conf
Obviously, there was heavy editing of the ruleset before starting the service, because firewalld's rules were (a) insufficient for my needs and (b) a bloated mess.


You did show devices virbr1 and docker0 too. These are probably created by libvirtd and docker, respectively. Both of those services tend to inject their own firewall rules. They either co-operate with firewalld or do their own mayhem. Neither is desirable, if you have stable setup where you know the required ruleset and can thus manage it without those services. (Assuming you can tell them to not mess with your rules.)

beargfr
Posts: 8
Joined: 2020/07/15 04:47:29

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by beargfr » 2021/02/02 17:26:11

TrevorH wrote:
2021/02/02 09:36:21
If you are running firewalld (which is default) then it runs a daemon that makes sure your firewall rules match what it thinks they should look like. It will change the rules back if it sees them altered which is why you should never use the iptables (or nft) commands to alter any firewall rules. The effect of altering the rules behind firewalld's back will be temporary and it will revert those changes when it feels like it.
Understood, thanks. What I'm still trying to find/figure out is how to coerce firewalld into "doing the same thing" as what I accomplished with that iptables command, and harden it so that it survives reboots and anything else that firewalld might decide to do.

I'd really appreciated being pointed in the right direction to do that.
Thanks,
Bear

tunk
Posts: 1204
Joined: 2017/02/22 15:08:17

Re: Host network bridge and kvm/libvirtd/qemu problems

Post by tunk » 2021/02/02 17:55:56

(The previous chapter is about firewalld.service, but nobody recommends it for "real work".)

Post Reply