KVM QEMU Port Forwarding Not Working - Connection Refused

Issues related to configuring your network
Post Reply
own3mall
Posts: 23
Joined: 2013/08/30 17:50:11

KVM QEMU Port Forwarding Not Working - Connection Refused

Post by own3mall » 2020/07/11 20:34:51

Hi All,

I followed this guide for setting up persistent iptables forwarding rules for various KVM guests:

https://wiki.libvirt.org/page/Networking (Forwarding Incoming Connections - Using Hooks Script)

While the above guide works perfectly in CentOS 7, I am unable to get the forwarding to work in CentOS 8. I cannot figure out why. The setup is nearly identical to the working version I have on my CentOS 7 server. The iptables rules are there on my CentOS 8 server:

Code: Select all

iptables -t nat -L -n -v

Code: Select all


Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:29901 to:192.168.122.10:29901
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:29901 to:192.168.122.10:29901
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:29900 to:192.168.122.10:29900
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:29900 to:192.168.122.10:29900
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:29899 to:192.168.122.10:29899
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:29899 to:192.168.122.10:29899
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:28901 to:192.168.122.10:28901
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:28901 to:192.168.122.10:28901
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:28900 to:192.168.122.10:28900
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:28900 to:192.168.122.10:28900
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:28899 to:192.168.122.10:28899
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:28899 to:192.168.122.10:28899
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:27900 to:192.168.122.10:27900
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:27900 to:192.168.122.10:27900
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:27899 to:192.168.122.10:27899
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:27899 to:192.168.122.10:27899
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:13139 to:192.168.122.10:13139
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:13139 to:192.168.122.10:13139
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12310 to:192.168.122.10:12310
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12310 to:192.168.122.10:12310
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12309 to:192.168.122.10:12309
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12309 to:192.168.122.10:12309
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12308 to:192.168.122.10:12308
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12308 to:192.168.122.10:12308
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12307 to:192.168.122.10:12307
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12307 to:192.168.122.10:12307
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12306 to:192.168.122.10:12306
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12306 to:192.168.122.10:12306
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12305 to:192.168.122.10:12305
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12305 to:192.168.122.10:12305
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12304 to:192.168.122.10:12304
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12304 to:192.168.122.10:12304
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12303 to:192.168.122.10:12303
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12303 to:192.168.122.10:12303
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12302 to:192.168.122.10:12302
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12302 to:192.168.122.10:12302
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12301 to:192.168.122.10:12301
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12301 to:192.168.122.10:12301
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12300 to:192.168.122.10:12300
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12300 to:192.168.122.10:12300
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12299 to:192.168.122.10:12299
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12299 to:192.168.122.10:12299
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12210 to:192.168.122.10:12210
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12210 to:192.168.122.10:12210
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12209 to:192.168.122.10:12209
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12209 to:192.168.122.10:12209
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12208 to:192.168.122.10:12208
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12208 to:192.168.122.10:12208
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12207 to:192.168.122.10:12207
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12207 to:192.168.122.10:12207
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12206 to:192.168.122.10:12206
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12206 to:192.168.122.10:12206
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12205 to:192.168.122.10:12205
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12205 to:192.168.122.10:12205
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12204 to:192.168.122.10:12204
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12204 to:192.168.122.10:12204
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:12203 to:192.168.122.10:12203
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:12203 to:192.168.122.10:12203
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:6667 to:192.168.122.10:6667
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:6667 to:192.168.122.10:6667
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:6515 to:192.168.122.10:6515
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:6515 to:192.168.122.10:6515
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:6500 to:192.168.122.10:6500
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:6500 to:192.168.122.10:6500
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:3783 to:192.168.122.10:3783
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:3783 to:192.168.122.10:3783

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    3   207 RETURN     all  --  *      *       192.168.122.0/24     224.0.0.0/24        
    0     0 RETURN     all  --  *      *       192.168.122.0/24     255.255.255.255     
    0     0 MASQUERADE  tcp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  udp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  all  --  *      *       192.168.122.0/24    !192.168.122.0/24    

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
However, when I try to connect using my main br0 IP address to one of the forwarded ports, I get a connection refused message, and the traffic doesn't appear to be forwarded to the guest (192.168.122.10 in this case).

Anyone know what might be going on?

I've disabled selinux and am NOT using firewalld (it's disabled). I'm using iptables directly.
Top
User avatar
jlehtone
Posts: 4531
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by jlehtone » 2020/07/12 09:28:36

The CentOS 8 kernel has nftables and iptables is a mere wrapper to nft tool.

To see everything that is in nftables, use:

Code: Select all

sudo nft list ruleset
If all rules in that are from you, then the question is what is wrong in them.
If there are other rules, then the question is who did add them?
Top
own3mall
Posts: 23
Joined: 2013/08/30 17:50:11

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by own3mall » 2020/07/14 17:36:54

All of the set rules were created by me. I don't know why it's not working.

I just tried the process again using this qemu hook script:

Code: Select all

if [ "${1}" = "testvm" ]; then

   # Update the following variables to fit your setup
   GUEST_IP=192.168.144.57

   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
		for i in $(/usr/bin/seq 1 42001 | /usr/bin/tail -n 20); do
			/usr/sbin/iptables -D FORWARD -o virbr0 -d $GUEST_IP -p tcp --dport ${i} -j ACCEPT
			/usr/sbin/iptables -D FORWARD -o virbr0 -d $GUEST_IP -p udp --dport ${i} -j ACCEPT
			/usr/sbin/iptables -t nat -D PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
			/usr/sbin/iptables -t nat -D PREROUTING -p udp --dport ${i} -j DNAT --to $GUEST_IP:${i}
		done
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
		for i in $(/usr/bin/seq 1 42001 | /usr/bin/tail -n 20); do
			/usr/sbin/iptables -I FORWARD -o virbr0 -d $GUEST_IP -p tcp --dport ${i} -j ACCEPT
			/usr/sbin/iptables -I FORWARD -o virbr0 -d $GUEST_IP -p udp --dport ${i} -j ACCEPT
			/usr/sbin/iptables -t nat -I PREROUTING -p tcp --dport ${i} -j DNAT --to $GUEST_IP:${i}
			/usr/sbin/iptables -t nat -I PREROUTING -p udp --dport ${i} -j DNAT --to $GUEST_IP:${i}
		done
   fi
fi
With the above, the ports are still not forwarding correctly for incoming remote connections. It's like the forwarding isn't working at all, so I don't understand what I'm missing. Again, the above worked in CentOS 7.

So, if I should be using nftables directly, what are the equivalent commands? It sounds like the libvirt documentation's script should be updated to use nftables?
Top
User avatar
jlehtone
Posts: 4531
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by jlehtone » 2020/07/14 19:09:52

Did you run nft list ruleset? It shows the actual rules (inserted by you) in nftables syntax.
(Syntax to insert individual rule differs a little from the ruleset, but nftables.service reads ruleset from file(s).)
Top
own3mall
Posts: 23
Joined: 2013/08/30 17:50:11

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by own3mall » 2020/07/14 22:53:26

I wanted to obfuscate my internal IPs, but that's not important. Here is the result of the nft list ruleset command:

Code: Select all

[root ~]# nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
		iifname "virbr0" meta l4proto udp udp dport 53 counter packets 8 bytes 502 accept
		iifname "virbr0" meta l4proto tcp tcp dport 53 counter packets 0 bytes 0 accept
		iifname "virbr0" meta l4proto udp udp dport 67 counter packets 28 bytes 9184 accept
		iifname "virbr0" meta l4proto tcp tcp dport 67 counter packets 0 bytes 0 accept
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 55001 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 55001 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 55000 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 55000 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54999 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54999 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54998 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54998 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54997 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54997 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54996 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54996 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54995 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54995 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54994 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54994 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54993 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54993 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54992 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54992 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54991 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54991 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport 54990 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport 54990 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 29901 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 29901 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 29900 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 29900 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 29899 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 29899 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 28901 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 28901 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 28900 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 28900 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 28899 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 28899 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 27900 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 27900 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 27899 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 27899 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 13139 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 13139 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12310 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12310 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12309 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12309 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12308 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12308 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12307 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12307 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12306 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12306 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12305 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12305 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12304 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12304 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12303 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12303 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12302 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12302 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12301 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12301 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12300 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12300 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12299 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12299 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12210 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12210 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12209 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12209 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12208 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12208 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12207 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12207 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12206 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12206 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12205 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12205 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12204 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12204 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 12203 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 12203 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 6667 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 6667 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 6515 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 6515 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 6500 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 6500 counter packets 1 bytes 40 accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.10 udp dport 3783 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 3783 counter packets 0 bytes 0 accept
		oifname "virbr0" ip daddr 192.168.122.0/24 ct state related,established counter packets 37 bytes 13289 accept
		iifname "virbr0" ip saddr 192.168.122.0/24 counter packets 43 bytes 4913 accept
		iifname "virbr0" oifname "virbr0" counter packets 0 bytes 0 accept
		oifname "virbr0" counter packets 0 bytes 0 reject
		iifname "virbr0" counter packets 0 bytes 0 reject
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		oifname "virbr0" meta l4proto udp udp dport 68 counter packets 28 bytes 9198 accept
	}
}
table ip6 filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
table bridge filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
table ip nat {
	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
		meta l4proto udp udp dport 55001 counter packets 0 bytes 0 dnat to 192.168.122.11:55001
		meta l4proto tcp tcp dport 55001 counter packets 0 bytes 0 dnat to 192.168.122.11:55001
		meta l4proto udp udp dport 55000 counter packets 0 bytes 0 dnat to 192.168.122.11:55000
		meta l4proto tcp tcp dport 55000 counter packets 0 bytes 0 dnat to 192.168.122.11:55000
		meta l4proto udp udp dport 54999 counter packets 0 bytes 0 dnat to 192.168.122.11:54999
		meta l4proto tcp tcp dport 54999 counter packets 0 bytes 0 dnat to 192.168.122.11:54999
		meta l4proto udp udp dport 54998 counter packets 0 bytes 0 dnat to 192.168.122.11:54998
		meta l4proto tcp tcp dport 54998 counter packets 0 bytes 0 dnat to 192.168.122.11:54998
		meta l4proto udp udp dport 54997 counter packets 0 bytes 0 dnat to 192.168.122.11:54997
		meta l4proto tcp tcp dport 54997 counter packets 0 bytes 0 dnat to 192.168.122.11:54997
		meta l4proto udp udp dport 54996 counter packets 0 bytes 0 dnat to 192.168.122.11:54996
		meta l4proto tcp tcp dport 54996 counter packets 0 bytes 0 dnat to 192.168.122.11:54996
		meta l4proto udp udp dport 54995 counter packets 0 bytes 0 dnat to 192.168.122.11:54995
		meta l4proto tcp tcp dport 54995 counter packets 0 bytes 0 dnat to 192.168.122.11:54995
		meta l4proto udp udp dport 54994 counter packets 0 bytes 0 dnat to 192.168.122.11:54994
		meta l4proto tcp tcp dport 54994 counter packets 0 bytes 0 dnat to 192.168.122.11:54994
		meta l4proto udp udp dport 54993 counter packets 0 bytes 0 dnat to 192.168.122.11:54993
		meta l4proto tcp tcp dport 54993 counter packets 0 bytes 0 dnat to 192.168.122.11:54993
		meta l4proto udp udp dport 54992 counter packets 0 bytes 0 dnat to 192.168.122.11:54992
		meta l4proto tcp tcp dport 54992 counter packets 0 bytes 0 dnat to 192.168.122.11:54992
		meta l4proto udp udp dport 54991 counter packets 0 bytes 0 dnat to 192.168.122.11:54991
		meta l4proto tcp tcp dport 54991 counter packets 0 bytes 0 dnat to 192.168.122.11:54991
		meta l4proto udp udp dport 54990 counter packets 0 bytes 0 dnat to 192.168.122.11:54990
		meta l4proto tcp tcp dport 54990 counter packets 0 bytes 0 dnat to 192.168.122.11:54990
		meta l4proto udp udp dport 29901 counter packets 0 bytes 0 dnat to 192.168.122.10:29901
		meta l4proto tcp tcp dport 29901 counter packets 0 bytes 0 dnat to 192.168.122.10:29901
		meta l4proto udp udp dport 29900 counter packets 0 bytes 0 dnat to 192.168.122.10:29900
		meta l4proto tcp tcp dport 29900 counter packets 0 bytes 0 dnat to 192.168.122.10:29900
		meta l4proto udp udp dport 29899 counter packets 0 bytes 0 dnat to 192.168.122.10:29899
		meta l4proto tcp tcp dport 29899 counter packets 0 bytes 0 dnat to 192.168.122.10:29899
		meta l4proto udp udp dport 28901 counter packets 0 bytes 0 dnat to 192.168.122.10:28901
		meta l4proto tcp tcp dport 28901 counter packets 0 bytes 0 dnat to 192.168.122.10:28901
		meta l4proto udp udp dport 28900 counter packets 0 bytes 0 dnat to 192.168.122.10:28900
		meta l4proto tcp tcp dport 28900 counter packets 0 bytes 0 dnat to 192.168.122.10:28900
		meta l4proto udp udp dport 28899 counter packets 0 bytes 0 dnat to 192.168.122.10:28899
		meta l4proto tcp tcp dport 28899 counter packets 0 bytes 0 dnat to 192.168.122.10:28899
		meta l4proto udp udp dport 27900 counter packets 0 bytes 0 dnat to 192.168.122.10:27900
		meta l4proto tcp tcp dport 27900 counter packets 0 bytes 0 dnat to 192.168.122.10:27900
		meta l4proto udp udp dport 27899 counter packets 0 bytes 0 dnat to 192.168.122.10:27899
		meta l4proto tcp tcp dport 27899 counter packets 0 bytes 0 dnat to 192.168.122.10:27899
		meta l4proto udp udp dport 13139 counter packets 0 bytes 0 dnat to 192.168.122.10:13139
		meta l4proto tcp tcp dport 13139 counter packets 0 bytes 0 dnat to 192.168.122.10:13139
		meta l4proto udp udp dport 12310 counter packets 0 bytes 0 dnat to 192.168.122.10:12310
		meta l4proto tcp tcp dport 12310 counter packets 0 bytes 0 dnat to 192.168.122.10:12310
		meta l4proto udp udp dport 12309 counter packets 0 bytes 0 dnat to 192.168.122.10:12309
		meta l4proto tcp tcp dport 12309 counter packets 0 bytes 0 dnat to 192.168.122.10:12309
		meta l4proto udp udp dport 12308 counter packets 0 bytes 0 dnat to 192.168.122.10:12308
		meta l4proto tcp tcp dport 12308 counter packets 0 bytes 0 dnat to 192.168.122.10:12308
		meta l4proto udp udp dport 12307 counter packets 0 bytes 0 dnat to 192.168.122.10:12307
		meta l4proto tcp tcp dport 12307 counter packets 0 bytes 0 dnat to 192.168.122.10:12307
		meta l4proto udp udp dport 12306 counter packets 0 bytes 0 dnat to 192.168.122.10:12306
		meta l4proto tcp tcp dport 12306 counter packets 0 bytes 0 dnat to 192.168.122.10:12306
		meta l4proto udp udp dport 12305 counter packets 0 bytes 0 dnat to 192.168.122.10:12305
		meta l4proto tcp tcp dport 12305 counter packets 0 bytes 0 dnat to 192.168.122.10:12305
		meta l4proto udp udp dport 12304 counter packets 0 bytes 0 dnat to 192.168.122.10:12304
		meta l4proto tcp tcp dport 12304 counter packets 0 bytes 0 dnat to 192.168.122.10:12304
		meta l4proto udp udp dport 12303 counter packets 0 bytes 0 dnat to 192.168.122.10:12303
		meta l4proto tcp tcp dport 12303 counter packets 0 bytes 0 dnat to 192.168.122.10:12303
		meta l4proto udp udp dport 12302 counter packets 0 bytes 0 dnat to 192.168.122.10:12302
		meta l4proto tcp tcp dport 12302 counter packets 0 bytes 0 dnat to 192.168.122.10:12302
		meta l4proto udp udp dport 12301 counter packets 0 bytes 0 dnat to 192.168.122.10:12301
		meta l4proto tcp tcp dport 12301 counter packets 0 bytes 0 dnat to 192.168.122.10:12301
		meta l4proto udp udp dport 12300 counter packets 0 bytes 0 dnat to 192.168.122.10:12300
		meta l4proto tcp tcp dport 12300 counter packets 0 bytes 0 dnat to 192.168.122.10:12300
		meta l4proto udp udp dport 12299 counter packets 0 bytes 0 dnat to 192.168.122.10:12299
		meta l4proto tcp tcp dport 12299 counter packets 0 bytes 0 dnat to 192.168.122.10:12299
		meta l4proto udp udp dport 12210 counter packets 0 bytes 0 dnat to 192.168.122.10:12210
		meta l4proto tcp tcp dport 12210 counter packets 0 bytes 0 dnat to 192.168.122.10:12210
		meta l4proto udp udp dport 12209 counter packets 0 bytes 0 dnat to 192.168.122.10:12209
		meta l4proto tcp tcp dport 12209 counter packets 0 bytes 0 dnat to 192.168.122.10:12209
		meta l4proto udp udp dport 12208 counter packets 0 bytes 0 dnat to 192.168.122.10:12208
		meta l4proto tcp tcp dport 12208 counter packets 0 bytes 0 dnat to 192.168.122.10:12208
		meta l4proto udp udp dport 12207 counter packets 0 bytes 0 dnat to 192.168.122.10:12207
		meta l4proto tcp tcp dport 12207 counter packets 0 bytes 0 dnat to 192.168.122.10:12207
		meta l4proto udp udp dport 12206 counter packets 0 bytes 0 dnat to 192.168.122.10:12206
		meta l4proto tcp tcp dport 12206 counter packets 0 bytes 0 dnat to 192.168.122.10:12206
		meta l4proto udp udp dport 12205 counter packets 0 bytes 0 dnat to 192.168.122.10:12205
		meta l4proto tcp tcp dport 12205 counter packets 0 bytes 0 dnat to 192.168.122.10:12205
		meta l4proto udp udp dport 12204 counter packets 0 bytes 0 dnat to 192.168.122.10:12204
		meta l4proto tcp tcp dport 12204 counter packets 0 bytes 0 dnat to 192.168.122.10:12204
		meta l4proto udp udp dport 12203 counter packets 0 bytes 0 dnat to 192.168.122.10:12203
		meta l4proto tcp tcp dport 12203 counter packets 0 bytes 0 dnat to 192.168.122.10:12203
		meta l4proto udp udp dport 6667 counter packets 0 bytes 0 dnat to 192.168.122.10:6667
		meta l4proto tcp tcp dport 6667 counter packets 0 bytes 0 dnat to 192.168.122.10:6667
		meta l4proto udp udp dport 6515 counter packets 0 bytes 0 dnat to 192.168.122.10:6515
		meta l4proto tcp tcp dport 6515 counter packets 0 bytes 0 dnat to 192.168.122.10:6515
		meta l4proto udp udp dport 6500 counter packets 0 bytes 0 dnat to 192.168.122.10:6500
		meta l4proto tcp tcp dport 6500 counter packets 1 bytes 40 dnat to 192.168.122.10:6500
		meta l4proto udp udp dport 3783 counter packets 0 bytes 0 dnat to 192.168.122.10:3783
		meta l4proto tcp tcp dport 3783 counter packets 0 bytes 0 dnat to 192.168.122.10:3783
	}

	chain INPUT {
		type nat hook input priority 100; policy accept;
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 192.168.122.0/24 ip daddr 224.0.0.0/24 counter packets 3 bytes 207 return
		ip saddr 192.168.122.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return
		meta l4proto tcp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 3 bytes 180 masquerade to :1024-65535 
		meta l4proto udp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 15 bytes 1140 masquerade to :1024-65535 
		ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade 
	}

	chain OUTPUT {
		type nat hook output priority -100; policy accept;
	}
}
table ip mangle {
	chain PREROUTING {
		type filter hook prerouting priority mangle; policy accept;
	}

	chain INPUT {
		type filter hook input priority mangle; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority mangle; policy accept;
	}

	chain OUTPUT {
		type route hook output priority mangle; policy accept;
	}

	chain POSTROUTING {
		type filter hook postrouting priority mangle; policy accept;
		oifname "virbr0" meta l4proto udp udp dport 68 counter packets 28 bytes 9198 # CHECKSUM fill
	}
}
No traffic is forwarded when using any of the above ports. I don't know why.
Top
User avatar
jlehtone
Posts: 4531
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by jlehtone » 2020/07/15 09:09:24

I can't spot obvious error.

Elementary check: Is the ip_forward enabled?

I use:

Code: Select all

$ cat /etc/sysctl.d/myrouter.conf
net.ipv4.ip_forward = 1
Documentation on nft:
Quick reference: https://wiki.nftables.org/wiki-nftables ... inutes#Nat
RHEL 8 intro: https://access.redhat.com/documentation ... networking
DNAT: https://access.redhat.com/documentation ... g-nftables
Sets: https://access.redhat.com/documentation ... h-nftables and http://wiki.nftables.org/wiki-nftables/index.php/Sets
Intervals: https://wiki.nftables.org/wiki-nftables ... /Intervals
Forwarding: https://access.redhat.com/documentation ... g-nftables
Script that creates router's ruleset: https://wiki.gentoo.org/wiki/Nftables/E ... er_example

The nftables.service reads /etc/sysconfig/nftables.conf
I have added to that file:

Code: Select all

include "/etc/nftables/myrouter.nft"
The /etc/nftables/myrouter.nft I did write starting with ruleset created by firewalld.

You don't modify port in your forwardings, just the address. Therefore, you could less rules in some chains:

Code: Select all

define forward_ports = { 54990-55001, 27899-29901, 13139, 12299-12310 }
table ip nat {
	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
		meta l4proto udp udp dport $forward_ports counter dnat to 192.168.122.11
		meta l4proto tcp tcp dport $forward_ports counter dnat to 192.168.122.11
	}
}

table ip filter {
	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		oifname "virbr0" ip daddr 192.168.122.0/24 ct state related,established counter accept
		iifname "virbr0" ip saddr 192.168.122.0/24 counter accept
		oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport $forward_ports counter accept
		oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.11 tcp dport $forward_ports counter accept
		iifname "virbr0" oifname "virbr0" counter accept
		oifname "virbr0" counter reject
		iifname "virbr0" counter reject
	}
}
(Unless you really want counter for each port. Btw, counters are optional in nftables.)

Now, lets consider that libvirt injects some rules when it starts the "virbr0" network
and then some more (in your hook script), when you start a VM.

You could have a named set (or define) of ports in the base ruleset (that exists even when libvirt does not run)
and in hook add/remove:

Code: Select all

nft add rule ip filter FORWARD position NN oifname "virbr0" meta l4proto udp ip daddr 192.168.122.11 udp dport \$forward_ports counter accept
nft add rule ip nat PREROUTING meta l4proto udp udp dport $forward_ports counter dnat to 192.168.122.11
Position: https://wiki.nftables.org/wiki-nftables ... n_position

If you have named set, then you can modify the list of ports dynamically.

Firewalld adds rule ct status dnat accept:

Code: Select all

        chain filter_FORWARD {
                type filter hook forward priority filter; policy accept;
                ct state established,related accept
                ct status dnat accept
                iifname "lo" accept
                iifname "eno1" oifname "eno2" accept
                meta l4proto { icmp, ipv6-icmp } accept
                ct state invalid drop
                reject with icmpx type admin-prohibited
        }
That allows packets that have been modified by a dnat rule. Hence, you would need to modify just the prerouting rules.
See (I just found this page and it seems more useful that the other links) https://stosb.com/blog/explaining-my-configs-nftables/


Note:
Your INPUT filter looks very open:

Code: Select all

	chain INPUT {
		type filter hook input priority filter; policy accept;
		iifname "virbr0" meta l4proto udp udp dport 53 counter packets 8 bytes 502 accept
		iifname "virbr0" meta l4proto tcp tcp dport 53 counter packets 0 bytes 0 accept
		iifname "virbr0" meta l4proto udp udp dport 67 counter packets 28 bytes 9184 accept
		iifname "virbr0" meta l4proto tcp tcp dport 67 counter packets 0 bytes 0 accept
	}
Like host accepts everything.

Mine:

Code: Select all

        chain INPUT {
                type filter hook input priority filter; policy accept;
                ct state established,related accept
                ct status dnat accept
                iifname "lo" accept
                tcp dport ssh    ct state new,untracked counter accept
                meta l4proto { icmp, ipv6-icmp } accept
                ct state invalid drop
                reject with icmpx type admin-prohibited
        }
This does not affect routing.
If this router would offer DHCP&DNS services (to, say virbr0), they would be around "allow ssh" rule.
Top
own3mall
Posts: 23
Joined: 2013/08/30 17:50:11

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by own3mall » 2020/07/15 15:12:41

Yes, IPv4 forwarding is enabled:

Code: Select all

[roo ~]# cat /proc/sys/net/ipv4/ip_forward
1
[root ~]# cat /proc/sys/net/ipv4/conf/br0/forwarding
1
Do you think it's because I'm missing the following in my rules?

Code: Select all

ct status dnat accept
firewalld is disabled in my case though.

Have you tested to see if your KVM virtual machines are able to receive traffic on your forwarded ports with your setup?
Top
User avatar
jlehtone
Posts: 4531
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by jlehtone » 2020/07/15 18:20:23

I don't route on VM hosts. My VMs directly on external subnets, either with (SR-IOV) passthrough device or via bridge.

You can listen traffic with tcpdump on the host.
I would listen for the client that attempts connection on the external interface, and on the virbr0.
If things work, there is incoming packet, forwarded natted packet, incoming reply, and outgoing reply (with hosts external IP as source).
If packet hits a reject rule, then that type of reply should show.
If packet gets as far as leaving from virbr0 to 192.168.122.10 and the reply is a "reject", then VM has an issue.

You had one packet that has got dnat and did get accepted in filter:

Code: Select all

meta l4proto tcp tcp dport 6500 counter packets 1 bytes 40 dnat to 192.168.122.10:6500
oifname "virbr0" meta l4proto tcp ip daddr 192.168.122.10 tcp dport 6500 counter packets 1 bytes 40 accept


The iptables.service, firewalld.service, and nftables.service all three do the same thing; load rules to kernel.
Services, like libvirtd and fail2ban do load rules too.
The only thing that counts, is the resulting ruleset.
Top
own3mall
Posts: 23
Joined: 2013/08/30 17:50:11

Re: KVM QEMU Port Forwarding Not Working - Connection Refused

Post by own3mall » 2020/07/24 19:47:11

I figured it out. The NAT routing was working properly from the KVM host server (using my originally posted iptables rules). On one of my KVM guest servers with only ONE DHCP NIC, I was using the wrong IP address to test (how embarrassing). The server I originally had problems with, I had to setup a second route for the second NIC (the DHCP one with an internal private LAN IP) before traffic could be sent back to an originating request:

https://unix.stackexchange.com/question ... swer-23345

I used tcpdump on the host and guests to see traffic was indeed making it to the guest servers, but no response traffic was being sent back until I added new ip routes on the KVM guests as mentioned in the article linked above.

Thanks for your help. I got it all working (and learned a ton). :D
Top
Post Reply

Return to “8 /8-Stream / 9-Stream - Networking Support”