firewall-cmd - add-forward-port don't work

Issues related to configuring your network
eduardolucioac
Posts: 23
Joined: 2017/01/25 12:40:27

firewall-cmd - add-forward-port don't work

Post by eduardolucioac » 2019/08/22 20:33:07

I have a KVM server (host) with multiple virtual machines (guests).

My goal is my host forward port 222 to port 22 of a guest running an ssh service.

This works...

Code: Select all

iptables -I OUTPUT -d 0.0.0.0/0 -j ACCEPT
iptables -I FORWARD -d 0.0.0.0/0 -j ACCEPT
iptables -I INPUT -d 0.0.0.0/0 -j ACCEPT
iptables -t nat -I PREROUTING -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
This doesn't work...

Code: Select all

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter OUTPUT 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter INPUT 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 nat PREROUTING 0 -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
firewall-cmd --reload
This doesn't work too...

Code: Select all

firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload
QUESTION: Why rules set with `firewall-cmd` don't work?

NOTE I: The firewall-cmd is the CentOS 7's default firewall service. This seems to me to be a problem with no solution! I searched many, many forums and nothing works! I'm starting to believe this is a limitation or bug in firewall-cmd...

NOTE II: I know that ssh itself provides the means to make this possible, but I really want this process to be "transparent" for the user to access the guest directly.

[Refs.: https://serverfault.com/q/915257/276753 ,
https://serverfault.com/q/980223/276753 ,
https://sebastianblade.com/how-to-modif ... n-centos7/ ,
https://www.rootusers.com/how-to-use-fi ... g-and-nat/ ]

----------------

SYMPTOM:

The command...

Code: Select all

ssh root@[HOST_IP] -p 222
... returns me the following error...
ssh: connect to host 172.16.13.8 port 222: Connection refused
Last edited by eduardolucioac on 2019/08/23 14:40:55, edited 1 time in total.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: firewall-cmd - add-forward-port don't work

Post by jlehtone » 2019/08/23 10:20:59

eduardolucioac wrote:
2019/08/22 20:33:07
I have a KVM server (host) with multiple virtual machines (guests).
Can we assume that the guests connect to outside via "routed, with NAT"?

The host has two "connections": to outside and to guests. Are they on the some zone (public)?
eduardolucioac wrote:
2019/08/22 20:33:07
The command...

Code: Select all

ssh root@[HOST_IP] -p 222
... returns me the following error...
ssh: connect to host 172.16.13.8 port 23: Connection refused
The 172.16.13.8 is clearly not 10.1.0.9. Does HOST_IP == 172.16.13.8?
Port 23 is neither 22 nor 222. If you really get that message, then there
is something that you haven't told us.


There is no "don't work" nor "works". Every command does something.
Have you looked what rules does the add-forward-port add into tables filter, nat, and mangle?

eduardolucioac
Posts: 23
Joined: 2017/01/25 12:40:27

Re: firewall-cmd - add-forward-port don't work

Post by eduardolucioac » 2019/08/23 14:40:05

Can we assume that the guests connect to outside via "routed, with NAT"?
-> Yes!
The host has two "connections": to outside and to guests. Are they on the some zone (public)?
-> How can I get this information for you?

Thanks! :D

eduardolucioac
Posts: 23
Joined: 2017/01/25 12:40:27

Re: firewall-cmd - add-forward-port don't work

Post by eduardolucioac » 2019/08/23 14:44:28

Code: Select all

The command...

ssh root@[HOST_IP] -p 222

... returns me the following error...

ssh: connect to host 172.16.13.8 port 23: Connection refused
The 172.16.13.8 is clearly not 10.1.0.9. Does HOST_IP == 172.16.13.8?
Port 23 is neither 22 nor 222. If you really get that message, then there
is something that you haven't told us.
-> Sorry, my mistake! Corrected! Port 222.

Code: Select all

Have you looked what rules does the add-forward-port add into tables filter, nat, and mangle?
-> How can I get this information for you?

eduardolucioac
Posts: 23
Joined: 2017/01/25 12:40:27

Re: firewall-cmd - add-forward-port don't work

Post by eduardolucioac » 2019/08/23 15:06:18

iptables-save output...

iptables-save - SSH WORKS...

Code: Select all

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*nat
:PREROUTING ACCEPT [1:70]
:INPUT ACCEPT [1:70]
:OUTPUT ACCEPT [2:146]
:POSTROUTING ACCEPT [3:206]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -p tcp -m tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*mangle
:PREROUTING ACCEPT [672:77587]
:INPUT ACCEPT [610:68993]
:FORWARD ACCEPT [58:7886]
:OUTPUT ACCEPT [655:151604]
:POSTROUTING ACCEPT [713:159490]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*security
:INPUT ACCEPT [609:68793]
:FORWARD ACCEPT [58:7886]
:OUTPUT ACCEPT [660:152010]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*raw
:PREROUTING ACCEPT [672:77587]
:OUTPUT ACCEPT [655:151604]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j ACCEPT
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j ACCEPT
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
iptables-save - SSH DOESN'T WORK...

Code: Select all

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*nat
:PREROUTING ACCEPT [1:72]
:INPUT ACCEPT [1:72]
:OUTPUT ACCEPT [5:371]
:POSTROUTING ACCEPT [5:371]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PREROUTING_direct -p tcp -m tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*mangle
:PREROUTING ACCEPT [12:1319]
:INPUT ACCEPT [11:1259]
:FORWARD ACCEPT [1:60]
:OUTPUT ACCEPT [12:1070]
:POSTROUTING ACCEPT [12:1070]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*security
:INPUT ACCEPT [11:1259]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12:1070]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*raw
:PREROUTING ACCEPT [12:1319]
:OUTPUT ACCEPT [12:1070]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FORWARD_direct -j ACCEPT
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A INPUT_direct -j ACCEPT
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A OUTPUT_direct -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
iptables-save - SSH DOESN'T WORK TOO...

Code: Select all

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*nat
:PREROUTING ACCEPT [5:371]
:INPUT ACCEPT [1:67]
:OUTPUT ACCEPT [2:134]
:POSTROUTING ACCEPT [2:134]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
-A PRE_public_allow -p tcp -m mark --mark 0x64 -j DNAT --to-destination 10.1.0.9:22
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*mangle
:PREROUTING ACCEPT [17:1649]
:INPUT ACCEPT [12:1285]
:FORWARD ACCEPT [5:364]
:OUTPUT ACCEPT [10:3037]
:POSTROUTING ACCEPT [14:3341]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
-A PRE_public_allow -p tcp -m tcp --dport 222 -j MARK --set-xmark 0x64/0xffffffff
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*security
:INPUT ACCEPT [12:1285]
:FORWARD ACCEPT [4:304]
:OUTPUT ACCEPT [10:3037]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*raw
:PREROUTING ACCEPT [17:1649]
:OUTPUT ACCEPT [10:3037]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8:2813]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDI_public_allow -m conntrack --ctstate NEW -m mark --mark 0x64 -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:36 2019

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: firewall-cmd - add-forward-port don't work

Post by jlehtone » 2019/08/23 16:38:23

You have three interfaces: ens33, br0, and virbr0
The virbr0 seems to have subnet 10.1.0.0/24

Who is the br0?

Commands to inspect network configuration:

Code: Select all

nmcli
nmcli c s
ip -4 ad
ip ro
brctl show
Other ways to see rules:

Code: Select all

iptables -S
iptables -vnL
(They can limit to single chain.)

The first rule in FORWARD of "works" is ACCEPT. That allows every packet to be forwarded. The others have rules to accept dnatted packets, but later in the chain.


It is also possible to monitor traffic. Not sure about the firewall log options, but tcpdump could listen on the host.

Code: Select all

tcpdump -nn -i ens33 port 222
tcpdump -nn -i virbr0 port 22
An incoming packet should show up in ens33.
Dnatted packet and then reply should show in virbr0.
The reply should pass the ens33 too.

eduardolucioac
Posts: 23
Joined: 2017/01/25 12:40:27

Re: firewall-cmd - add-forward-port don't work

Post by eduardolucioac » 2019/08/23 18:46:40

Who is the br0?
-> An interface bridged to ens33.
The virbr0 seems to have subnet 10.1.0.0/24
-> Yes!

The result of executing the command...

Code: Select all

[eduardo@eduardo-nb DEV_GUESTS]$ ssh root@172.16.13.8 -p 222
ssh: connect to host 172.16.13.8 port 222: Connection refused
... with this command...

Code: Select all

[root@localhost ~]# tcpdump -nn -i ens33 port 222
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
15:41:44.725676 IP 172.16.13.1.43294 > 172.16.13.8.222: Flags [S], seq 1577348226, win 29200, options [mss 1460,sackOK,TS val 1804734059 ecr 0,nop,wscale 7], length 0
... and this command...

Code: Select all

[root@localhost ~]# tcpdump -nn -i virbr0 port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on virbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
... running.

eduardolucioac
Posts: 23
Joined: 2017/01/25 12:40:27

Re: firewall-cmd - add-forward-port don't work

Post by eduardolucioac » 2019/08/23 18:59:46

The first rule in FORWARD of "works" is ACCEPT. That allows every packet to be forwarded. The others have rules to accept dnatted packets, but later in the chain.
We may be able to solve the problem if we find out why this works...

Code: Select all

iptables -I FORWARD -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload
... and why this not...

Code: Select all

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: firewall-cmd - add-forward-port don't work

Post by jlehtone » 2019/08/24 08:17:44

eduardolucioac wrote:
2019/08/23 18:46:40
Who is the br0?
-> An interface bridged to ens33.
What is that for?

If ens33 is enslaved to br0, then typically the ens33 should have no IP address and the address for "outside" is on the br0.
eduardolucioac wrote:
2019/08/23 18:59:46
We may be able to solve the problem if we find out why
Indeed.

Lets look at the initial FORWARD again (could use iptables -S FORWARD):

Code: Select all

  # XXX
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable # YYY
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct # ZZZ
-A FORWARD -j FORWARD_IN_ZONES # WWW
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
I did leave out couple empty chains (that have no effect here).
* The "works" adds an allow rule at XXX
* The "direct" adds an allow rule at ZZZ
* The "forward_port" adds an allow rule at WWW
* The YYY rule drops packets that would go into 10.1.0.0/24 before ZZZ and WWW

However, the system default FORWARD does not look like that. It is about:

Code: Select all

-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct # ZZZ
-A FORWARD -j FORWARD_IN_ZONES # WWW
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
The libvirt, when it does start "a network" (e.g. virbr0), injects rules (probably via firewalld):

Code: Select all

*filter
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
*nat
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
These are at the top of FORWARD and POSTROUTING, before regular firewalld ruleset. Since the netfilter operates on "first match" principle, these "rule".

Personally, I use bridged mode rather than routed, so I have almost no experience with this. However, I do have a faint recollection of seeing this issue before.

How to go around that? The direct rules have "passthrough" option. If I did guess right, the priority 3 should place our rule just before the first reject by libvirt (you definitely want your rule after the "related,established" rule):

Code: Select all

firewall-cmd --direct --passthrough --add-rule ipv4 filter FORWARD 3 -o virbr0 -d 10.1.0.9 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 nat PREROUTING 0 -p tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
(I would reboot the machine to see how libvirt and firewalld really play together.)


However, that is a "hack". IMHO, one should do this via the libvirt. That is our culprit that puts its rules where we don't expect them. No we have two paths:
A. Change where the libvirt-rules go. I think there is an upstream (bugzilla) request about that.
B. Make the libvirt add our rule into FORWARD when it adds rules. Libvirt does have documentation for doing that (filtering access to/from guest).

Check the option B.

eduardolucioac
Posts: 23
Joined: 2017/01/25 12:40:27

Re: firewall-cmd - add-forward-port don't work

Post by eduardolucioac » 2019/08/26 17:42:10

"What is that for?"/"Personally, I use bridged mode rather than routed, so I have almost no experience with this. However, I do have a faint recollection of seeing this issue before."
->

I understand your questions... This server (KVM host) does not need to have a bridge since the only ip it has is an internet ip. The bridge is only there for testing so we can test other scenarios.
"If ens33 is enslaved to br0, then typically the ens33 should have no IP address and the address for "outside" is on the br0."
-> This is exactly the case! Look!

Code: Select all

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 00:0c:29:76:c7:01 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0c:29:76:c7:01 brd ff:ff:ff:ff:ff:ff
    inet 172.16.13.8/24 brd 172.16.13.255 scope global noprefixroute br0
       valid_lft forever preferred_lft forever
    inet6 fe80::24e6:5ae3:86a0:e007/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:7b:a1:04 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/24 brd 10.1.0.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:7b:a1:04 brd ff:ff:ff:ff:ff:ff
6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:c0:78:af brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fec0:78af/64 scope link 
       valid_lft forever preferred_lft forever
"The libvirt, when it does start "a network" (e.g. virbr0), injects rules (probably via firewalld):"
-> I was already starting to suspect that! I will investigate and bring the answer to the forum.

Post Reply