Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Issues related to configuring your network
User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by jlehtone » 2020/05/04 07:55:29

Those FINAL_REJECTs are all from inside to port 5355/udp. IPv4 224.0.0.252 is a non-routable multicast address and so are IPv6 ffx2::/16.
https://en.wikipedia.org/wiki/Multicast_address
https://en.wikipedia.org/wiki/Link-Loca ... Resolution

In other words, the device 192.168.1.51 (I presume that the IPv6 address belongs to same device, and BTW, the fe80:: contains the MAC address too) attempts multicast name resolution (whatever that is).

Multicast DNS would be allowed, but that is to 224.0.0.251 5353/udp:

Code: Select all

        chain filter_IN_internal_allow {

                ip daddr 224.0.0.251 udp dport mdns ct state new,untracked accept
                ip6 daddr ff02::fb udp dport mdns ct state new,untracked accept
I have never understood multicast, but this looks like "works as designed".

The real issue:
The STATE_INVALID_DROP (from facebook) comes from the INPUT:

Code: Select all

IN=enp3s0 OUT= DST=65.64.63.62

Code: Select all

        chain filter_INPUT {
                ct state invalid log prefix "STATE_INVALID_DROP: "
                ct state invalid drop
If these are replies within existing connection (which they probably are) then the masquerade should have recognized them as such and restore the the original source (into the DST), which would have changed the routing into

Code: Select all

IN=enp3s0 OUT=enp2s0
and then the packet should match

Code: Select all

        chain filter_FORWARD {
                ct state established,related accept
As far as I can see, the problem is not in the firewall rules. They look fine.

The problem is thus that masquerade in kernel does not pick valid replies and "demasquerade" them.
(Or sites like Facebook send invalid packages, or a man in the middle mangles packages.)

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/05/04 13:11:20

jlehtone wrote:
2020/05/04 07:55:29
The problem is thus that masquerade in kernel does not pick valid replies and "demasquerade" them.
(Or sites like Facebook send invalid packages, or a man in the middle mangles packages.)
Cool, thank you, a lot of good information.

Before work today, I tried the direct nftables entries. Was super bare-bones (from the Gentoo site), but I uninstalled firewalld, rebooted, placed the two basic firewall entries in under /etc/nftables, then included them to load on boot time. Also, enabled packet forwarding. It seemed like maybe it worked? But it also looked kludgy, and I couldn't talk to the Internet from the server itself, so I backed everything out, and went back to firewalld for the time being. It's broke, but it's broke like it was before, not worse.

Will do more research during the day and, if I find something worth trying out, I'll try that after work.

It feels like, with prior versions of CentOS, there were tons of rock-solid examples of how to set your firewall up in this configuration, but now, with nftables and firewalld, there's not a lot of guidance.

It's also possible I'm hitting a bug with helpers or something.

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/05/04 13:27:22

jlehtone wrote:
2020/04/30 17:09:00
Plan B, for CentOS 8:
1. Erase firewalld
2. Write nftables ruleset
3. Enable nftables.service to load the ruleset on boot
https://wiki.gentoo.org/wiki/Nftables/Examples
This was the procedure I started this morning. It's probably something I should start after work. :lol:

I removed firewalld and rebooted the system.

I had two files in /etc/nftables that looked like this:

nftables_firewall.nft:

Code: Select all

#!/usr/sbin/nft -f

# firewall
table ip filter {
        # allow all packets sent by the firewall machine itself
        chain output {
                type filter hook output priority 100; policy accept;
        }

        # allow LAN to firewall, disallow WAN to firewall
        chain input {
                type filter hook input priority 0; policy accept;
                iifname "enp2s0" accept
                iifname "enp3s0" drop
        }

        # allow packets from LAN to WAN, and WAN to LAN if LAN initiated the connection
        chain forward {
                type filter hook forward priority 0; policy drop;
                iifname "enp2s0" oifname "enp3s0" accept
                iifname "enp3s0" oifname "enp2s0" ct state related,established accept
        }
}
nftables_nat.nft:

Code: Select all

#!/usr/sbin/nft -f

# NAT
table ip nat {
        chain prerouting {
                type nat hook prerouting priority 0; policy accept;
        }

        # for all packets to WAN, after routing, replace source address with primary IP of WAN interface
        chain postrouting {
                type nat hook postrouting priority 100; policy accept;
                oifname "enp3s0" masquerade
        }
}
I enabled forwarding manually with this:

Code: Select all

sysctl -w net.ipv4.ip_forward=1
Adding an entry to /etc/sysctl.d named 10-forwarding.conf to enable ip_forwarding.

I also added these two files to the /etc/sysconfig.d/nftables.conf file so they would load on boot. I then rebooted.

After reboot, I could see that ip fowarding was enabled, and my phone (on wifi) could access the Internet.

I realized I had ran out of time, so I didn't get a chance to really test things upstairs, but the one thing that did not work was the server didn't have any access to the Internet after I did this. ping 8.8.8.8 failed, etc. But my phone (downstairs) could get to the Internet, so that's cool, maybe? It's also possible that my phone was detecting no Internet access and switching to cellular. Will test this more tonight.
jlehtone wrote:
2020/04/30 17:09:00
[EDIT]:
Found a 8.0 vs 8.1 change: viewtopic.php?p=307811#p307811
(Knew that I had posted it, but not the details.)
Is this the "Zone drift" you mention?
Regarding "zone drift" yes, possibly. I don't know if "zone drift" was allowing me to do something before, and then they locked it up, so that loophole is what's causing me issues or not. But in the firewalld.conf there's a switch called "AllowZoneDrifting" which, when enabled, does not fix my problem. There's also another feature called "AutomaticHelpers" which now defaults to "system" but I've also tried "yes" and "no" which did nothing.

It's also possible that "AllowZoneDrifting" might be in the .conf file, but not in 0.7.0_5 because on the firewalld.org page, they talk about removing the feature in one set of versions, which broke a lot of firewalls, so they added it back in as an option, but people need to move away from zone drifitng because it breaks the firewall security.

It doesn't seem like my simple use case, if configured properly, would bump up against "zone drifting" seeing as I have two zones, and they're very clearly delimited: internal is inside, external is outside. Two physical interfaces.

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/05/04 15:55:32

jlehtone wrote:
2020/05/04 07:55:29
As far as I can see, the problem is not in the firewall rules. They look fine.

The problem is thus that masquerade in kernel does not pick valid replies and "demasquerade" them.
(Or sites like Facebook send invalid packages, or a man in the middle mangles packages.)
I spoke with some folks over on the #firewalld IRC chat, and they think it's 100% a conntrack issue, as all the other rules (which worked before the kernel upgrade on 2020-04-24T11:48:20Z [kernel.x86_64 4.18.0-147.8.1.el8_1]) have not changed. Although at this point, I've cleared and re-entered the firewalld rules from scratch so many times, it's fairly "clean" at this point.

Is this still the correct group to talk about conntracking/kernel issues?

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by jlehtone » 2020/05/04 16:58:02

I just noticed that the hook priorities on the Gentoo wiki look wrong.
https://access.redhat.com/documentation ... g-networks
https://access.redhat.com/documentation ... h-nftables
https://wiki.nftables.org/wiki-nftables ... tion_(NAT)
https://wiki.nftables.org/wiki-nftables ... ing_chains
The priorities used in iptables:
NF_IP_PRI_NAT_DST (-100): destination NAT (== PREROUTING)
NF_IP_PRI_FILTER (0): filtering operation, the filter table (== INPUT, OUTPUT, FORWARD)
NF_IP_PRI_NAT_SRC (100): source NAT (== POSTROUTING)

"When to use firewalld" in https://access.redhat.com/documentation ... g-networks is quite telling.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by jlehtone » 2020/05/04 17:03:50

amarand wrote:
2020/05/04 15:55:32
Is this still the correct group to talk about conntracking/kernel issues?
If we have set the rules correctly, then the issue is inside the kernel.
If the issue is in the kernel, then Red Hat is the only one who can make a difference.
(Note that RHEL 8.2 beta is already out and official release is in this month, if their 6-month release plan holds.)

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/05/04 17:43:43

jlehtone wrote:
2020/05/04 16:58:02
I just noticed that the hook priorities on the Gentoo wiki look wrong.
https://access.redhat.com/documentation ... g-networks
https://access.redhat.com/documentation ... h-nftables
https://wiki.nftables.org/wiki-nftables ... tion_(NAT)
https://wiki.nftables.org/wiki-nftables ... ing_chains
The priorities used in iptables:
NF_IP_PRI_NAT_DST (-100): destination NAT (== PREROUTING)
NF_IP_PRI_FILTER (0): filtering operation, the filter table (== INPUT, OUTPUT, FORWARD)
NF_IP_PRI_NAT_SRC (100): source NAT (== POSTROUTING)
Regarding the priorities.... Do you think the order might be problematic, and it's something I could adjust?
jlehtone wrote:
2020/05/04 16:58:02
"When to use firewalld" in https://access.redhat.com/documentation ... g-networks is quite telling.
Yes, very telling. And I've read that before.... My concern has been, firewalld has worked so well, up to this point. I think the change may have been from 8.0 to 8.1, not from 8.1 original to the latest three kernel revisions.

So seeing that...knowing my environment, it seems like I should be uninstalling firewalld (as you already suggested as a "Plan B"), and working directly with nftables to configure my "complex" (*laughs*) firewall?

Is there a good how-to out there to properly configure this exclusively using nft commands directly?

I would prefer to use firewalld...but I do see that if the upstream is stating that anything more complex than a workstation shouldn't be using firewalld, then maybe my configuration isn't supported? But nft should be.

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/05/04 17:44:43

jlehtone wrote:
2020/05/04 17:03:50
amarand wrote:
2020/05/04 15:55:32
Is this still the correct group to talk about conntracking/kernel issues?
If we have set the rules correctly, then the issue is inside the kernel.
If the issue is in the kernel, then Red Hat is the only one who can make a difference.
(Note that RHEL 8.2 beta is already out and official release is in this month, if their 6-month release plan holds.)
And from what you've seen, my rules are set correctly....

So it's probably a kernel/conntrack issue.

Having said that, is there a "stop gap" measure/workaround, so I can get all of my services back on-line again?

Post Reply