Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Issues related to configuring your network
amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/04/30 13:43:45

Had posted this against someone else's topic earlier, but decided it might be a separate, different issue.

Having issues related to RELATED,ESTABLISHED inbound packets since upgrading to the latest version of CentOS (8.1.1911) and the latest version of Firewalld (firewalld-0.7.0-5.el8_1.1.noarch).

Had never attributed my VPN issues to this problem until I saw a related post, talking about VPN being blocked after upgrade. My boss uses VPN every day, as do I, but he never has any issues connecting. With me, it takes quite a few tries. I have commercial grade Internet.

After having difficulties connecting to Apple-related sites, which use a single-sign on method and site to authenticate, I realized that something was off.

I created a new user account on my PC, just to perform some troubleshooting (I did NOT suspect the firewall at this point, as it had worked perfectly, with very basic masq/nat rules since the first release of CentOS 8), and I still had the issue.

I finally dug into the firewall, enabled logging, and lo and behold, I was seeing all of the return traffic being blocked. Port 443 and others.

Finally, after an entire week of troubleshooting, last night, I took a laptop and connected it directly to my cable modem, on the other side of the firewall, and the Apple connections worked fine. Move it back inside the firewall, problem comes back.

I created a CentOS bug, because behavior changed, somewhere in the firewall, and it wasn't documented, and I also can't figure out how to fix it.

https://bugs.centos.org/view.php?id=17310

(I think I did a fairly good job of documenting the steps I took.)

I do know two things have changed semi-recently:

One of the things was a security related issue relating to a feature called "AutomaticHelpers." They removed the automatic helpers by default, so the firewall matching is different, which I understand. But I believe (and I could be wrong, but it seems like maybe this is true), they removed Automatic Helpers, yet they haven't allowed people to override the sloppy, bad-security feature and turn it back on, while they figure out how to make it work without them?

Also, there's a security-related issue with a sloppy handling of packets called "Zone Drifting" which they've also disabled, to make the firewall more secure, but haven't done a very good job of explaining how to make it work with this feature disabled (by default). Again, this is one of those things they changed/removed, to make things more secure, but the documentation doesn't explain how to work around it.

Both of these features have entries in firewalld.conf, but turning one or both of these features in any direction (yes, no, system) does anything. Tried all permutations. Restarted my firewall system and the firewall service itself many, many times this week.

Getting a little frustrated, because I have decades of networking and firewall experience, and this stuff (basic dual-homed masqueraded NAT firewall) should "just work."

I'm sure it's something stupid/silly, but it's starting to impact my ability to get work done some days (when it takes 30+ minutes to finally get into VPN), and it also seems to be slowing down other sites and services when packets are being dropped, that should be allowed.

Aside from the information supplied in the bug report, are there any other configuration settings I could share here that would help with troubleshooting?

Again, my configuration is very basic:

1) Dual-homed (two, distinct network cards, with reliable naming and MAC address linking enabled so there's no drift [confirmed])
2) Inside is the trusted network, labeled Internal (as designed).
3) Outside is the non-trusted network (Internet), labeled External (as designed).
4) Masquerade is enabled.
5) NAT is enabled.
6) All systems inside the firewall should be able to connect to everything (all IPs, all ports/services) with no restrictions.
7) Only controlled services are enabled externally (https, ssh)
8) As has been the way since before the days of firewalld, all RELATED,ESTABLISHED packets inbound should be forwarded to the associated system. (This allows for return responses to work. I can see some of these return responses are being blocked in the logs. They were not blocked before the recent CentOS/Firewalld update.)

Hopefully someone can assist, because it's driving me a little batty. :mrgreen:

Also, one thing I noticed in the firewalld.conf file is the FirewallBackend is using "nftables" by default, versus the iptables stack? Did that also change recently? I'm wondering how many changes occurred in Firewalld in the last iteration? One of more of these moving parts is likely what's changed and caused behavior to change as well. Thanks again!

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by jlehtone » 2020/04/30 17:09:00

amarand wrote:
2020/04/30 13:43:45
Also, one thing I noticed in the firewalld.conf file is the FirewallBackend is using "nftables" by default, versus the iptables stack? Did that also change recently?
Define "recently".
The kernel of CentOS 8 does not have "iptables stack", aka netfilter. It does have "nftables stack".
Nftables has been available in upstream kernels >= 3.13. CentOS 7's kernel is based on upstream 3.10.
Nftables is thus almost as old as CentOS 7, but we did not encounter it before CentOS 8, last summer.

Basic router:
1. Internal interface to zone 'trusted'
2. External interface to zone 'external'
3. Add required services to 'external'
4. done
IME (on CentOS 7), this enables masquerade and routing too.
Alas, I have strong reservations about the way firewalld filters routed traffic.

Plan B, for CentOS 8:
1. Erase firewalld
2. Write nftables ruleset
3. Enable nftables.service to load the ruleset on boot
https://wiki.gentoo.org/wiki/Nftables/Examples


[EDIT]:
Found a 8.0 vs 8.1 change: viewtopic.php?p=307811#p307811
(Knew that I had posted it, but not the details.)
Is this the "Zone drift" you mention?

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/04/30 18:28:11

Recently: the past week or two.

Okay, so...I need to learn some entirely new thing? I had just gotten used to firewalld doing a bang-up job, and I didn't have to "write any code." I don't really want to write rules by hand. Is firewalld not a supported configuration in CentOS 8? The "Write nftables ruleset" sounds a lot less user-friendly than firewalld.

Also, my use-case is fairly popular, I imagine:

A few dozen systems (Windows, Mac, Linux, IoT, set-top boxes) inside the firewall, needing 100% access to outside the firewall. NAT. Masquerade. Only two services are allowed in (ssh from a limited, single server outside the firewall, and https...that's about it).

Is there a simple way to accomplish this with...whatever's being recommended in CentOS 8.1?

I like the ability to write complex rules if I need to, but honestly, I just want it to work at this point.

Back when I upgraded from CentOS 6 to 7, I had to re-learn firewalld. Now, are you saying firewalld is out?

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/04/30 18:29:36

I guess what I'm saying is, if there's a new way to do this basic thing, there probably should be someone out there who's written a "here's how you do it in CentOS 8.1" but I haven't found that yet. I rely almost entirely on others writing "how to" examples, and right now, 8.1 is failing in that respect. All the previous how to documents don't account for all these rapid changes that aren't well documented.

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/04/30 18:34:36

A lot of the posts I've read are people complaining about the inability to filter specific protocols/ports from the internal network, which is not a concern for me (yet). Really, I just want everything on the inside to be able to talk to everything on the outside, and every packet that comes in that matches with an associated established connection out, can also come in and be routed to the appropriate NAT'd internal system. It's a very basic configuration, that I'm shocked isn't well documented in the examples. I was up until midnight last night, pouring over dozens of documents, searching for a solution. Is no one doing this on 8.1? I can't be the only one.... This is why I stuck with CentOS 6 (and 7) as long as I did.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by jlehtone » 2020/04/30 22:03:10

I don't say that firewalld is out. I say that for some of my use cases it is not yet appropriate.

However, for you:
1. Install a test system. VM, probably, with two interfaces. CentOS 8 (i.e. now "8.1")
2. Assign one interface to builtin zone "trusted" (nmcli con mod eth1 connection.zone trusted)
3. Assign other interface to builtin zone "external" (nmcli con mod eth0 connection.zone external)
4. Add service https to zone "external" (firewall-cmd --zone=external --add-service=https --permanent)
5. Reboot
6. Check that "sysctl net.ipv4.ip_forward" shows "1"
7. Check that ruleset "nft list ruleset" has expected rules (for masquerade and filter)

If all checks out, then deploy to production.

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/04/30 22:09:03

The part that confuses me is, in this example, I'm using nmcli and firewall-cmd. Is there a reason why it's not one-stop-shopping yet on the firewall side?

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by jlehtone » 2020/05/01 08:43:42

Just guessing here.

https://access.redhat.com/discussions/1455033

Implies that zone is a property of a connection.
Interface is in a zone rather than zone has interfaces.

The property is stored in the configuration of the connection.
Most connection types have configuration stored in the "ifcfg*"-files.
The ifcfg-files are legacy, for the network.service.

It is logical to set zone of connection when a connection is created. With NetworkManager (NM).
If you want a connection to "drop", it makes no sense to create it with default zone ("public").
You could change default to drop in firewalld (FD), create connection with NM, set zone of connection to drop in FD, and then restore default zone in FD, but what about the connections that use the default during this?

The config is in one place.
Should all frontends (NM, FD, and legacy system-config-network) be able to manipulate it?

amarand
Posts: 38
Joined: 2006/09/12 19:09:07
Location: Columbus, Ohio, USA
Contact:

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by amarand » 2020/05/01 12:22:46

jlehtone wrote:
2020/05/01 08:43:42
Just guessing here.

https://access.redhat.com/discussions/1455033

Implies that zone is a property of a connection.
Interface is in a zone rather than zone has interfaces.

The property is stored in the configuration of the connection.
Most connection types have configuration stored in the "ifcfg*"-files.
The ifcfg-files are legacy, for the network.service.

It is logical to set zone of connection when a connection is created. With NetworkManager (NM).
If you want a connection to "drop", it makes no sense to create it with default zone ("public").
You could change default to drop in firewalld (FD), create connection with NM, set zone of connection to drop in FD, and then restore default zone in FD, but what about the connections that use the default during this?

The config is in one place.
Should all frontends (NM, FD, and legacy system-config-network) be able to manipulate it?

Okay, so the weird thing is, everything was working just fine before the latest update to 0.7.0_5 of Firewalld and the overall update to CentOS 8.1 from 8.0.

I don't have any issues with which zone an interface is in (notice how there are only two zones listed as Active - and this configuration worked since the first release of CentOS 8 and even CentOS 7 before that):

# firewall-cmd --get-active-zones
external
interfaces: enp3s0
internal
interfaces: enp2s0

# firewall-cmd --list-all-zones
block
target: %%REJECT%%
icmp-block-inversion: no
interfaces:
sources:
services:
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


dmz
target: default
icmp-block-inversion: no
interfaces:
sources:
services: ssh
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


drop
target: DROP
icmp-block-inversion: no
interfaces:
sources:
services:
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


external (active)
target: %%REJECT%%
icmp-block-inversion: no
interfaces: enp3s0
sources:
services: http https imap imaps smtp smtps
ports:
protocols:
masquerade: yes
forward-ports:
source-ports:
icmp-blocks:
rich rules:
rule family="ipv4" source address="205.166.94.0/24" service name="ssh" log level="notice" accept

home
target: default
icmp-block-inversion: no
interfaces:
sources:
services: cockpit dhcpv6-client mdns samba-client ssh
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


internal (active)
target: ACCEPT
icmp-block-inversion: no
interfaces: enp2s0
sources:
services: RH-Satellite-6 amanda-client amanda-k5-client amqp amqps apcupsd audit bacula bacula-client bb bgp bitcoin bitcoin-rpc bitcoin-testnet bitcoin-testnet-rpc bittorrent-lsd ceph ceph-mon cfengine cockpit condor-collector ctdb dhcp dhcpv6 dhcpv6-client distcc dns dns-over-tls docker-registry docker-swarm dropbox-lansync elasticsearch etcd-client etcd-server finger freeipa-4 freeipa-ldap freeipa-ldaps freeipa-replication freeipa-trust ftp ganglia-client ganglia-master git grafana gre high-availability http https imap imaps ipp ipp-client ipsec irc ircs iscsi-target isns jenkins kadmin kdeconnect kerberos kibana klogin kpasswd kprop kshell ldap ldaps libvirt libvirt-tls lightning-network llmnr managesieve matrix mdns memcache minidlna mongodb mosh mountd mqtt mqtt-tls ms-wbt mssql murmur mysql nfs nfs3 nmea-0183 nrpe ntp nut openvpn ovirt-imageio ovirt-storageconsole ovirt-vmconsole plex pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy prometheus proxy-dhcp ptp pulseaudio puppetmaster quassel radius rdp redis redis-sentinel rpc-bind rsh rsyncd rtsp salt-master samba samba-client samba-dc sane sip sips slp smtp smtp-submission smtps snmp snmptrap spideroak-lansync spotify-sync squid ssdp ssh steam-streaming svdrp svn syncthing syncthing-gui synergy syslog syslog-tls telnet tentacle tftp tftp-client tile38 tinc tor-socks transmission-client upnp-client vdsm vnc-server wbem-http wbem-https wsman wsmans xdmcp xmpp-bosh xmpp-client xmpp-local xmpp-server zabbix-agent zabbix-server
ports: 5201/tcp
protocols:
masquerade: yes
forward-ports:
source-ports:
icmp-blocks:
rich rules:


libvirt
target: ACCEPT
icmp-block-inversion: no
interfaces:
sources:
services: dhcp dhcpv6 dns ssh tftp
ports:
protocols: icmp ipv6-icmp
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
rule priority="32767" reject

public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: dhcpv6-client ssh
ports: 3389/tcp 5201/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


trusted
target: ACCEPT
icmp-block-inversion: no
interfaces:
sources:
services:
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


work
target: default
icmp-block-inversion: no
interfaces:
sources:
services: cockpit dhcpv6-client ssh
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Firewalld Blocking RELATED,ESTABLISHED Inbound Since Update

Post by jlehtone » 2020/05/01 13:09:25

These are the builtin internal and external:

Code: Select all

internal
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: cockpit dhcpv6-client mdns samba-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

external
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: ssh
  ports: 
  protocols: 
  masquerade: yes
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
You have made changes to both. Some are obvious, but not all.

What is the goal of external rich rule:

Code: Select all

rule family="ipv4" source address="205.166.94.0/24" service name="ssh" log level="notice" accept
The glaring issue though is that both zones have:
masquerade: yes

Post Reply