Firewalld + ipset = big problems

[acicali]

Hi,

I've been using CentOS 6 for many moons and have recently setup a server using CentOS 8.2 (sorry CentOS 7... I skipped right over you =P). Everything has been working rather well, except now I've hit a problem that I can't find a solution to (even after many hours of Googling). The problem is, after adding a file of IP addresses to an IPSET, firewall-cmd seems no longer stable. First, it takes *extremely long* to --reload, second it may even crash and throw DBus errors.

The steps to reproduce are fairly straightforward...

• curl https://www.ipdeny.com/ipblocks/data/countries/cn.zone --output cn.zone # sorry China, it's not personal
• firewall-cmd --permanent --new-ipset=blocklist_v4 --type=hash:net --option=family=inet --option=hashsize=4096 --option=maxelem=200000
• firewall-cmd --permanent --ipset=blocklist_v4 --add-entries-from-file="cn.zone"
• firewall-cmd --permanent --zone=drop --add-source=ipset:blocklist_v4
• firewall-cmd --reload

At this point, firewalld will drain 100% cpu for long enough to be concerned. If I toss one or two more countries into the mix, typically my only way out is to shut firewalld down (systemctl stop firewalld), use firewall-offline-cmd to remove and delete the ipset rule, then restart firewalld.

Here's a bug filed with RedHat (that *might* be related) mentioning slow loading of XML files:
https://bugzilla.redhat.com/show_bug.cgi?id=1416817

Here's a bug filed with CentOS (that definitely *seems* related) mentioning slow --reload after adding an ipset rule:
https://bugs.centos.org/view.php?id=17525

Note that in the CentOS bug report, it mentions "a large ipset of 100k entries", however firewalld seems unreasonably problematic even with just the CN zone, which contains a total of about 8.5k entries.

Please correct me if I'm totally off-base here... but isn't this a fairly common need on a public-facing production server? How is it possible that this could be non-functioning? Perhaps most CentOS 8 deployments rely on a different machine for firewalling? It's hard for me to understand how this could be broken. Am I just being a whiny baby and I should go away and write my own operating system plus firewall if CentOS isn't good enough for me?

Thanks for reading! I very much appreciate any advice you can throw my way.

[BShT]

once you create an ipset entry and a firewall rule you don´t need to restart

just insert (or remove) more and more entries to ipset

[BShT]

feed an ipset table with a huge list of IPs can be a long task and leave your firewall at inconsistent state until it ends

so i never restart to feed ipset, in my case is to allow, duplicated IPs at br.zone will never be duplicated at ipset table

cat populate.sh

#!/bin/bash

curl https://www.ipdeny.com/ipblocks/data/countries/br.zone --output /etc/firewall/br.zone

BRASIL_IPS=/etc/firewall/br.zone

for IPS in $(cat ${BRASIL_IPS} | egrep -v "^#"); do
ipset -A brasil ${IPS}

done

##crontab it once a week or at your need

[acicali]

Since those ipblocks can change, I'd typically create a process that removes and recreates these ipsets monthly or so. A bigger problem is that once these ipsets are in place, firewalld in general is problematic. If the server also employs fail2ban, that could easily allow anyone to to create a denial-of-service scenario... any process that touches firewalld might cause system issues.

[TrevorH]

When you want to recreate the set, use a different one and then use ipset swap to swap the one for the other. That's atomic and quick and then you can delete the old one afterwards.

[acicali]

@BSht - so this would be outside of firewalld? My concern is that once these ipsets are in place, the server seems no longer "safe" having firewalld installed... if the server is ever restarted it could take an *hour* just for firewalld to finish goofing around.

[TrevorH]

Friends don't let friends use firewalld...

Having said that, make sure your system is fully up to date using yum update as there were performance problems in earlier versions of firewalld that were addressed by updates.

[acicali]

I did yum update it.... "friends don't let friends use firewalld..." - is that for real? Are there a lot of know issues with it? If that's my path forward (learning to write policies directly in nftables) then so be it. Having someone with experience (looking at you, TrevorH ) confirm that for me would be good though - I don't want to march in the wrong direction needlessly.

[acicali]

Just want to clarify - I find firewalld's syntax to be fairly easy and definitely prefer it over iptables... but I need a working solution, and I'm not getting that vibe from firewalld.

[BShT]

@BSht - so this would be outside of firewalld? My concern is that once these ipsets are in place, the server seems no longer "safe" having firewalld installed... if the server is ever restarted it could take an *hour* just for firewalld to finish goofing around.

yes, you create your firewall rules and your ipset table then you feed ipset table

the beauty of ipset is speed and "hot swap"

processes your data outside firewall