loading ip entries from file to ipset via firewall-cmd

Support for security such as Firewalls and securing linux
Post Reply
arcis
Posts: 7
Joined: 2021/08/02 04:30:09

loading ip entries from file to ipset via firewall-cmd

Post by arcis » 2022/02/03 11:13:51

I'm running CentOS 8 Stream. When I try to load IP addresses from a file into ipset, it takes forever. Some of my services are accessible via ipset entries, and the feeds are updated twice a week. I can't shut down the service for 10-15 minutes to load addresses into ipset. On CentOS 7, this used to be quite fast. What can I do to speed up the address loading in ipsets? I'm aware that firewalld does not support ipset swapping. Is it a good idea to downgrade the firewalld version?

I am creating ipset as -

Code: Select all

firewall-cmd --permanent --new-ipset=allow4 --type=hash:net --option=family=inet --option=hashsize=4096 --option=maxelem=200000
Loading entries as -

Code: Select all

firewall-cmd --permanent --ipset=allow4 --add-entries-from-file="$IPV4_CA"

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: loading ip entries from file to ipset via firewall-cmd

Post by jlehtone » 2022/02/03 13:04:33

I think I've seen mentions of firewall having issues, like memory consumption.
Not sure whether it was firewalld, netfilter, or nftables.

The backend (of firewalld) in Stream 8 in nftables. CentOS 7 had netfilter (aka "iptables").
Command "iptables" in Stream 8 is a wrapper/translator for/to "nft".

One could remove firewalld.service and use nftables.service instead. Then one has to write nftables ruleset.
The nftables has "sets", which seem analogous to ipsets. The question is: Would that be more efficient?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: loading ip entries from file to ipset via firewall-cmd

Post by TrevorH » 2022/02/03 13:11:27

It was firewalld. It's a toy.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

arcis
Posts: 7
Joined: 2021/08/02 04:30:09

Re: loading ip entries from file to ipset via firewall-cmd

Post by arcis » 2022/02/03 13:23:40

jlehtone wrote:
2022/02/03 13:04:33
I think I've seen mentions of firewall having issues, like memory consumption.
Not sure whether it was firewalld, netfilter, or nftables.

The backend (of firewalld) in Stream 8 in nftables. CentOS 7 had netfilter (aka "iptables").
Command "iptables" in Stream 8 is a wrapper/translator for/to "nft".

One could remove firewalld.service and use nftables.service instead. Then one has to write nftables ruleset.
The nftables has "sets", which seem analogous to ipsets. The question is: Would that be more efficient?
The problem appears to be an overlap check that employs a for loop. After disabling the function in client.py, I rebuilt a custom rpm. The for loop consumes resources and consumes a significant amount of time. Python programs irritate me because they are slow and cause problems.

I wish there were tables similar to pf, where a user could add a table entry to pf.conf and populate it with addresses and subnets. A good lesson to be learned here is that not every version upgrade works as intended.

arcis
Posts: 7
Joined: 2021/08/02 04:30:09

Re: loading ip entries from file to ipset via firewall-cmd

Post by arcis » 2022/02/03 13:25:09

TrevorH wrote:
2022/02/03 13:11:27
It was firewalld. It's a toy.
I've adopted firewalld; it's good, but it could be better.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: loading ip entries from file to ipset via firewall-cmd

Post by jlehtone » 2022/02/03 14:25:11

arcis wrote:
2022/02/03 13:23:40
I wish there were tables similar to pf, where a user could add a table entry to pf.conf and populate it with addresses and subnets.
I don't know pf, but that sounds like what one would end up with nftables.service.


A zone-based firewall is conceptually nice ... if user can adopt that line of thought.

The issue with FirewallD before version 0.9, the one in RHEL 8.5, was that it was in practice only for INPUT.
The 0.9 did add support for FORWARD filter "policies" that are essential in router (although EL8 as router has other issues too).
The 1.0 (RHEL 9) will finally support OUTPUT filter ...

arcis
Posts: 7
Joined: 2021/08/02 04:30:09

Re: loading ip entries from file to ipset via firewall-cmd

Post by arcis » 2022/02/03 14:28:32


The problem appears to be an overlap check that employs a for loop. After disabling the function in client.py, I rebuilt a custom rpm. The for loop consumes resources and consumes a significant amount of time. Python programs irritate me because they are slow and cause problems.

I wish there were tables similar to pf, where a user could add a table entry to pf.conf and populate it with addresses and subnets. A good lesson to be learned here is that not every version upgrade works as intended.
It worked, from 12+ minutes to 1 minute 47 seconds.

Code: Select all

time firewall-cmd --permanent --ipset=whitelist4 --add-entries-from-file="$IPV4_CA"
success

real    1m47.113s
user    1m46.051s
sys     0m0.142s

arcis
Posts: 7
Joined: 2021/08/02 04:30:09

Re: loading ip entries from file to ipset via firewall-cmd

Post by arcis » 2022/02/04 05:45:14

A zone-based firewall is conceptually nice ... if user can adopt that line of thought.
The reason I chose firewalled is that I want to make certain services available to a specific ASN, region, or country. This significantly reduces the attack surface, as well as mail notifications in my case.
The issue with FirewallD before version 0.9, the one in RHEL 8.5, was that it was in practice only for INPUT.
The 0.9 did add support for FORWARD filter "policies" that are essential in router (although EL8 as router has other issues too).
The 1.0 (RHEL 9) will finally support OUTPUT filter ...
I upgraded firewalld to 1.0.3 locally, and I hope to be able to use it extensively in the near future.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: loading ip entries from file to ipset via firewall-cmd

Post by jlehtone » 2022/02/04 11:57:18

arcis wrote:
2022/02/04 05:45:14
A zone-based firewall is conceptually nice ... if user can adopt that line of thought.
The reason I chose firewalled is that I want to make certain services available to a specific ASN, region, or country.
We have been writing zone-based rulesets long before firewalld was created. Firewalld is just a convenience tool that lets user express needs in certain way and then translates those to actual ruleset. This is nice in, say "port forward", where "I want X to go to Y" converts into multiple (dNAT and filter) rules in kernel.

Post Reply