fail2ban with firewallcmd-ipset does not work on 7.3 1611

Support for security such as Firewalls and securing linux
Post Reply
smithgcovert1
Posts: 8
Joined: 2015/12/31 20:57:57

fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by smithgcovert1 » 2016/12/18 01:02:17

Hello,

This is both a public service announcement, and a request for help:

After upgrading to 7.3 1611 this week, after all of my initial checks and looks, it seemed like everything was A-OK. But then, after logging into my internet facing server a couple of days later, I found that there had been over 5000 failed ssh login attempts.

Since I use fail2ban to block these, that was really weird. Its not unusual for me to find that 100 or so attempts had been made, but since fail2ban blocks the ip addresses after 3 failed attempts, I usually don't get more than that.

It turns out that somehow the upgrade to 7.3 has broken the fail2ban action that I had been using: "firewallcmd-ipset"

Looking at the output of firewall-cmd, everything appeared correct:
- firewall-cmd --direct --get-all-rules showed what I would expect as the set of jail rules
- ipset list showed what I would expect as the correct list of banned IP addresses in the above rules

Everything seemed correct: fail2ban was running the correct commands, the required ip rules and ip addresses were in the right place.

Sure enough, I tried it from a remote machine I have access to, and after 10 failed attempts, I was still able to reach my machine on the ssh port. The firewall rules were not taking effect.

To work around the problem in the short term, I switched my default action from "firewallcmd-ipset" to "iptables-allports"

This still seems to work -- but to me the results are much more messy. My iptables are now filled with individually blocked ip addresses.

Has anyone else hit this, or found a solution to get the "firewallcmd-ipset" action type working again?

Cheers,
Greg

mghe
Posts: 766
Joined: 2015/11/24 12:04:43
Location: Katowice, Poland

Re: fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by mghe » 2016/12/18 07:34:58

I have similar configuration on one of my server and update server form 7.2 to 7.3 without any problems.

If you use firewall-cmd you should settings banaction = firewallcmd-ipset.

Did you check status of sshd jail and logs of fail2ban?

Could yo show config of sshd jail?

smithgcovert1
Posts: 8
Joined: 2015/12/31 20:57:57

Re: fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by smithgcovert1 » 2016/12/18 21:06:17

I will need to reconfigure my server back to using firewallcmd-ipset as my banaction to recreate some of the problem.

But that is what I did have in my config -- in order to get things working, I had to switch:

Code: Select all

banaction = firewallcmd-ipset
to

Code: Select all

banaction = iptables-allports
I have no customizations for sshd jail other than setting it "enabled=true"

At the time it not working, it seemed more like a problem between firewalld and the actual network tables, than between fail2ban and firewall-cmd.

What I mean by that:
No error messages in fail2ban logs. The firewall-cmd commands were succeeding correctly
A call to "fail2ban-client show sshd" showed the jail was enabled, and listed the IP addresses which were supposed to be banned.
A call to "firewall-cmd --direct --get-all-rules" showed the list of drop rules that should be enabled

Code: Select all

ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-noscript src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-modsecurity src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -m set --match-set fail2ban-postfix-sasl src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-badbots src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-nohome src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports smtp,465,submission -m set --match-set fail2ban-postfix src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-roundcube-auth src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-shellshock src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports 10000 -m set --match-set fail2ban-webmin-auth src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports pop3,pop3s,imap,imaps,submission,465,sieve -m set --match-set fail2ban-dovecot src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-overflows src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-fakegooglebot src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd-ddos src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-botsearch src -j REJECT --reject-with icmp-port-unreachable
ipv4 filter INPUT 0 -p tcp -m multiport --dports http,https -m set --match-set fail2ban-apache-auth src -j REJECT --reject-with icmp-port-unreachable
- ipset list showed the correct list of ip addresses that should be banned, and matched with the above rules.

What seemed to be missing was: it was like the above rules that are shown by the firewall-cmd were not actually applied to the iptables filters. I couldn't find any reference to the above rules in "iptables -L"

I did notice that firewalld is one of the components updated in 7.3. Maybe there is something generally wonky or weird with my firewalld setup. I do have a lot of rules configured for firewalld, as my server is a NAT router, internet server, and my VPN. So there are rules for those -- unfortunately I don't really have enough experience to debug firewalld.

Cheers,
Greg

mikeyp
Posts: 1
Joined: 2017/03/29 03:30:57

Re: fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by mikeyp » 2017/03/29 03:52:11

I have seen pretty much the same behavior - fail2ban adds IP's to the IPSets, but firewalld does not block those IP's. This used to work, not sure exactly what broke it, but I suspect a change in firewalld.

There is a relevant issue open against fail2ban FirewallD Ban Actions #1474

The problem seems to be triggered by the use of firewall-cmd --direct to configure the firewall. The solution proposed in that issue is to change the ban and unban actions to add/remove the offending IP to/from the firewalld 'drop' zone with a custom firewalld action. The guts of that change is the following commands to manipulate firewalld:

Code: Select all

actionban = firewall-cmd --add-source=<ip> --zone=drop && firewall-cmd --add-source=<ip> --zone=drop --permanent
actionunban = firewall-cmd --remove-source=<ip> --zone=drop && firewall-cmd --remove-source=<ip> --zone=drop --permanent
Details are in the github issue. I implemented this, and it seems to be working so far, at least for bans. I still need to verify unbans and timeouts work correctly.

ILMostro
Posts: 3
Joined: 2015/04/25 05:47:31

Re: fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by ILMostro » 2017/06/10 03:58:24

This is an excellent source of information. I've noticed the fail2ban service ineffective on my RHEL7.3 system as well. The logs are always, essentially, empty, except for startup, shutdown, or update action messages. I believe the earlier method to deny IPs was to use iptables, which had stayed in place long after the move to firewalld was made. Eventually, using the iptables method proved problematic, obviously; and the rpm packages were missing an active owner/contributor for support. The addition of the `fail2ban-firewalld` package seemed to resolve any issues, at least if judging by the fact that there were no more SELinux messages and other run-time errors in plain sight.

Ultimately, however, it seems that additional investigation MUST be performed around the fail2ban software on RHEL/CentOS systems (perhaps even on Fedora), as it's used with the assumption that it works as intended.

Historical record of bugzilla.redhat.com bugs to somewhat support my introductory claims:

https://www.google.com/search?num=100&n ... redhat.com

ajtucker
Posts: 1
Joined: 2017/10/12 09:55:05

Re: fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by ajtucker » 2017/10/12 10:46:32

I've had the same problem: fail2ban with firewalld and ipset stopped working recently. I've used some of the workarounds mentioned, but they're slow to run on my firewall and miss the efficiency of ipset.

Looking around, firewalld has had support for ipset for some time, and this is included in CentOS. My current setup is based on an example from the firewalld blog. I've created a new action file under /etc/fail2ban/action.d/firewallcmd-ipset-new.conf containing the following:

Code: Select all

[INCLUDES]

before =

[Definition]

actionstart = firewall-cmd --permanent --new-ipset=fail2ban-<name> --type=hash:ip --option=timeout=<bantime>
              firewall-cmd --reload

actionstop = firewall-cmd --permanent --delete-ipset=fail2ban-<name>
             firewall-cmd --reload
             ipset flush fail2ban-<name>
             ipset destroy fail2ban-<name>

actionban = ipset add fail2ban-<name> <ip> timeout <bantime> -exist

actionunban = ipset del fail2ban-<name> <ip> -exist

[Init]
Then in /etc/fail2ban/jail.d/00-firewalld.conf I changed the banaction to read banaction = firewall-cmd-ipset-new.

Finally, firewalld needs to be told what to do with incoming connections matching this new ipset. I should figure out the incantation and put it in the action configuration file, but for now I just use the GUI firewall-config to set the zone for the source fail2ban-sshd to be "drop".

GlubGcams
Posts: 1
Joined: 2018/01/12 20:57:49

Re: fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by GlubGcams » 2018/01/12 21:24:44

I came across this thread with exactly the same issue as the OP, and I also read through the thread over on Github.

I have just managed to get fail2ban working with the original firewallcmd-ipset action script.

What I discovered is that, like the OP, issuing firewall-cmd --direct --get-all-rules showed the fail2ban rules present. However when I looked at the actual iptables ruleset (i.e. iptables --list -vn), I found no such rules present in the active iptables chains. Even performing a reload of the firewall did not bring the actual rules back into the iptables chains.

I then discovered that the direct rules firewall-cmd was showing as present, had been stored "permanently" in the file:
/etc/firewalld/direct.xml

What I think has happened is that when I had fail2ban running, at some point I have issued a firewall-cmd --runtime-to-permanent command, which has then resulted in the fail2ban rules being added to the above mentioned file, and considered "permanent". For reasons unknown, firewall-cmd is not adding these back into the active iptables ruleset, and I think this is where the root cause lies.

Now to the fix..

By removing the rules from the file above (I simply edited the xml file and deleted the lines between the <direct></direct> tags), reloading the firewall (firewall-cmd --reload) and then restarting fail2ban, my fail2ban firewall rules re-appeared in the actual iptables chains and all is working and well again.

I'll need to be sure to stop fail2ban before doing a "firewall-cmd --runtime-to-permanent" again, otherwise I think the problem will re-appear, when the fail2ban jails get permanently added into the firewallcmd direct ruleset.

I hope the above helps somebody else!

larwood
Posts: 66
Joined: 2011/07/27 12:07:30
Location: Perth WA, Australia

Re: fail2ban with firewallcmd-ipset does not work on 7.3 1611

Post by larwood » 2019/10/23 05:08:50

My search results directed me here so I thought this post was worth updating in the event it may assist others.

A few days ago I updated my server from CentOS 7.6.1810 to 7.7.1908. During some post checks I found that fail2ban wasn't effectively blocking IPs and these are some of the relevant errors in the /var/log/fail2ban.log:

Code: Select all

# cat /var/log/fail2ban.log | grep -i 'failed to start jail'
2019-10-23 03:01:24,143 fail2ban.actions        [5014]: ERROR   Failed to start jail 'sshd' action 'firewallcmd-ipset': Error starting action

2019-10-23 03:01:11,257 fail2ban.action         [1164]: ERROR   ipset del fail2ban-sshd 41.70.23.240 -exist -- stdout: ''
2019-10-23 03:01:11,258 fail2ban.action         [1164]: ERROR   ipset del fail2ban-sshd 41.70.23.240 -exist -- stderr: 'ipset v7.1: The set with the given name does not exist\n'
2019-10-23 03:01:11,258 fail2ban.action         [1164]: ERROR   ipset del fail2ban-sshd 41.70.23.240 -exist -- returned 1
2019-10-23 03:01:11,259 fail2ban.actions        [1164]: ERROR   Failed to execute unban jail 'sshd' action 'firewallcmd-ipset' info '{'matches':...
Archived fail2ban logs indicate the errors starting about 10 minutes after the ipset update from v6.38 to v7.1 during my CentOS update.

For me the core issue is I had configured a bantime value of 2419200 (28 days) but this value is no longer valid with ipset v7.1.

When fail2ban starts it creates the ipset using /etc/fail2ban/action.d/firewallcmd-ipset.conf. When I ran the command manually the issue was revealed.

Code: Select all

# ipset create fail2ban-sshd hash:ip timeout 2419200
ipset v7.1: Syntax error: '2419200' is out of range 0-2147483
I reduced the bantime value to 2073600 (24 days) and restarted the fail2ban.service and all good.

Post Reply