NetworkManager: Policy based routing, SRC doesn't get picked up

[obakefilter]

I'm trying to create an IP rule/route list on a CentOS 7 machine using NetworkManager. I have 3 VLANS on the machine, one of them has 1 IP, the second has 2 and the third has three. I set an source IP based policy that looks like the following:

Interfaces:

Code: Select all

2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.168.10.231/24 brd 10.168.10.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
3: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet Y.Y.Y.242/27 brd Y.Y.Y.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet Y.Y.Y.243/27 brd Y.Y.Y.255 scope global secondary noprefixroute ens192
       valid_lft forever preferred_lft forever
4: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet x.x.x.72/27 brd x.x.x.95 scope global noprefixroute ens224
       valid_lft forever preferred_lft forever
    inet x.x.x.73/27 brd x.x.x.95 scope global secondary noprefixroute ens224
       valid_lft forever preferred_lft forever
    inet x.x.x.88/27 brd x.x.x.95 scope global secondary noprefixroute ens224
       valid_lft forever preferred_lft forever

rule-DMZ:

Code: Select all

from 172.18.0.20 lookup kamin
from 172.18.0.30 lookup kameg
from 172.18.0.40 lookup freeswitch
from 172.18.0.50 lookup rtpengine
from 172.18.0.60 lookup turn
from 172.18.0.70 lookup uploader
from 172.18.0.80 lookup voipmonitor

route-DMZ:

Code: Select all

default via X.X.X.94 src X.X.X.72 dev ens224 table kamin
10.168.10.0/24 dev ens160 table kamin
192.168.1.0/24 via 10.168.10.254 dev ens160 table kamin
192.168.2.0/24 via 10.168.10.254 dev ens160 table kamin
192.168.20.0/24 via 10.168.10.254 dev ens160 table kamin
192.168.110.0/24 via 10.168.10.254 dev ens160 table kamin
default via X.X.X.94 src X.X.X.88 dev ens224 table kameg
192.168.1.0/24 via 10.168.10.254 dev ens160 table kameg
192.168.2.0/24 via 10.168.10.254 dev ens160 table kameg
192.168.20.0/24 via 10.168.10.254 dev ens160 table kameg
192.168.110.0/24 via 10.168.10.254 dev ens160 table kameg
default via Y.Y.Y.225 src Y.Y.Y.242 table freeswitch
192.168.1.0/24 dev ens160 table freeswitch
192.168.110.0/24 dev ens160 table freeswitch
default via X.X.X.94 src X.X.X.73 dev ens224 table rtpengine
192.168.2.0/24 via 10.168.10.254 dev ens160 table rtpengine
192.168.20.0/24 via 10.168.10.254 dev ens160 table rtpengine
default via Y.Y.Y.225 src Y.Y.Y.243 table turn
192.168.1.0/24 dev ens160 table turn
192.168.110.0/24 dev ens160 table turn
default via 10.168.10.254 table uploader
default via 10.168.10.254 table voipmonitor

Although I've installed the NetworkManager-config-routing-rules package and enabled it, and even though the rules do appear in the correct form when running the ip rule show and ip route list commands, the Kernel seems to ignore the src addresses set for the rules - and uses the default address for each interface. For example, running curl checkip.amazonaws.com will show the exact same external IP (x.x.x.72) for when I come from 172.18.0.20, or 172.18.0.30 or 172.18.0.50 (I am running a Docker engine on this machine and every docker has a different static IP, I use the internal shells to run the CURLs).

How can I make sure that the external IP does get picked up correctly with this setup?

[jlehtone]

I don't think that it works like that.

A new packet is sent from your machine*. Routing states that it should leave through interface 'ens224'.
Since it leaves through ens224 and it is a new packet, it needs source address. Lets use ens224's address.
How could you route based on address that is decided after the routing decision?

Replies are a different story. The reply is an answer to something that had destination. It is logical to use
that destination as source address in the reply. Then you can route according to the source.

The SELECTOR in rule can be more than just "from x". See man ip-rule


* Assuming that "Docker" is just a process in the host. If the host truly does routing between container subnet and outside subnet, then the host can do SNAT to hide the private address of the container.

[obakefilter]

I don't think that it works like that.

A new packet is sent from your machine*. Routing states that it should leave through interface 'ens224'.
Since it leaves through ens224 and it is a new packet, it needs source address. Lets use ens224's address.
How could you route based on address that is decided after the routing decision?

Replies are a different story. The reply is an answer to something that had destination. It is logical to use
that destination as source address in the reply. Then you can route according to the source.

The SELECTOR in rule can be more than just "from x". See man ip-rule


* Assuming that "Docker" is just a process in the host. If the host truly does routing between container subnet and outside subnet, then the host can do SNAT to hide the private address of the container.

The routing is in fact working well, meaning, when I send a packet from the container that has the static IP of 172.18.0.20 it follows the routing table of kamin, and goes out through the right interface. Problem is that the source address chosen is ALWAYS the default address of this interface (may it be ens224 or ens160), I want to be able to manipulate this address using the different IPs on each interface.

Is there a better way to acheieve that?

[obakefilter]

Essentialy I would like each docker to have a different WAN address.

[jlehtone]

Well, if the packets truly have identifiable source addresses before leave an interface, then you must ask:
What does change 172.18.0.20 into x.y.z.w?

That does not change automatically. It must be NAT.
While routing can do static NAT, I presume that your netfilter does MASQUERADE (a for of SNAT).
Easy to check. What have you in:

Code: Select all

sudo iptables -t nat -S

[obakefilter]

Well, if the packets truly have identifiable source addresses before leave an interface, then you must ask:
What does change 172.18.0.20 into x.y.z.w?

That does not change automatically. It must be NAT.
While routing can do static NAT, I presume that your netfilter does MASQUERADE (a for of SNAT).
Easy to check. What have you in:

Code: Select all

sudo iptables -t nat -S

Here's what I'm seeing -

Code: Select all

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N OUTPUT_direct
-N POSTROUTING_ZONES
-N POSTROUTING_ZONES_SOURCE
-N POSTROUTING_direct
-N POST_public
-N POST_public_allow
-N POST_public_deny
-N POST_public_log
-N PREROUTING_ZONES
-N PREROUTING_ZONES_SOURCE
-N PREROUTING_direct
-N PRE_public
-N PRE_public_allow
-N PRE_public_deny
-N PRE_public_log
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -j OUTPUT_direct
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.27.0.0/16 ! -o br-c61b504e2e91 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o wrtcpriv -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING -s 172.18.0.20/32 -d 172.18.0.20/32 -p udp -m udp --dport 5099 -j MASQUERADE
-A POSTROUTING -s 172.18.0.20/32 -d 172.18.0.20/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A POSTROUTING -s 172.18.0.20/32 -d 172.18.0.20/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.80/32 -d 172.18.0.80/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.18.0.60/32 -d 172.18.0.60/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.60/32 -d 172.18.0.60/32 -p udp -m udp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.60/32 -d 172.18.0.60/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.18.0.60/32 -d 172.18.0.60/32 -p udp -m udp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.18.0.30/32 -d 172.18.0.30/32 -p udp -m udp --dport 5060 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-c61b504e2e91 -j RETURN
-A DOCKER -i wrtcpriv -j RETURN
-A DOCKER -d X.X.X.72/32 ! -i wrtcpriv -p udp -m udp --dport 5099 -j DNAT --to-destination 172.18.0.20:5099
-A DOCKER -d 10.168.10.231/32 ! -i wrtcpriv -p tcp -m tcp --dport 3000 -j DNAT --to-destination 172.18.0.20:3000
-A DOCKER -d X.X.X.72/32 ! -i wrtcpriv -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.20:443
-A DOCKER -d 10.168.10.231/32 ! -i wrtcpriv -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.80:80
-A DOCKER -d Y.Y.Y.243/32 ! -i wrtcpriv -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.60:443
-A DOCKER -d Y.Y.Y.243/32 ! -i wrtcpriv -p udp -m udp --dport 443 -j DNAT --to-destination 172.18.0.60:443
-A DOCKER -d Y.Y.Y.243/32 ! -i wrtcpriv -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.60:80
-A DOCKER -d Y.Y.Y.243/32 ! -i wrtcpriv -p udp -m udp --dport 80 -j DNAT --to-destination 172.18.0.60:80
-A DOCKER -d X.X.X.88/32 ! -i wrtcpriv -p udp -m udp --dport 5060 -j DNAT --to-destination 172.18.0.30:5060
-A POSTROUTING_ZONES -o ens160 -g POST_public
-A POSTROUTING_ZONES -o ens256 -g POST_public
-A POSTROUTING_ZONES -o ens161 -g POST_public
-A POSTROUTING_ZONES -o ens224 -g POST_public
-A POSTROUTING_ZONES -o ens193 -g POST_public
-A POSTROUTING_ZONES -o ens192 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens160 -g PRE_public
-A PREROUTING_ZONES -i ens256 -g PRE_public
-A PREROUTING_ZONES -i ens161 -g PRE_public
-A PREROUTING_ZONES -i ens224 -g PRE_public
-A PREROUTING_ZONES -i ens193 -g PRE_public
-A PREROUTING_ZONES -i ens192 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
-A PRE_public_allow -p udp -m mark --mark 0x64 -j DNAT --to-destination 172.18.0.50
-A PRE_public_allow -p udp -m mark --mark 0x65 -j DNAT --to-destination 172.18.0.40

[obakefilter]

I've added the following iptables rules, but it didn't make any difference -

Code: Select all

iptables -t nat -A POSTROUTING -o ens224 -j SNAT -s 172.18.0.30 --to X.X.X.88
iptables -t nat -A POSTROUTING -o ens224 -j SNAT -s 172.18.0.50 --to X.X.X.73
iptables -t nat -A POSTROUTING -o ens192 -j SNAT -s 172.18.0.60 --to Y.Y.Y.243

[jlehtone]

If you now print your active POSTROUTING with

Code: Select all

sudo iptables -t nat -S POSTROUTING

you will probably see your rules after the

Code: Select all

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.27.0.0/16 ! -o br-c61b504e2e91 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o wrtcpriv -j MASQUERADE

Overall, your netfilter rules seem to be managed by firewalld.service (which is the default).
If firewalld manages rules, then we should tell firewalld to add rules, rather than do it "manually" (but manual can be used for quick test).

However, most of the rules in POSTROUTING do not look like they would come from firewalld.
I suspect that Docker injects its own rules and thus overrides/messes up firewalld's schemes.
Then it is the Docker that should insert the rules that we want when it starts each container.

We have actually two questions:
1. Is it possible to have such netfilter ruleset that does what you want?
If yes,
2. How to get the services to create that ruleset on boot?

For the first, you must at least insert (-I) your SNAT rules before the catch-all masquerade rules.


You can see statistics (how many packets have matched each rule) and rule numbers with:

Code: Select all

sudo iptables -t nat --lin -vnL POSTROUTING

[obakefilter]

If you now print your active POSTROUTING with

Code: Select all

sudo iptables -t nat -S POSTROUTING

you will probably see your rules after the

Code: Select all

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.27.0.0/16 ! -o br-c61b504e2e91 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o wrtcpriv -j MASQUERADE

Overall, your netfilter rules seem to be managed by firewalld.service (which is the default).
If firewalld manages rules, then we should tell firewalld to add rules, rather than do it "manually" (but manual can be used for quick test).

However, most of the rules in POSTROUTING do not look like they would come from firewalld.
I suspect that Docker injects its own rules and thus overrides/messes up firewalld's schemes.
Then it is the Docker that should insert the rules that we want when it starts each container.

We have actually two questions:
1. Is it possible to have such netfilter ruleset that does what you want?
If yes,
2. How to get the services to create that ruleset on boot?

For the first, you must at least insert (-I) your SNAT rules before the catch-all masquerade rules.


You can see statistics (how many packets have matched each rule) and rule numbers with:

Code: Select all

sudo iptables -t nat --lin -vnL POSTROUTING

Thank you, for future reference:
I disabled the automatic MASQUERADING from Docker by setting com.docker.network.bridge.enable_ip_masquerade: 'false'.
And then, added the following 'direct' firewall rules on Firewalld:

Code: Select all

ipv4 nat POSTROUTING 0 -o ens224 -j SNAT -s 172.18.0.30 --to-source X.X.X.88
ipv4 nat POSTROUTING 0 -o ens224 -j SNAT -s 172.18.0.50 --to-source X.X.X.73
ipv4 nat POSTROUTING 0 -o ens192 -j SNAT -s 172.18.0.60 --to-source Y.Y.Y.243
ipv4 nat POSTROUTING 0 -s 172.18.0.0/16 ! -o wrtcpriv -j MASQUERADE