Static routes for default are ignored on reboot.

[voluhar]

I want to establish configuration where my network will looks like this:
public internet <router> LAN <CentOS8 VM with local IP and routed public IP>
Router is OPNsense in HA with CARP IP: 172.16.128.1/22 (.2 for master .3 for slave).
CentOS8 VM LAN IP 172.16.128.10.
On router I added route 172.31.4.101/32 -> 172.16.128.10 and on CentOS 8 VM I added "ip route add default via 172.16.128.1 dev ens18 src 172.31.4.101"
I was able to SSH to 172.31.4.101 and also if I ping from CentOS 8, on router I see that packets are going from correct IP (172.31.4.101)

I am testing with my computer on WAN side of this router and verifying with packet capture on router WAN interface.
To here everything is exactly as I want to be, but I am unable to configure it to work after reboot.
More than I read more confused I am.

I tried to add one test route with nmtui and it makes file /etc/sysconfig/network-scripts/route-ens18 with following content:

Code: Select all

ADDRESS0=1.1.1.1
NETMASK0=255.255.255.255
GATEWAY0=172.16.128.1

This is ok for reboot, route is listed in ip route list
It is also ok if I delete all and insert

Code: Select all

1.1.1.1/32 via 172.16.128.1

But if I insert any of lines below, this is not seen in ip route list after reboot.

Code: Select all

default via 172.16.128.1
0.0.0.0/0 via 172.16.128.1
default via 172.17.128.1 src 172.31.4.101 

The workaround is:

Code: Select all

0.0.0.0/1 via 172.16.128.1 src 172.31.4.101
128.0.0.0/1 via 172.16.128.1 src 172.31.4.101

Please let me know what I am doing wrong that "default via 172.16.128.1 src 172.31.4.101" is not working.

And also config file of interface:

Code: Select all

[root@CentOS8-DMZ-test01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens18
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens18
UUID=6a549f1b-26dd-46d0-9c12-2c3a7902404e

IPADDR=172.16.128.10
PREFIX=22
#GATEWAY="172.16.128.1"
DNS1=172.16.128.2
DNS2=172.16.128.3

IPADDR1=172.31.4.101
PREFIX1=32

DEVICE=ens18
ONBOOT=yes

[hunter86_bg]

By default CentOS8 uses NetworkManager!
Check configuration there:

Code: Select all

 nmcli con show
 nmcli con show <network from previous command> 

[voluhar]

If following will work than it will be ok:

Code: Select all

nmcli> set ipv4.routes 0.0.0.0/0 172.16.128.1 src=172.31.4.96
Error: failed to set 'routes' property: invalid prefix '0'; <1-32> allowed
nmcli>

So same problem as with route in file.

I really hate that so simple thing must be so complicated. So please tell me how can I set NetworkManager to do this:

Code: Select all

ip route add default via 172.16.128.1 src 172.31.4.101

Is it even possible to do it with NetworkManager ?

As I am looking in nmtui there are options for static routes, but none with src address.
But with nmcli I get error above. If I set gateway in file I do not know, is there option to specify SRC address in it?

[TrevorH]

Do you have NetworkManager-dispatcher-routing-rules.noarch installed?

[voluhar]

Code: Select all

[root@CentOS8-DMZ-test01 ~]# yum list installed NetworkManager
Installed Packages
NetworkManager.x86_64                                                                            1:1.20.0-3.el8                                                                             @anaconda
[root@CentOS8-DMZ-test01 ~]#

I assume not.
I installed NetworkManager-dispatcher-routing-rules.noarch and as I tested it still do not take default or 0.0.0.0/0 from /etc/sysconfig/network-scripts/route-ens18 after reboot.
Also:

Code: Select all

nmcli> set ipv4.routes
Enter 'routes' value: 0.0.0.0/0 172.16.128.1 src=172.31.4.96
Error: failed to set 'routes' property: invalid prefix '0'; <1-32> allowed
nmcli>

And "ip route add default via 172.16.128.1 src 172.31.4.101" is not policy based routing or I am wrong ?
As I read this package is that /etc/sysconfig/network-scripts/route-ens18 should work but as I see it work as same as it is installed or not. In both cases 0.0.0.0/1 and 128.0.0.0/1 work but route for /0 does not work.
Should I test anything else or provide any additional informations ?

[jlehtone]

I want to establish configuration where my network will looks like this:
public internet <router> LAN <CentOS8 VM with local IP and routed public IP>
Router is OPNsense in HA with CARP IP: 172.16.128.1/22 (.2 for master .3 for slave).
CentOS8 VM LAN IP 172.16.128.10.
On router I added route 172.31.4.101/32 -> 172.16.128.10 and
on CentOS 8 VM I added "ip route add default via 172.16.128.1 dev ens18 src 172.31.4.101"

I did stop reading at this point.

The router has at least public IP 172.31.4.101
The router does SNAT (aka masquerade) for all outgoing (from 172.16.128.0/22 to WAN) traffic
The router does DNAT (to 172.16.128.10) for traffic that arrives to 172.31.4.101

All the VM needs, is the 172.16.128.10. Its default route is via 172.16.128.1 by default.

If the router has only one public IP, then everything is fine with the VM but everyone else on the LAN is not.

If the router has two public IP's, one for LAN (masquerade) and another for the VM, then the router
must route so that traffic from 172.16.128.10 must be send out with src 172.31.4.101.


In other words, you have an OPNsense question.

[voluhar]

The router has at least public IP 172.31.4.101
The router does SNAT (aka masquerade) for all outgoing (from 172.16.128.0/22 to WAN) traffic
The router does DNAT (to 172.16.128.10) for traffic that arrives to 172.31.4.101

Well I did stop read at this point as it is totally clear to me what router do, if I will have OPNsense question I will ask on their forum, and if you will read further you will see that I verify configuration with packet capture.

I want to have private network between router and VMs and public IPs routed to VMs with /32 route on my router and VM LAN IP as gateway no NAT!.
IPs are correctly set on virtual machines (CentOS).
But problem is that if I set gateway in config file "ifcfg-ens18" than ping goes out by VMs LAN address instead of its public /32 address.
To achieve that default source IP is publi /32 than "ip route add default via GATEWAY src PUBLIC_IP" must be run and everything works perfectly. No NAT on router!

But as I described it is not problem that my configuration will not work (if you will read further, you will see that I verify configuration with packet pacture). But problem is that it is not possible to set it on CentOS 8 without workaround.
Let me explain again, workaround in /etc/sysconfig/network-scripts/route-ens18 is

Code: Select all

0.0.0.0/1 via 172.16.128.1 src 172.31.4.101
128.0.0.0/1 via 172.16.128.1 src 172.31.4.101

As I already mentioned "0.0.0.0/0" or "default" does not work.

And I only wan't to know what is correct CentOS 8 way to make permanent configuration like "ip route add default via DEF_GW src SRC_IP".
Nothing special, no routing by source network, just simple ip route command transferred to permanent configuration.

[jlehtone]

Perhaps I was too hasty.

How about this:
* A (CentOS) PC is connected to two networks, lanA and lanB
* Both networks do have a gateway out, but we want the PC to use lanB's gateway

In other words,
* if PC gets config from DHCP for lanA, then it should ignore offered gateway
* if PC does not get config for lanB from DHCP, then it should explicitly set gateway for that connection

To achieve those, update existing connections:

Code: Select all

nmcli con mod lanA-connection ipv4.never-default TRUE
nmcli con mod lanB-connection ipv4.gateway 172.31.4.???

Alas, you don't seem to have two connections nor two networks. If you had, then the router would have a (public) IP in the same subnet as the 172.31.4.101 (of the VM). That IP would be the gateway for that subnet.

If VM is a member of a subnet with public address 172.31.4.101, then there is no need in the router to route "172.31.4.101/32 -> 172.16.128.10". Instead, it will simply forward traffic for 172.31.4.101 out of its interface that is in the same subnet as the VM.


Either there is something that I still miss or your network topology makes no sense at all.

[voluhar]

I appreciate your concerns about configuration of my network, but I gather pros and cons and this is my final configuration.
Only one network interface per VM and routed IPs.
Why? Because I want traffic to return to router/firewall for each public IP. I do not want traffic to go between VMs with public IP directly because in that way I can not firewall it. Ok, yes if firewall is set on hypervisor it will make its work but than firewall must be set on two places and much more space for errors. Also physical machines are not included in that case.
But this is offtopic.
I also appreciate how to set up this configuration that you purpose, thank you, but I did not ask for this king of solution.
And second more important reason is IP waste. If I get /29 network than I can use 3 IPs (1 master router, 1 slave router, 1 CARP interface of routers in HA, 3 for further allocation to VMs, 1 network address and 1 broadcast address).
If I use my routed approach I can use 8 IPs -> tested. Why? Router does not care if it is network or broadcast address, it route it further and does not complain. Also on computer there is no problem because it is /32 address and not .0 address in some other network.
Offtopic again...

Unanswered question is still how to make "ip route add default via 172.16.128.1 src 172.31.4.101" permanent.

I also make some chart that visualisation will be easier. And please note, this is not home router that is capable of doing only NAT.

EDIT: Typo in image.

[jlehtone]

Good picture. This is what I would do:
Create multiple VLANs; one for "LAN" and one for each VM. That keeps the VMs isolated from each other and the LAN.
The requirement is that both the router and hosts support it.

For example:

Code: Select all

LAN: vlanID 128 net 172.16.128.0/24 gw 172.16.128.[123]
VM1: vlanID 129 net 10.16.129.0/24 gw 10.16.129.[123] VM 10.16.129.11
VM2: vlanID 130 net 10.16.130.0/24 gw 10.16.130.[123] VM 10.16.130.17

All the NAT and firewalling will occur at the OPNsense.
You have to set up the VLANs on the hypervisors, because you don't want the VMs to know that their "untagged" traffic is passed through shared cable.

I understand that the public /29 is precious, but there are plenty of private address space for the hidden (V)LANs.

Would this be feasible with your hypervisors?