Page 1 of 1

Virbr0 problems

Posted: 2020/04/13 11:25:41
by VictoriaV
Good afternoon to everyone,

First of all, I'm new to virtualization, so I may be asking a very basic question, but I've been with it for two weeks now and can't find an answer that answer my problem.
I apologize if this is already answered in any other topic, I have not found it :(

My scenario is as follows, I have a virtual machine created with kvm running at centos 7. When I created it, I generated the virbr0 interface

# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.525400992295 yes virbr0-nic
vnet0

I am trying to send traffic from 192.168.122.173 to 10.0.3.12, and return it (I use NAT for it)

[Scenario.png file]

Traffic goes through the observed vlan (361). But I think virbr0 has an implicit NAT that complicates his return...

I see the traffic go in and out on the vlan:

[Centos7]# tcpdump -vvv -n -i ens160.361
tcpdump: listening on ens160.361, link-type EN10MB (Ethernet), capture size 262144 bytes
14:16:32.392248 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
172.21.0.241.38596 > 10.0.3.12.cadlock2: [udp sum ok] UDP, length 1472
14:16:32.392540 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
172.21.0.241.38596 > 192.168.122.173.cadlock2: [udp sum ok] UDP, length 1472

But I only see the traffic Machine 1 - Router A and not Router A - Machine 1 in the virbr0 interface:

[Centos7]# tcpdump -vvv -n -i virbr0
tcpdump: listening on virbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:17:39.358478 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
192.168.122.173.cadlock2 > 10.0.3.12.cadlock2: [udp sum ok] UDP, length 1472
14:17:39.858313 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1500)
192.168.122.173.cadlock2 > 10.0.3.12.cadlock2: [udp sum ok] UDP, length 1472

[Centos7]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
[...]
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0



Also I have not seen that the virbr0 knows the MAC of the Machine 1...This surprised me because the machine does ping the router (and the router ping the machine)

It is NAT problem by default in virbr0 ? What could be happening?

I have tried IPTables and it has not worked...I have no more ideas to fix it :( :( :(

Regards,

Victoria

Re: Virbr0 problems

Posted: 2020/04/13 16:32:19
by jlehtone
VictoriaV wrote:
2020/04/13 11:25:41
When I created it, I generated the virbr0 interface
Libvirt, when installed, creates one virtual network named "default" that autostarts. It shows as a bridge in the host, with name "virbr0", provides DHCP and DNS to VM's connected to it, has the subnet 192.168.122.0/24, allows routing through the host, and has sNAT.

If the host masquerades (sNAT) the traffic from the VM's, then everything from VMs appears to originate from 172.21.0.241.

The router apparently knows two networks: 172.21.0.x/y and 10.0.3.z/w. Traffic from those subnets it can reply directly. For other destinations it must pass packets to the next router, its "default route", which is probably somewhere in 10.0.3.z/w.
If the host does not sNAT, then router will receive traffic from 192.168.122.173 and how would it know that replies have to be sent to "router 172.21.0.241"?

I'd say that NAT is necessary, unless the router's config gets static route to 192.168.122.0/24.


CentOS 7's firewall is by default configured via firewalld. The Libvirtd, when it starts virtual networks, does add rules too. The (iptables) rules contain both filter and nat; what to let in and pass through, and what to NAT.

Re: Virbr0 problems

Posted: 2020/04/13 17:28:56
by VictoriaV
Hello jlehtone

Thnx for your answer :-)

In the router A I have a NAT rule so that all traffic arriving at 10.0.3.12 goes to 192.168.122.173 (Machine 1).

So when I send a packet from 192.168.122.173 to 10.0.3.12, it goes back to 192.168.122.173 with that router NAT rule.


I have seen that if I ping another address of the router, if I see the ICMP on the virbr0 interface (in addition to ens160.361).

Could it be a problem that the IP addresses of the machine and the interface that joins out with a different IP range are? I mean..192.168.X.X and 172.21.X.X.

Regards,

Victoria

Re: Virbr0 problems

Posted: 2020/04/14 18:23:06
by jlehtone
"Source 10.0.3.12" or "Destination 10.0.3.12"?

sNAT, aka Masquerade, changes the source address of an outgoing package (after routing) to hide the real source.
dNAT, aka port forwarding, changes the destination address of incoming package (before routing) to redirect it into different destination.


Who needs to connect to 192.168.122.173?

Re: Virbr0 problems

Posted: 2020/04/15 07:12:56
by VictoriaV
Hello!

Traffic will follow this flow:
Machine 1 will send a packet to 10.0.3.12, once it reaches the router, it has a Nat so that all packets going to 10.0.3.12 are changed to 192.168.122.173, so the packets will return to server with source 10.0.3.12 and destination 192.168.122.173.

From what I understand, once the packet returns from the router to Machine 1 with the IP source 10.0.3.12 and destination 192.168.122.173, when it reaches the virbr0 interface, will it change the source?If so ... Is it possible to disable it?


Regards,

Victoria

Re: Virbr0 problems

Posted: 2020/04/15 08:19:48
by jlehtone
Lets clarify. "Machine 1" (M1) is the VM? You want the M1 to talk with itself, without knowing that it does?

Lets track traffic, assuming the host does masquerade and everything else is "correct":

Code: Select all

1. Client process in M1 sends a packet P, from=192.168.122.173 to=10.0.3.12
2. Host routes P to go via 172.21.0.240
3. Host routes P to go out from link ens160.361
4. Host sNAT P from=172.21.0.241 to=10.0.3.12
5. Router A (RA) receives P
6. RA dNAT P from=172.21.0.241 to=192.168.122.173
7. RA has route "to 192.168.122.0/24 via 172.21.0.241"
8. RA routes P via 172.21.0.241
9. RA sNAT P from=10.0.3.12 to=192.168.122.173
10. Host routes P to 192.168.122.173 via the virtual subnet
11. M1 receives P from=10.0.3.12 to=192.168.122.173

12. Server process in M1 creates a reply R  from=192.168.122.173 to=10.0.3.12
13. Host routes R to go via 172.21.0.240
14. Host routes R to go out from link ens160.361
15. Host sNAT R from=172.21.0.241 to=10.0.3.12
16. Router A (RA) receives R
17. RA dNAT R from=172.21.0.241 to=192.168.122.173
18. RA routes R via 172.21.0.241
19. RA sNAT R from=10.0.3.12 to=192.168.122.173
20. Host routes R to 192.168.122.173 via the virtual subnet
21. M1 receives R from=10.0.3.12 to=192.168.122.173
22. Client process in M1 receives reply (from server process in M1)
Is that how you have set things up?
Is that what you want to happen?
Why M1 has to talk with M1 via RA?
What happens, if host talks with RA?
What happens, if somebody else in 172.21.0.x/y talks with RA?

Re: Virbr0 problems

Posted: 2020/04/21 12:22:25
by VictoriaV
Hello jlehtone,


Sorry for the delay and reply, and thnx for your answer.
I tried to simplicy the scenario, not having NAT on the router. But I have encountered a similar problem, although it seems to be more limited.

Mi scenario is :

Guess (Machine1) with eth1 172.21.0.244 ---- Host with data1 172.21.0.243 ----eno33559296.361 no IP --------------- Cisco router with Giga2.361 172.21.0.242 and Loopback3 10.0.3.12

When I ping from 172.21.0.244 to 172.21.0.242, i dont get reply. Anyway I can see de ARP in eno33559296.361 and in vnet1 (the same) :

[root@localhost wuivvi01]# tcpdump -vvv -n -i eno33559296.361 -e arp

tcpdump: WARNING: eno33559296.361: no IPv4 address assigned
tcpdump: listening on eno33559296.361, link-type EN10MB (Ethernet), capture size 65535 bytes
13:36:32.006870 52:54:00:1f:14:e5 > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 172.21.0.242 tell 172.21.0.244, length 28
13:36:33.063307 52:54:00:1f:14:e5 > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 172.21.0.242 tell 172.21.0.244, length 28

Cisco ARP Answer:
Apr 21 11:37:14: IP ARP: rcvd req src 172.21.0.244 5254.001f.14e5, dst 172.21.0.242 GigabitEthernet2.361 tableid 0
Apr 21 11:37:14: IP ARP: sent rep src 172.21.0.242 0050.5693.662e,
dst 172.21.0.244 5254.001f.14e5 GigabitEthernet2.361

But the answer is never seen on the host :-(


More info:

[root@localhost wuivvi01]# brctl show
bridge name bridge id STP enabled interfaces
data1 8000.005056934942 no eno33559296.361
vnet1
virbr0 8000.5254004dba99 yes virbr0-nic
vnet0



[root@localhost wuivvi01]# ifconfig
data1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.21.0.243 netmask 255.255.255.248 broadcast 172.21.0.247
inet6 fe80::68:23ff:fe72:5219 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:93:49:42 txqueuelen 0 (Ethernet)
RX packets 4943 bytes 318292 (310.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4194 bytes 408858 (399.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno16780032: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.X.X netmask 255.255.252.0 broadcast 192.168.143.255
inet6 fe80::250:56ff:fe93:36cd prefixlen 64 scopeid 0x20<link>
ether 00:50:56:93:36:cd txqueuelen 1000 (Ethernet)
RX packets 1189952 bytes 94574358 (90.1 MiB)
RX errors 0 dropped 1858 overruns 0 frame 0
TX packets 35708 bytes 7350058 (7.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno33559296: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:50:56:93:49:42 txqueuelen 1000 (Ethernet)
RX packets 1287364 bytes 83511497 (79.6 MiB)
RX errors 0 dropped 1086446 overruns 0 frame 0
TX packets 4307 bytes 363941 (355.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno50338560: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:50:56:93:f2:70 txqueuelen 1000 (Ethernet)
RX packets 4278 bytes 1413708 (1.3 MiB)
RX errors 0 dropped 15 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eno33559296.361: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::250:56ff:fe93:4942 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:93:49:42 txqueuelen 0 (Ethernet)
RX packets 4943 bytes 318292 (310.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4194 bytes 408858 (399.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens161: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.140.211 netmask 255.255.252.0 broadcast 192.168.143.255
inet6 fe80::250:56ff:fe93:c28f prefixlen 64 scopeid 0x20<link>
ether 00:50:56:93:c2:8f txqueuelen 1000 (Ethernet)
RX packets 1134387 bytes 85902694 (81.9 MiB)
RX errors 0 dropped 3 overruns 0 frame 0
TX packets 62281 bytes 6936362 (6.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 1550 bytes 118124 (115.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1550 bytes 118124 (115.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:4d:ba:99 txqueuelen 0 (Ethernet)
RX packets 795379 bytes 51053016 (48.6 MiB)
RX errors 0 dropped 1557 overruns 0 frame 0
TX packets 46215 bytes 7283454 (6.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:54:00:3e:21:e3 txqueuelen 500 (Ethernet)
RX packets 16261 bytes 2453996 (2.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 844831 bytes 67242806 (64.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fc54:ff:fe1f:14e5 prefixlen 64 scopeid 0x20<link>
ether fe:54:00:1f:14:e5 txqueuelen 500 (Ethernet)
RX packets 2589 bytes 176822 (172.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2203 bytes 186751 (182.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0




[root@localhost wuivvi01]# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
PREROUTING_direct all -- anywhere anywhere
PREROUTING_ZONES_SOURCE all -- anywhere anywhere
PREROUTING_ZONES all -- anywhere anywhere

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
OUTPUT_direct all -- anywhere anywhere

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
RETURN all -- 192.168.122.0/24 base-address.mcast.net/24
RETURN all -- 192.168.122.0/24 255.255.255.255
MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
POSTROUTING_direct all -- anywhere anywhere
POSTROUTING_ZONES_SOURCE all -- anywhere anywhere
POSTROUTING_ZONES all -- anywhere anywhere

Chain OUTPUT_direct (1 references)
target prot opt source destination

Chain POSTROUTING_ZONES (1 references)
target prot opt source destination
POST_public all -- anywhere anywhere [goto]
POST_public all -- anywhere anywhere [goto]
POST_public all -- anywhere anywhere [goto]
POST_public all -- anywhere anywhere [goto]
POST_public all -- anywhere anywhere [goto]
POST_public all -- anywhere anywhere [goto]

Chain POSTROUTING_ZONES_SOURCE (1 references)
target prot opt source destination

Chain POSTROUTING_direct (1 references)
target prot opt source destination

Chain POST_public (6 references)
target prot opt source destination
POST_public_log all -- anywhere anywhere
POST_public_deny all -- anywhere anywhere
POST_public_allow all -- anywhere anywhere

Chain POST_public_allow (1 references)
target prot opt source destination

Chain POST_public_deny (1 references)
target prot opt source destination

Chain POST_public_log (1 references)
target prot opt source destination

Chain PREROUTING_ZONES (1 references)
target prot opt source destination
PRE_public all -- anywhere anywhere [goto]
PRE_public all -- anywhere anywhere [goto]
PRE_public all -- anywhere anywhere [goto]
PRE_public all -- anywhere anywhere [goto]
PRE_public all -- anywhere anywhere [goto]
PRE_public all -- anywhere anywhere [goto]

Chain PREROUTING_ZONES_SOURCE (1 references)
target prot opt source destination

Chain PREROUTING_direct (1 references)
target prot opt source destination

Chain PRE_public (6 references)
target prot opt source destination
PRE_public_log all -- anywhere anywhere
PRE_public_deny all -- anywhere anywhere
PRE_public_allow all -- anywhere anywhere

Chain PRE_public_allow (1 references)
target prot opt source destination

Chain PRE_public_deny (1 references)
target prot opt source destination

Chain PRE_public_log (1 references)
target prot opt source destination
[root@localhost wuivvi01]#



Regards,

Victoria

Re: Virbr0 problems

Posted: 2020/04/22 13:31:15
by jlehtone
VictoriaV wrote:
2020/04/21 12:22:25
Mi scenario is :

Guess (Machine1) with eth1 172.21.0.244 ---- Host with data1 172.21.0.243 ----eno33559296.361 no IP --------------- Cisco router with Giga2.361 172.21.0.242 and Loopback3 10.0.3.12
That is quite different from what you had before.

Code: Select all

Guest 172.21.0.244 ---\
                        bridge --- vlan.361 -- eno33559296 ==== 172.21.0.242 Cisco
Host  172.21.0.243 ---/
* What is the prefix of 172.21.0.x/y ?
* Does Cisco have rules about 172.21.0.244 ?

What does tcpdump -vvv -n -i $IF host 172.21.0.242, where $IF is data1, eno33559296.361, eno33559296
when guest does ping -c 1 172.21.0.242 ?

How about host running ping -c 1 172.21.0.242 ?

There are other commands that IMHO give useful data:

Code: Select all

ip li
ip -4 ad
ip ro
nmcli
nmcli d s
nmcli c s
iptables -S
iptables -t nat -S