[CLOUDSTACK-9770] Virtual router / Network regression since 4.9.1.0 with public interface eth2

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.10.0.0, 4.9.2.0, 4.9.1.0
• Fix Version/s: Future, 4.10.0.0
• Component/s: Virtual Router
• Security Level: Public (Anyone can view this level - this is the default.)
• Labels: regresion
• Environment: CloudStack with advanced network installation
• Flags: Important
• Assignee: Unassigned
• Reporter: Bruno Demion
• Created: 02/Feb/17 18:59
• Updated: 27/Sep/17 13:18
• Resolved: 09/Feb/17 08:15

Description

I found a (possible) bug introduce by CLOUDSTACK-9339 [1] (Pull Request PR1659 [2]) on CloudStack Advanced network installation.

Since this changes (9339), the public network's route on eth2 (public interface) in VR is missing.

Before on VR, we have sometimes like:

ip route show table Table_eth2
212.217.2.0/24 dev eth2 table Table_eth2 scope link
default via 212.217.2.1 dev eth2
...

where 212.217.2.0/24 is the public network and 212.217.2.1 the default gateway.

After with 4.9.1.0+ the ip route command shows only:
default via 212.217.2.1 dev eth2 proto static
throw 10.230.1.0/24 proto static
throw 169.254.0.0/16 proto static

(missing route for public network)

The changes 9339 introduce the iptables connmark to add 0x2 mark on ip packets from internal VMs IP and an ip rule to use the Table_eth2 network table for these ip packets.
So if another machine into the public network try to reach a virtual machine inside CloudStack using their public IP, the packets's travel is:
source_machine--> VR (de-NAT) --> VM_inside_CS --> VR (NAT+using Table_eth2) --> default_public_gateway --> source machine

The issue is if the default_public_gateway refuse to forward IP packets with the source IP and destination IP in the same network (often when the gateway is a firewall), then the connection between a machine into public network is not possible with all VM behind the CS virtual router.

The correct network path for the packet must be:
source_machine--> VR (de-nat) --> VM_inside_CS --> VR (NAT+using Table_eth2) --> source machine (directly because on public network)

To fix the issue (workaround), just execute this command on the virtual router:
ip route add dev eth2 table Table_eth2212.217.2.0/24

Please note: this issue isn't visible on CloudStack upgrade installation from anterior version of 4.9.1.0+ until you decide to restart with clean up the network in CS.

What is the best way to fix this bug?

Thanks

[1] https://issues.apache.org/jira/browse/CLOUDSTACK-9339
[2] https://github.com/apache/cloudstack/pull/1659

Issue Links

• Blocked: CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces
• Blocked: CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces
• is duplicated by: CLOUDSTACK-10097 NAT loopback not working on VPC
• links to: GitHub Pull Request #1929