[CLOUDSTACK-9431] Network usage stats from VR in VPC are wrong after upgrading to ACS 4.7

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.7.1
• Component/s: Usage, Virtual Router
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Patrick D.
• Reporter: Simon Godard
• Created: 19/Jul/16 01:15
• Updated: 25/Jul/16 20:38

Description

After upgrading to ACS 4.7.1 and our Virtual routers to 4.6.0, we noticed that the Network usage (bytes sent and received) were not good anymore. Bytes sent are now 0 and bytes received appear to be what used to be bytes sent before the update.

We are using Advanced networking with VPC on Xen Server 6.5.

I have checked the CloudStack Java code that is handling retrieving the network stats and nothing changed for a long time. What changed is the way the Virtual router is configured (now using Python scripts). After comparing the previous iptables rules, I noticed something weird with the NETWORK_STATS_eth1 chain:

iptables -L NETWORK_STATS_eth1 -n -v -x

Chain NETWORK_STATS_eth1 (1 references)
    pkts      bytes    target      prot opt in     out     source               destination         
       0        0        all            --  *      eth1    0.0.0.0/0            10.188.216.0/24     
   47170  7755561        all            --  *      eth1    10.188.216.0/24      0.0.0.0/0           
       0        0        all            --  *      eth1    0.0.0.0/0            10.188.218.0/24     
   71957 689541123       all            --  *      eth1    10.188.218.0/24      0.0.0.0/0  

The rules are out of order and the in and out too. Now, if we compare those rules against the previous ones (version 4.4.4 of the VR):

iptables -L NETWORK_STATS_eth1 -n -v -x

Chain NETWORK_STATS_eth1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
35167 2673K            all  --  any    eth1    10.158.216.0/22      anywhere            
33036 2511K            all  --  eth1   any     anywhere             10.158.216.0/22     

Once I noticed the differences, I tried to find the root cause in CsAddress.py
The following lines appear to be problematic.

self.fw.append(["", "front", "-A NETWORK_STATS_%s -o %s -s %s" %
                            ("eth1", "eth1", self.address['network'])])
self.fw.append(["", "front", "-A NETWORK_STATS_%s -o %s -d %s" %
                            ("eth1", "eth1", self.address['network'])])

After updating them to be the following (note the -i instead of -o on the second line), the network stats are back to normal:

self.fw.append(["", "front", "-A NETWORK_STATS_%s -o %s -s %s" %
                            ("eth1", "eth1", self.address['network'])])
self.fw.append(["", "front", "-A NETWORK_STATS_%s -i %s -d %s" %
                            ("eth1", "eth1", self.address['network'])])