Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-6779

[OVS] Expunging VM (deleting vif) deletes all the rules from ovs bridge flow table

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 4.4.0
    • Fix Version/s: Future
    • Security Level: Public (Anyone can view this level - this is the default.)
    • Labels:
    • Environment:
      Latest build from 4.4 with commit d130530bd3e1cd6d8249d5045e00e4e4e2201521

      Description

      [OVS] Expunging VM (deleting vif) deletes all the rules from ovs bridge flow table

      Steps to reproduce:
      ================
      1.Bring up CS in advanced zone with min of 2 xen hosts in a cluster
      2.Add physical network with GRE isolation
      3.Create network offering with connectivity service and OVS as the provider
      4.Create an isolated network with above offering and deploy few vms (4-5) in that network and make sure that vms are spanned across both the hosts
      5.After this verify flow table rules on the ovs bridge
      6.Expunge one of the vms created at step4
      7.Again verify flow table rules on the host from where vm was deleted

      Result:
      =====
      All the flow table rules were deleted from the OVS bridge. Only default flow rule is present.

      Attaching management server log file and ovstunnel log files from both the hosts.
      Please look for xapi5 and xapi7 in ovstunnel log files.

      Flow table rules before destroy the vm:
      [root@Rack1Pod1Host13 ~]# ovs-ofctl dump-flows xapi5
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=374.111s, table=0, n_packets=4, n_bytes=768, priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:1,output:2,output:3,output:4,output:5
      cookie=0x0, duration=300.162s, table=0, n_packets=0, n_bytes=0, priority=1000,ip,in_port=6,nw_dst=224.0.0.0/24 actions=drop
      cookie=0x0, duration=374.122s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=5,nw_dst=224.0.0.0/24 actions=NORMAL
      cookie=0x0, duration=5350.811s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=4,nw_dst=224.0.0.0/24 actions=NORMAL
      cookie=0x0, duration=6016.913s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=3,nw_dst=224.0.0.0/24 actions=NORMAL
      cookie=0x0, duration=6753.464s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
      cookie=0x0, duration=6890.423s, table=0, n_packets=7594, n_bytes=6858309, priority=0 actions=NORMAL
      cookie=0x0, duration=374.1s, table=0, n_packets=0, n_bytes=0, priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:1,output:2,output:3,output:4,output:5
      cookie=0x0, duration=374.132s, table=0, n_packets=5, n_bytes=810, priority=1200,in_port=5,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
      cookie=0x0, duration=5350.823s, table=0, n_packets=8, n_bytes=1236, priority=1200,in_port=4,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
      cookie=0x0, duration=300.173s, table=0, n_packets=0, n_bytes=0, priority=1000,in_port=6,dl_dst=ff:ff:ff:ff:ff:ff actions=drop
      cookie=0x0, duration=6016.924s, table=0, n_packets=8, n_bytes=1236, priority=1200,in_port=3,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
      cookie=0x0, duration=6753.474s, table=0, n_packets=6, n_bytes=852, priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL

      After destroying the vm:
      [root@Rack1Pod1Host13 ~]# ovs-ofctl dump-flows xapi5
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=877.901s, table=0, n_packets=46, n_bytes=9748, priority=0 actions=NORMAL
      [root@Rack1Pod1Host13 ~]#

      1. management-server.rar
        767 kB
        Sanjeev N
      2. ovstunnel.log_host13
        131 kB
        Sanjeev N
      3. ovstunnel.log_host14
        71 kB
        Sanjeev N

        Issue Links

          Activity

          Hide
          sanjeevn Sanjeev N added a comment -

          Attached log files from MS, ovstunnel files from both the hosts inside the cluster.

          Show
          sanjeevn Sanjeev N added a comment - Attached log files from MS, ovstunnel files from both the hosts inside the cluster.
          Hide
          murali.reddy Murali Reddy added a comment -

          It seems OpenVswitch daemon is crashing after VIF unplug on every VM destroy. I see a segfault message in dmesg for every VM expunge operation from CloudStack. Unfortunately OVS does not persist the OF rules, after restart, so even though bridge is created beack flow rules are missing. Will have to figure if there is way to persist the openflow rules.

          2275292.304366] /local/domain/74/device/vif/0: Closed
          [2275292.643164] ovs-vswitchd[3678]: segfault at 55 ip 080b71f5 sp bfadcfe0 error 4 in ovs-vswitchd[8048000+d3000]

          Show
          murali.reddy Murali Reddy added a comment - It seems OpenVswitch daemon is crashing after VIF unplug on every VM destroy. I see a segfault message in dmesg for every VM expunge operation from CloudStack. Unfortunately OVS does not persist the OF rules, after restart, so even though bridge is created beack flow rules are missing. Will have to figure if there is way to persist the openflow rules. 2275292.304366] /local/domain/74/device/vif/0: Closed [2275292.643164] ovs-vswitchd [3678] : segfault at 55 ip 080b71f5 sp bfadcfe0 error 4 in ovs-vswitchd [8048000+d3000]
          Hide
          murali.reddy Murali Reddy added a comment -

          OpenFlow rules are not meant to be persisted by OpenVswitch. This applies to both XenServer and KVM. So we need a holisitc way in CloudStack to reprogram the openflow rules to each bridge when there is a crash of OVS daemon and on host restart. This will be bigger effort I am afraid i can not fix it in for 4.4 release. I opened a bug CLOUDSTACK-6902 to document this behaviour as known issue.

          I am marking this bug for 4.5

          Show
          murali.reddy Murali Reddy added a comment - OpenFlow rules are not meant to be persisted by OpenVswitch. This applies to both XenServer and KVM. So we need a holisitc way in CloudStack to reprogram the openflow rules to each bridge when there is a crash of OVS daemon and on host restart. This will be bigger effort I am afraid i can not fix it in for 4.4 release. I opened a bug CLOUDSTACK-6902 to document this behaviour as known issue. I am marking this bug for 4.5

            People

            • Assignee:
              Unassigned
              Reporter:
              sanjeevn Sanjeev N
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development