[CLOUDSTACK-6715] [SDN] Inconsistency in ovs-flow table after vm migration from one host to another

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 4.4.0
• Fix Version/s: 4.4.0
• Component/s: Management Server, Network Controller
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: Latest build from 4.4 with commit e6961fd21bb6d793302c234d0f409f66dc498072
• Assignee: Murali Reddy
• Reporter: Sanjeev N
• Created: 20/May/14 06:22
• Updated: 17/Jun/14 09:21
• Resolved: 17/Jun/14 09:21

Description

[SDN] Inconsistency in ovs-flow table after vm migration from one host to another

Steps to reproduce:
================
1.Bring up CS in advanced zone with two xen hosts in a cluster
2.Create physical network with GRE isolation
3.Create isolated network with OVS provider
4.Deploy few vms in the network and make sure that all the vms(including VR) are deployed on only one host.
5.Now migrate one vm to another host in the cluster and verify flow tables for this isolated network bridge on both the xen hosts

Result:
=======
Inconsistency in flow tables on both the xen hosts

Following it the flow table from the host before vm migration (All the vms and VR were only on this host before vm migration):
[root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=1173.14s, table=0, n_packets=0, n_bytes=0, priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:1,output:2
cookie=0x0, duration=1173.15s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
cookie=0x0, duration=1203.276s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=1,nw_dst=224.0.0.0/24 actions=NORMAL
cookie=0x0, duration=1226.258s, table=0, n_packets=901, n_bytes=85612, priority=0 actions=NORMAL
cookie=0x0, duration=1173.129s, table=0, n_packets=0, n_bytes=0, priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:1,output:2
cookie=0x0, duration=1203.286s, table=0, n_packets=0, n_bytes=0, priority=1200,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL
cookie=0x0, duration=1173.167s, table=0, n_packets=7, n_bytes=1494, priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL

Flow table after vm migration on the same host:
[root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=992.789s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
[root@xen-host-14 ~]#

Ports info on the bridge after vm migration:
[root@xen-host-14 ~]# ovs-vsctl list-ports xapi4
t986-2-1
vif11.0
vif12.0

Flow table on the target host where vm was migrated to:
[root@xen-host-13 ~]# ovs-ofctl dump-flows xapi4
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=1025.129s, table=0, n_packets=0, n_bytes=0, priority=1100,dl_dst=ff:ff:ff:ff:ff:ff actions=output:2
cookie=0x0, duration=1025.139s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24 actions=NORMAL
cookie=0x0, duration=1032.932s, table=0, n_packets=0, n_bytes=0, priority=1000,ip,in_port=1,nw_dst=224.0.0.0/24 actions=drop
cookie=0x0, duration=1033.247s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
cookie=0x0, duration=1025.119s, table=0, n_packets=0, n_bytes=0, priority=1100,ip,nw_dst=224.0.0.0/24 actions=output:2
cookie=0x0, duration=1032.942s, table=0, n_packets=0, n_bytes=0, priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff actions=drop
cookie=0x0, duration=1025.148s, table=0, n_packets=1, n_bytes=42, priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff actions=NORMAL

[root@xen-host-13 ~]# ovs-vsctl list-ports xapi4
t986-1-2
vif17.0

Following is the log snippet from both the hosts during vm migration (tunnel creation during vm migration):

2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 2 to host 1 with GRE key 986
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge', 'xapi4', 'name']
2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi4', 't986-2-1', '--', 'set', 'interface', 't986-2-1', 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.13']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'port', 't986-2-1', 'interfaces']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:key']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
2014-05-20 11:31:22 DEBUG [root] Tunnel interface validated:['/usr/bin/ovs-vsctl', 'get', 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'ofport']
2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:22 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network', '--minimal']
2014-05-20 11:31:22 DEBUG [root] The command exited with the error code: 1 (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
2014-05-20 11:31:22 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,ip,nw_dst=224.0.0.0/24,actions=drop']
2014-05-20 11:31:22 DEBUG [root] Broadcast drop rules added
2014-05-20 11:31:22 DEBUG [root] Successfully created tunnel from host 2 to host 1 with GRE key 986
2014-05-20 11:31:22 DEBUG [root] #### VMOPS exit create_tunnel ####
2014-05-20 11:31:29 DEBUG [root] Executing:['cat', '/etc/xensource/network.conf']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'iface-to-br', 'vif19.0']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network', '--minimal']
2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1 (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'br-to-vlan', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'list-ports', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface', 't986-2-1', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface', 'vif11.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface', 'vif12.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface', 'vif19.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'del-flows', 'xapi4', ',in_port=3']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:1,output:2']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:1,output:2']

ovstunnel log from the host where vm has been migrated to:

2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter setup_ovs_bridge ####
2014-05-20 11:31:21 DEBUG [root] About to manually create the bridge:xapi4
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--', '--may-exist', 'add-br', 'xapi4', '--', 'set', 'bridge', 'xapi4', 'other_config:gre_key=OVSTunnel986']
2014-05-20 11:31:21 DEBUG [root] Bridge has been manually created:
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set', 'Bridge', 'xapi4', 'external_ids:xs-network-uuid=98636962-280c-bb8f-1e57-d79bcc295c1f']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set', 'Bridge', 'xapi4', 'stp_enable=true']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'bridge', 'xapi4', 'other_config:gre_key']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'other-config:is-ovs-tun-network=True']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=ovs-host-setup']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-set', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'other-config:ovs-host-setup=1,2']
2014-05-20 11:31:21 DEBUG [root] Setup_ovs_bridge completed with result:SUCCESS:xapi4
2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit setup_ovs_bridge ####
2014-05-20 11:31:21 DEBUG [root] #### VMOPS enter create_tunnel ####
2014-05-20 11:31:21 DEBUG [root] Creating tunnel from host 1 to host 2 with GRE key 986
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=30', 'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge', 'xapi4', 'name']
2014-05-20 11:31:21 DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi4', 't986-1-2', '--', 'set', 'interface', 't986-1-2', 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.14']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'port', 't986-1-2', 'interfaces']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:key']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
2014-05-20 11:31:21 DEBUG [root] Tunnel interface validated:['/usr/bin/ovs-vsctl', 'get', 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'ofport']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:21 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network', '--minimal']
2014-05-20 11:31:21 DEBUG [root] The command exited with the error code: 1 (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
2014-05-20 11:31:21 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,ip,nw_dst=224.0.0.0/24,actions=drop']
2014-05-20 11:31:21 DEBUG [root] Broadcast drop rules added
2014-05-20 11:31:21 DEBUG [root] Successfully created tunnel from host 1 to host 2 with GRE key 986
2014-05-20 11:31:21 DEBUG [root] #### VMOPS exit create_tunnel ####
2014-05-20 11:31:29 DEBUG [root] Executing:['cat', '/etc/xensource/network.conf']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'iface-to-br', 'vif17.0']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list', 'bridge=xapi4', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network', '--minimal']
2014-05-20 11:31:29 DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get', 'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network', '--minimal']
2014-05-20 11:31:29 DEBUG [root] The command exited with the error code: 1 (stderr output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
)
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'br-to-vlan', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'list-ports', 'xapi4']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface', 't986-1-2', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface', 'vif17.0', 'ofport']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff,actions=NORMAL']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,ip,nw_dst=224.0.0.0/24,actions=NORMAL']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:2']
2014-05-20 11:31:29 DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4', 'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:2']
~

Issue Links

• Is contained by: CLOUDSTACK-6779 [OVS] Expunging VM (deleting vif) deletes all the rules from ovs bridge flow table