Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.1
    • Fix Version/s: 4.1.0
    • Component/s: Network Controller
    • Security Level: Public (Anyone can view this level - this is the default.)
    • Labels:
      None
    • Environment:
      centos 6.4
      HP BL460 G1

      Description

      Hi,

      I have setup a cloudstack instance where my "root" eth device is a vlan tagged bond0.60 (as the network I am on has a different default VLAN id than my test vlans).

      so I am setup like this:

      bond0.60 / cloudbr0 == management network / ip of box (bond0 == nothing)

      bond0.60 Link encap:Ethernet HWaddr 00:17:A4:77:48:3C
      inet6 addr: fe80::217:a4ff:fe77:483c/64 Scope:Link
      UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
      RX packets:37189 errors:0 dropped:0 overruns:0 frame:0
      TX packets:34030 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:4476334 (4.2 MiB) TX bytes:31055747 (29.6 MiB)
      cloudbr0 Link encap:Ethernet HWaddr 00:17:A4:77:48:3C
      inet addr:172.18.102.8 Bcast:172.18.102.255 Mask:255.255.255.0
      inet6 addr: fe80::217:a4ff:fe77:483c/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:36531 errors:0 dropped:0 overruns:0 frame:0
      TX packets:32606 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:4435824 (4.2 MiB) TX bytes:30976056 (29.5 MiB)

      when it went to setup a new guest network (with a vlan id of 80) it created it ontop of the bond0.60 like:

      bond0.60.80 Link encap:Ethernet HWaddr 00:17:A4:77:48:3C
      inet6 addr: fe80::217:a4ff:fe77:483c/64 Scope:Link
      UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:0 (0.0 b) TX bytes:13777 (13.4 KiB)

      [root@slo-cnkvm004 ~]# brctl show
      bridge name bridge id STP enabled interfaces
      cloud0 8000.000000000000 no
      cloudVirBr80 8000.0017a477483c no bond0.60.80

      which doesn't seem to work and I am pretty sure is syntactically wrong. I can't ping any guests that come up on that network. When creating new devices it should I believe be creating them off of the base eth device (ie eth0, or bond0).

        Activity

        Hide
        Marcus Sorensen added a comment -

        I'll take a look at this when I get a moment, I'll wait until I have time
        before assigning it to myself. I'm thinking this has to do with the 'bond'
        device, since it's normally supposed to look in /proc/net/vlan, to see if
        the parent interface is tagged, and then look up the parent of THAT
        interface.

        Show
        Marcus Sorensen added a comment - I'll take a look at this when I get a moment, I'll wait until I have time before assigning it to myself. I'm thinking this has to do with the 'bond' device, since it's normally supposed to look in /proc/net/vlan, to see if the parent interface is tagged, and then look up the parent of THAT interface.
        Hide
        Marcus Sorensen added a comment -

        Just a few quick things... cloudbr0 is being set as your guest bridge,
        right? See /etc/cloud/agent/agent.properties

        Is there a /proc/net/vlan/bond0.60? If so, can you send the contents? It
        should be building guest bridges off of whatever is listed in the "Device:"
        line in this file.

        There should also be some logs in the /var/log/cloud/agent/agent.log file
        that might indicate what's going on.

        Show
        Marcus Sorensen added a comment - Just a few quick things... cloudbr0 is being set as your guest bridge, right? See /etc/cloud/agent/agent.properties Is there a /proc/net/vlan/bond0.60? If so, can you send the contents? It should be building guest bridges off of whatever is listed in the "Device:" line in this file. There should also be some logs in the /var/log/cloud/agent/agent.log file that might indicate what's going on.
        Hide
        danny webb added a comment -

        [root@slo-cnkvm005 network-scripts]# cat /etc/cloud/agent/agent.properties
        #Storage
        #Thu Apr 11 15:02:23 BST 2013
        guest.network.device=cloudbr0
        workers=5
        private.network.device=cloudbr0
        port=8250
        resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
        pod=3
        zone=4
        guid=81b405d4-1c9b-3b93-b41b-5331afff6305
        public.network.device=cloudbr0
        cluster=5
        local.storage.uuid=488ba4b4-961f-42ac-a8c8-a1a3fcc2ac7b
        domr.scripts.dir=scripts/network/domr/kvm
        LibvirtComputingResource.id=16
        host=172.18.102.5

        [root@slo-cnkvm005 network-scripts]# cat /proc/net/vlan/bond0.60
        bond0.60 VID: 60 REORDER_HDR: 1 dev->priv_flags: 2001
        total frames received 1910633
        total bytes received 2601941203
        Broadcast/Multicast Rcvd 10555

        total frames transmitted 1006816
        total bytes transmitted 75539004
        total headroom inc 0
        total encap on xmit 0
        Device: bond0
        INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
        EGRESS priority mappings:

        there is nothing really of interest in the agent.log......just the normal startup messages:

        2013-04-11 15:02:22,215 INFO [cloud.agent.AgentShell] (main:null) Implementation Version is 4.0.1.20130201075054
        2013-04-11 15:02:22,215 INFO [cloud.agent.AgentShell] (main:null) agent.properties found at /etc/cloud/agent/agent.properties
        2013-04-11 15:02:22,216 INFO [cloud.agent.AgentShell] (main:null) Defaulting to using properties file for storage
        2013-04-11 15:02:22,218 INFO [cloud.agent.AgentShell] (main:null) Defaulting to the constant time backoff algorithm
        2013-04-11 15:02:22,293 INFO [cloud.agent.Agent] (main:null) id is
        2013-04-11 15:02:22,431 INFO [resource.virtualnetwork.VirtualRoutingResource] (main:null) VirtualRoutingResource _scriptDir to use: scripts/network/domr/kvm
        2013-04-11 15:02:22,749 INFO [kvm.resource.LibvirtComputingResource] (main:null) No libvirt.vif.driver specififed. Defaults to BridgeVifDriver.
        2013-04-11 15:02:22,759 INFO [cloud.agent.Agent] (main:null) Agent [id = new : type = LibvirtComputingResource : zone = 4 : pod = 3 : workers = 5 : host = 172.18.102.5 : port = 8250
        2013-04-11 15:02:22,769 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to 172.18.102.5:8250
        2013-04-11 15:02:23,248 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done
        2013-04-11 15:02:23,530 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) Default Builder inited.
        2013-04-11 15:02:23,611 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Proccess agent startup answer, agent id = 16
        2013-04-11 15:02:23,611 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Set agent id 16
        2013-04-11 15:02:23,618 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Startup Response Received: agent id = 16

        Show
        danny webb added a comment - [root@slo-cnkvm005 network-scripts] # cat /etc/cloud/agent/agent.properties #Storage #Thu Apr 11 15:02:23 BST 2013 guest.network.device=cloudbr0 workers=5 private.network.device=cloudbr0 port=8250 resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource pod=3 zone=4 guid=81b405d4-1c9b-3b93-b41b-5331afff6305 public.network.device=cloudbr0 cluster=5 local.storage.uuid=488ba4b4-961f-42ac-a8c8-a1a3fcc2ac7b domr.scripts.dir=scripts/network/domr/kvm LibvirtComputingResource.id=16 host=172.18.102.5 [root@slo-cnkvm005 network-scripts] # cat /proc/net/vlan/bond0.60 bond0.60 VID: 60 REORDER_HDR: 1 dev->priv_flags: 2001 total frames received 1910633 total bytes received 2601941203 Broadcast/Multicast Rcvd 10555 total frames transmitted 1006816 total bytes transmitted 75539004 total headroom inc 0 total encap on xmit 0 Device: bond0 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 EGRESS priority mappings: there is nothing really of interest in the agent.log......just the normal startup messages: 2013-04-11 15:02:22,215 INFO [cloud.agent.AgentShell] (main:null) Implementation Version is 4.0.1.20130201075054 2013-04-11 15:02:22,215 INFO [cloud.agent.AgentShell] (main:null) agent.properties found at /etc/cloud/agent/agent.properties 2013-04-11 15:02:22,216 INFO [cloud.agent.AgentShell] (main:null) Defaulting to using properties file for storage 2013-04-11 15:02:22,218 INFO [cloud.agent.AgentShell] (main:null) Defaulting to the constant time backoff algorithm 2013-04-11 15:02:22,293 INFO [cloud.agent.Agent] (main:null) id is 2013-04-11 15:02:22,431 INFO [resource.virtualnetwork.VirtualRoutingResource] (main:null) VirtualRoutingResource _scriptDir to use: scripts/network/domr/kvm 2013-04-11 15:02:22,749 INFO [kvm.resource.LibvirtComputingResource] (main:null) No libvirt.vif.driver specififed. Defaults to BridgeVifDriver. 2013-04-11 15:02:22,759 INFO [cloud.agent.Agent] (main:null) Agent [id = new : type = LibvirtComputingResource : zone = 4 : pod = 3 : workers = 5 : host = 172.18.102.5 : port = 8250 2013-04-11 15:02:22,769 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to 172.18.102.5:8250 2013-04-11 15:02:23,248 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done 2013-04-11 15:02:23,530 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) Default Builder inited. 2013-04-11 15:02:23,611 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Proccess agent startup answer, agent id = 16 2013-04-11 15:02:23,611 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Set agent id 16 2013-04-11 15:02:23,618 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Startup Response Received: agent id = 16
        Hide
        danny webb added a comment - - edited

        it is my "guest" device in the agent.properties, but in my zone / pod the guest network was setup on the fly with:

        cloudVirBr70

        cloudVirBr70 8000.0017a4774830 no bond0.70
        vnet2

        (that is from another node though as only one node has anything on the pod guest network)

        the guest network that is causing issues is a secondary "guest" network that I setup after setting up the zone.

        Show
        danny webb added a comment - - edited it is my "guest" device in the agent.properties, but in my zone / pod the guest network was setup on the fly with: cloudVirBr70 cloudVirBr70 8000.0017a4774830 no bond0.70 vnet2 (that is from another node though as only one node has anything on the pod guest network) the guest network that is causing issues is a secondary "guest" network that I setup after setting up the zone.
        Hide
        Marcus Sorensen added a comment -

        Ok, so if I understand correctly (and I don't think I do), you set up a zone without specifying the "traffic label" for the guest traffic, and cloudstack created cloudVirBr70 for you for guest traffic, then later you added another guest network and then set the traffic label to "cloudbr0"? Or can you guide me through the steps to reproduce?

        Show
        Marcus Sorensen added a comment - Ok, so if I understand correctly (and I don't think I do), you set up a zone without specifying the "traffic label" for the guest traffic, and cloudstack created cloudVirBr70 for you for guest traffic, then later you added another guest network and then set the traffic label to "cloudbr0"? Or can you guide me through the steps to reproduce?
        Hide
        danny webb added a comment -

        ok

        I will try to walk you through my network setup (which may be wrong). I modified the steps from the networking section of this to get my setup working:

        https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.1-incubating/html/Installation_Guide/hypervisor-kvm-install-flow.html (specifically the network section)

        so on the virt hosts I have 3 bonds setup in /etc/sysconfig/network-scripts (ie out of cloudstack control):

        [root@slo-cnkvm002 ~]# ifconfig -a bond0.50
        bond0.50 Link encap:Ethernet HWaddr 00:17:A4:77:48:30
        inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link
        UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
        RX packets:91125 errors:0 dropped:0 overruns:0 frame:0
        TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:4801038 (4.5 MiB) TX bytes:648 (648.0 b)

        [root@slo-cnkvm002 ~]# ifconfig -a bond0.60
        bond0.60 Link encap:Ethernet HWaddr 00:17:A4:77:48:30
        inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link
        UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
        RX packets:7048696 errors:0 dropped:0 overruns:0 frame:0
        TX packets:3669691 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:7104994786 (6.6 GiB) TX bytes:3705458318 (3.4 GiB)

        [root@slo-cnkvm002 ~]# ifconfig -a bond0.70
        bond0.70 Link encap:Ethernet HWaddr 00:17:A4:77:48:30
        inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link
        UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
        RX packets:446772 errors:0 dropped:0 overruns:0 frame:0
        TX packets:112149 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:493027550 (470.1 MiB) TX bytes:10803061 (10.3 MiB)

        I then setup one cloudbrX (cloudbr0) device on bond0.60 setup in /etc/sysconfig/network-scripts (ie out of cloudstack control). I didn't setup a cloudbr1 device.

        cloudbr0 Link encap:Ethernet HWaddr 00:17:A4:77:48:30
        inet addr:172.18.102.6 Bcast:172.18.102.255 Mask:255.255.255.0
        inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:5094443 errors:0 dropped:0 overruns:0 frame:0
        TX packets:2725564 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:5825135265 (5.4 GiB) TX bytes:1382551076 (1.2 GiB)

        In this setup I can ping my system vms fine on their "public" interface which is on the cloudVirBr70 setup by cloudstack and on their management interface which is on cloudbr0.

        my network vlan setup is as such:

        vlan 50 nameif cloud-internal ip address 172.18.101.1 255.255.255.0 zone guest cidr (set when setting up the zone, but won't really use as will use the guest networks below)
        vlan 60 nameif cloud-admin ip address 172.18.102.1 255.255.255.0 management
        vlan 70 nameif cloud-dmz ip address 172.18.103.1 255.255.255.0 public

          1. two additional guest networks setup under infrastructure -> zone -> physical network -> guest network

        vlan 80 nameif cloud-proddmz ip address 10.10.10.0 255.255.255.0
        vlan 90 nameif cloud-devdmz ip address 10.10.11.0 255.255.255.0

        not sure if that helps you.

        Show
        danny webb added a comment - ok I will try to walk you through my network setup (which may be wrong). I modified the steps from the networking section of this to get my setup working: https://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.1-incubating/html/Installation_Guide/hypervisor-kvm-install-flow.html (specifically the network section) so on the virt hosts I have 3 bonds setup in /etc/sysconfig/network-scripts (ie out of cloudstack control): [root@slo-cnkvm002 ~] # ifconfig -a bond0.50 bond0.50 Link encap:Ethernet HWaddr 00:17:A4:77:48:30 inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:91125 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4801038 (4.5 MiB) TX bytes:648 (648.0 b) [root@slo-cnkvm002 ~] # ifconfig -a bond0.60 bond0.60 Link encap:Ethernet HWaddr 00:17:A4:77:48:30 inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:7048696 errors:0 dropped:0 overruns:0 frame:0 TX packets:3669691 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:7104994786 (6.6 GiB) TX bytes:3705458318 (3.4 GiB) [root@slo-cnkvm002 ~] # ifconfig -a bond0.70 bond0.70 Link encap:Ethernet HWaddr 00:17:A4:77:48:30 inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:446772 errors:0 dropped:0 overruns:0 frame:0 TX packets:112149 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:493027550 (470.1 MiB) TX bytes:10803061 (10.3 MiB) I then setup one cloudbrX (cloudbr0) device on bond0.60 setup in /etc/sysconfig/network-scripts (ie out of cloudstack control). I didn't setup a cloudbr1 device. cloudbr0 Link encap:Ethernet HWaddr 00:17:A4:77:48:30 inet addr:172.18.102.6 Bcast:172.18.102.255 Mask:255.255.255.0 inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:5094443 errors:0 dropped:0 overruns:0 frame:0 TX packets:2725564 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5825135265 (5.4 GiB) TX bytes:1382551076 (1.2 GiB) In this setup I can ping my system vms fine on their "public" interface which is on the cloudVirBr70 setup by cloudstack and on their management interface which is on cloudbr0. my network vlan setup is as such: vlan 50 nameif cloud-internal ip address 172.18.101.1 255.255.255.0 zone guest cidr (set when setting up the zone, but won't really use as will use the guest networks below) vlan 60 nameif cloud-admin ip address 172.18.102.1 255.255.255.0 management vlan 70 nameif cloud-dmz ip address 172.18.103.1 255.255.255.0 public two additional guest networks setup under infrastructure -> zone -> physical network -> guest network vlan 80 nameif cloud-proddmz ip address 10.10.10.0 255.255.255.0 vlan 90 nameif cloud-devdmz ip address 10.10.11.0 255.255.255.0 not sure if that helps you.
        Hide
        danny webb added a comment -

        ok, my problem was I tried to follow the directions . If I neglect to setup a cloudbr0 / 1 and have cloudstack do it on my virt host for me things go much better. when all else fails ask someone who has an already working implementation

        Show
        danny webb added a comment - ok, my problem was I tried to follow the directions . If I neglect to setup a cloudbr0 / 1 and have cloudstack do it on my virt host for me things go much better. when all else fails ask someone who has an already working implementation
        Hide
        danny webb added a comment -

        ok, it seems this is still an issue if you let cloudstack do the config. So I have blown away my install and am starting from scratch:

        my virt host before the install has only 2 devices:

        [root@slo-cnkvm001 ~]# ifconfig -a bond0
        bond0 Link encap:Ethernet HWaddr 00:17:A4:77:48:2C
        inet6 addr: fe80::217:a4ff:fe77:482c/64 Scope:Link
        UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
        RX packets:779329 errors:0 dropped:0 overruns:0 frame:0
        TX packets:272393 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:69545398 (66.3 MiB) TX bytes:149721368 (142.7 MiB)

        [root@slo-cnkvm001 ~]# ifconfig -a bond0.60
        bond0.60 Link encap:Ethernet HWaddr 00:17:A4:77:48:2C
        inet addr:172.18.102.5 Bcast:172.18.102.255 Mask:255.255.255.0
        inet6 addr: fe80::217:a4ff:fe77:482c/64 Scope:Link
        UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
        RX packets:288102 errors:0 dropped:0 overruns:0 frame:0
        TX packets:229952 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:30596209 (29.1 MiB) TX bytes:144701944 (137.9 MiB)

        and no non-standard bridges. note that the only IP is bound to bond0.60, as this host exists on a trunked interface whose default VLAN isn't one I want to be using.

        full details here: http://pastebin.com/aysPKGu2

        I then do an advanced zone:

        Zone: Slough
        Guest Cidr: left as default, not going to use it.

        Public = vlan 70 nameif cloud-dmz ip address 172.18.103.1 255.255.255.0 public
        Management = vlan 60 nameif cloud-admin ip address 172.18.102.1 255.255.255.0 management
        Guest net = vlan 50 nameif cloud-internal ip address 172.18.101.1 255.255.255.0 zone guest cidr

        after intitial setup I have this one the virt host:

        [root@slo-cnkvm002 ~]# brctl show
        bridge name bridge id STP enabled interfaces
        cloud0 8000.000000000000 no
        cloudbr0 8000.0017a4774830 no bond0.60
        virbr0 8000.5254001db6bc yes virbr0-nic

        cloud0 Link encap:Ethernet HWaddr B2:EF:86:35:1B:FC
        inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
        inet6 addr: fe80::b0ef:86ff:fe35:1bfc/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:0 errors:0 dropped:0 overruns:0 frame:0
        TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:0 (0.0 b) TX bytes:468 (468.0 b)

        cloudbr0 Link encap:Ethernet HWaddr 00:17:A4:77:48:30
        inet addr:172.18.102.6 Bcast:172.18.102.255 Mask:255.255.255.0
        inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link
        UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
        RX packets:1052 errors:0 dropped:0 overruns:0 frame:0
        TX packets:681 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:144230 (140.8 KiB) TX bytes:151059 (147.5 KiB)

        I enable the zone and wait for the system VMs to come online.

        once that is done I got to infrastructure -> zone -> physical network -> guest

        and add these 2 networks

        vlan 80 ip address 10.10.10.0 255.255.255.0
        vlan 90 ip address 10.10.11.0 255.255.255.0

        then I create a guest, and in doing so it creates this interface:

        cloudVirBr80 8000.0017a4774830 no bond0.60.80

        so moral of the story is, try starting off with a virt host with its root eth device on a vlan tagged interface. that seems to break the setup later on.

        Show
        danny webb added a comment - ok, it seems this is still an issue if you let cloudstack do the config. So I have blown away my install and am starting from scratch: my virt host before the install has only 2 devices: [root@slo-cnkvm001 ~] # ifconfig -a bond0 bond0 Link encap:Ethernet HWaddr 00:17:A4:77:48:2C inet6 addr: fe80::217:a4ff:fe77:482c/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:779329 errors:0 dropped:0 overruns:0 frame:0 TX packets:272393 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:69545398 (66.3 MiB) TX bytes:149721368 (142.7 MiB) [root@slo-cnkvm001 ~] # ifconfig -a bond0.60 bond0.60 Link encap:Ethernet HWaddr 00:17:A4:77:48:2C inet addr:172.18.102.5 Bcast:172.18.102.255 Mask:255.255.255.0 inet6 addr: fe80::217:a4ff:fe77:482c/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:288102 errors:0 dropped:0 overruns:0 frame:0 TX packets:229952 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:30596209 (29.1 MiB) TX bytes:144701944 (137.9 MiB) and no non-standard bridges. note that the only IP is bound to bond0.60, as this host exists on a trunked interface whose default VLAN isn't one I want to be using. full details here: http://pastebin.com/aysPKGu2 I then do an advanced zone: Zone: Slough Guest Cidr: left as default, not going to use it. Public = vlan 70 nameif cloud-dmz ip address 172.18.103.1 255.255.255.0 public Management = vlan 60 nameif cloud-admin ip address 172.18.102.1 255.255.255.0 management Guest net = vlan 50 nameif cloud-internal ip address 172.18.101.1 255.255.255.0 zone guest cidr after intitial setup I have this one the virt host: [root@slo-cnkvm002 ~] # brctl show bridge name bridge id STP enabled interfaces cloud0 8000.000000000000 no cloudbr0 8000.0017a4774830 no bond0.60 virbr0 8000.5254001db6bc yes virbr0-nic cloud0 Link encap:Ethernet HWaddr B2:EF:86:35:1B:FC inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0 inet6 addr: fe80::b0ef:86ff:fe35:1bfc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:468 (468.0 b) cloudbr0 Link encap:Ethernet HWaddr 00:17:A4:77:48:30 inet addr:172.18.102.6 Bcast:172.18.102.255 Mask:255.255.255.0 inet6 addr: fe80::217:a4ff:fe77:4830/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1052 errors:0 dropped:0 overruns:0 frame:0 TX packets:681 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:144230 (140.8 KiB) TX bytes:151059 (147.5 KiB) I enable the zone and wait for the system VMs to come online. once that is done I got to infrastructure -> zone -> physical network -> guest and add these 2 networks vlan 80 ip address 10.10.10.0 255.255.255.0 vlan 90 ip address 10.10.11.0 255.255.255.0 then I create a guest, and in doing so it creates this interface: cloudVirBr80 8000.0017a4774830 no bond0.60.80 so moral of the story is, try starting off with a virt host with its root eth device on a vlan tagged interface. that seems to break the setup later on.
        Hide
        Marcus Sorensen added a comment -

        Ok, I'll see if I can duplicate that, might not be today.

        One thing I would mention is that if you are starting out with your own
        tagged interface you want to avoid telling cloudstack about that tag since
        it isn't in control of it. Generally you want to also create a bridge for
        it as well and provide a traffic label. This way, you don't provide a vlan
        tag when filling out management details, you just tell cloudstack which
        bridge to tap into.

        So what might help is to create your bond0.60, then create brbond0.60 (or
        whatever you want to call your management bridge, cloudbr0 or whatever),
        then when you get to the physical network setup with the icons, provide
        "brbond0.60" as the traffic label for management. If you do this, when you
        get to the IP details, do not enter a vlan number for the pod/management
        info, leave it blank.

        If you want, you can do this for 50 and 70 as well, and cloudstack will
        only be in charge of creating the dynamic networks (ones provisioned via
        cloudstack).

        Show
        Marcus Sorensen added a comment - Ok, I'll see if I can duplicate that, might not be today. One thing I would mention is that if you are starting out with your own tagged interface you want to avoid telling cloudstack about that tag since it isn't in control of it. Generally you want to also create a bridge for it as well and provide a traffic label. This way, you don't provide a vlan tag when filling out management details, you just tell cloudstack which bridge to tap into. So what might help is to create your bond0.60, then create brbond0.60 (or whatever you want to call your management bridge, cloudbr0 or whatever), then when you get to the physical network setup with the icons, provide "brbond0.60" as the traffic label for management. If you do this, when you get to the IP details, do not enter a vlan number for the pod/management info, leave it blank. If you want, you can do this for 50 and 70 as well, and cloudstack will only be in charge of creating the dynamic networks (ones provisioned via cloudstack).
        Hide
        danny webb added a comment - - edited

        ok, I have tried a new setup where the root bond0 device is now on a totally separate network. The management is on a bond0.160 vlan tagged interface that I setup by hand. And still the Virtual router / additional guest networks keep doing the vlan chaining. The strange thing is the code works to create the initial bond / bridge for the public interface, ala:

        cloudVirBr170 8000.0017a4770400 no bond0.170
        vnet2

        but as soon as it trys to deploy the Virtual router and this new guest network it messes it up:

        cloudVirBr190 8000.0017a4770400 no bond0.160.190
        vnet4

        So there is something in the VR code or the deployment of guest networks that isn't quite right.

        here is a list of my network setup:

        http://pastebin.com/9X34nXUv

        Ah, also, my vlan setup has changed a bit because I am trying this in another DC:

        vlan 150 nameif cloud-guest – not really used
        vlan 160 nameif cloud-management – cloudbr0, created by hand during setup
        vlan 170 nameif cloud-public – dynamically created during setup of zone

        vlan 180 nameif cloud-dmz
        vlan 190 nameif cloud-internal – dynamically created wrong during creation of guest instance
        vlan 200 nameif cloud-admin
        vlan 210 nameif cloud-qa
        vlan 220 nameif cloud-dev
        vlan 230 nameif cloud-rm

        Show
        danny webb added a comment - - edited ok, I have tried a new setup where the root bond0 device is now on a totally separate network. The management is on a bond0.160 vlan tagged interface that I setup by hand. And still the Virtual router / additional guest networks keep doing the vlan chaining. The strange thing is the code works to create the initial bond / bridge for the public interface, ala: cloudVirBr170 8000.0017a4770400 no bond0.170 vnet2 but as soon as it trys to deploy the Virtual router and this new guest network it messes it up: cloudVirBr190 8000.0017a4770400 no bond0.160.190 vnet4 So there is something in the VR code or the deployment of guest networks that isn't quite right. here is a list of my network setup: http://pastebin.com/9X34nXUv Ah, also, my vlan setup has changed a bit because I am trying this in another DC: vlan 150 nameif cloud-guest – not really used vlan 160 nameif cloud-management – cloudbr0, created by hand during setup vlan 170 nameif cloud-public – dynamically created during setup of zone vlan 180 nameif cloud-dmz vlan 190 nameif cloud-internal – dynamically created wrong during creation of guest instance vlan 200 nameif cloud-admin vlan 210 nameif cloud-qa vlan 220 nameif cloud-dev vlan 230 nameif cloud-rm
        Hide
        danny webb added a comment -

        ok, I upgraded to the new 4.1 version and it seems whatever issue there was was fixed.

        Show
        danny webb added a comment - ok, I upgraded to the new 4.1 version and it seems whatever issue there was was fixed.
        Hide
        danny webb added a comment -

        issue was fixed in the 4.1.0 release, no idea why/how, just was

        Show
        danny webb added a comment - issue was fixed in the 4.1.0 release, no idea why/how, just was

          People

          • Assignee:
            Unassigned
            Reporter:
            danny webb
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development