Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-5505

[Automation] Private gateway not getting programmed in VPC router

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 4.3.0
    • 4.4.0
    • Network Controller
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None
    • KVM
      Branch : 4.3

    Description

      This issue found as part of regression automation
      integration.component.test_vpc_vms_deployment.TestVMDeployVPC.test_07_delete_network_with_rules

      Test cases performing below steps

      1726 # Validate the following
      1727 # 1. Create a VPC with cidr - 10.1.1.1/16
      1728 # 2. Add network1(10.1.1.1/24) and network2(10.1.2.1/24) to this VPC.
      1729 # 3. Deploy vm1 and vm2 in network1 and vm3 and vm4 in network2.
      1730 # 4. Create a PF /Static Nat/LB rule for vms in network1.
      1731 # 5. Create a PF /Static Nat/LB rule for vms in network2.
      1732 # 6. Create ingress network ACL for allowing all the above rules from
      1733 # public ip range on network1 and network2.
      1734 # 7. Create egress network ACL for network1 and network2 to access
      1735 # google.com.
      1736 # 8. Create a private gateway for this VPC and add a static route to
      1737 # this gateway
      1738 # 9. Create a VPN gateway for this VPC and add a static route to this
      1739 # gateway.
      1740 # 10. Make sure that all the PF,LB, Static NAT rules work as expected
      1741 # 11. Make sure that we are able to access google from all user Vms
      1742 # 12. Make sure that the newly added private gateway's and VPN
      1743 # gateway's static routes work as expected.
      1744 # Steps:
      1745 # 1. Delete the 1st network.
      1746 # 2. Delete account
      1747 # Validations:
      1748 # 1. As part of network deletion all the resources attached with
      1749 # network should get deleted. All other VMs and rules shall work as
      1750 # expected
      1751 # 2. All the resources associated with account should be deleted

      Steps to reproduce

      Step 1 : Create advanced zone in KVM
      Step 2 : Create VPC
      Step 3 : create 2 network inside
      Step 4 : Create Private Gateway
      Step 5 : Add static route

      Result

      Failed to add static route with error "Failed to create static route"

      MS log (look lob Job-Executor-102:ctx-12de6228)
      -------------------------------------------------------------------

      2013-12-13 15:23:12,952 DEBUG [c.c.a.ApiServlet] (catalina-exec-7:ctx-0c9be854) ===START=== 10.214.5.33 – GET command=createStaticRoute&response=json&sessionkey=RP7BqoAP0Roa1gkg68z4YhvFAO4%3D&g
      atewayid=528f0ba2-bd3d-4e76-9d45-d567cbd47d67&cidr=10.2.3.0%2F24&_=1386976993256
      2013-12-13 15:23:12,964 DEBUG [c.c.n.v.VpcManagerImpl] (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) Adding static route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4]
      2013-12-13 15:23:13,049 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-102:ctx-12de6228) Add job-2928 into job monitoring
      2013-12-13 15:23:13,049 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-102:ctx-12de6228) Executing AsyncJobVO {id:2928, userId: 2, accountId: 2, instanceType: StaticRoute, instanceId: 4, cm
      d: org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd, cmdInfo:

      {"id":"4","response":"json","sessionkey":"RP7BqoAP0Roa1gkg68z4YhvFAO4\u003d","cmdEventType":"STATIC.ROUTE.CREATE","ctxU serId":"2","gatewayid":"528f0ba2-bd3d-4e76-9d45-d567cbd47d67","httpmethod":"GET","_":"1386976993256","ctxAccountId":"2","ctxStartEventId":"14430","cidr":"10.2.3.0/24"}

      , cmdVersion: 0, status: IN_P
      ROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 29066118877352, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
      2013-12-13 15:23:13,053 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) submit async job-2928, details: AsyncJobVO {id:2928, userId: 2, accountId: 2, instanceTy
      pe: StaticRoute, instanceId: 4, cmd: org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd, cmdInfo:

      {"id":"4","response":"json","sessionkey":"RP7BqoAP0Roa1gkg68z4YhvFAO4\u003d","cmdEven tType":"STATIC.ROUTE.CREATE","ctxUserId":"2","gatewayid":"528f0ba2-bd3d-4e76-9d45-d567cbd47d67","httpmethod":"GET","_":"1386976993256","ctxAccountId":"2","ctxStartEventId":"14430","cidr":"10.2.3.0 /24"}

      , cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 29066118877352, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
      2013-12-13 15:23:13,055 DEBUG [c.c.a.ApiServlet] (catalina-exec-7:ctx-0c9be854 ctx-e80dd4b0) ===END=== 10.214.5.33 – GET command=createStaticRoute&response=json&sessionkey=RP7BqoAP0Roa1gkg68z4Y
      hvFAO4%3D&gatewayid=528f0ba2-bd3d-4e76-9d45-d567cbd47d67&cidr=10.2.3.0%2F24&_=1386976993256
      2013-12-13 15:23:13,060 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applying static routes for vpc [VPC [77-RayeesVPC]
      2013-12-13 15:23:13,070 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592525: Sending { Cmd , MgmtId: 29066118877352, via: 2(Rack2Host12.lab.vmops.com), Ver: v1,
      Flags: 100001, [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[

      {"id":4,"uuid":"7c159911-d38a-4dbf-a98d-708b099c7389","targetCidr":"10.2.3.0/24","accountId":2,"domainId":1,"g atewayId":4,"state":"Add","vpcId":77,"vlanTag":"2364","gateway":"10.2.3.1","netmask":"255.255.255.0","ipAddress":"10.2.3.2"}

      ],"accessDetails":

      {"zone.network.type":"Advanced","router.ip":"169.254.2 .108","router.name":"r-706-QA"}

      ,"wait":0}}] }
      2013-12-13 15:23:13,308 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 2-1828592525: Processing: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, [{"com.cloud.agent.api.rou
      ting.SetStaticRouteAnswer":{"results":["Failed"],"result":false,"wait":0}}] }
      2013-12-13 15:23:13,308 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592525: Received: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0,

      { SetStaticRou teAnswer } }
      2013-12-13 15:23:13,309 DEBUG [c.c.a.ApiServlet] (catalina-exec-14:ctx-75d4f4a5) ===START=== 10.223.240.194 – GET signature=m9opIJ%2BNeEGTeQ0NEl5GZnZtitU%3D&apiKey=LepyvCSNC55jvLUEoHU3GP9oI4QiT
      vgt9dsCdALVIf3NxlNBq7dmcHdCb_6NenlyV3QwRmEn5DnGp__U-5UwXA&command=queryAsyncJobResult&response=json&jobid=08d6fc64-19d4-433b-bcd2-20d487e43dd5
      2013-12-13 15:23:13,314 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Revoking static route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4]
      2013-12-13 15:23:13,319 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Marked static route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4] with state R
      evoke
      2013-12-13 15:23:13,322 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applying static routes for vpc [VPC [77-RayeesVPC]
      2013-12-13 15:23:13,331 DEBUG [c.c.a.ApiServlet] (catalina-exec-14:ctx-75d4f4a5 ctx-e2e99bdf ctx-a1b9efd2) ===END=== 10.223.240.194 – GET signature=m9opIJ%2BNeEGTeQ0NEl5GZnZtitU%3D&apiKey=Lepyv
      CSNC55jvLUEoHU3GP9oI4QiTvgt9dsCdALVIf3NxlNBq7dmcHdCb_6NenlyV3QwRmEn5DnGp__U-5UwXA&command=queryAsyncJobResult&response=json&jobid=08d6fc64-19d4-433b-bcd2-20d487e43dd5
      2013-12-13 15:23:13,340 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592526: Sending { Cmd , MgmtId: 29066118877352, via: 2(Rack2Host12.lab.vmops.com), Ver: v1,
      Flags: 100001, [{"com.cloud.agent.api.routing.SetStaticRouteCommand":{"staticRoutes":[{"id":4,"uuid":"7c159911-d38a-4dbf-a98d-708b099c7389","targetCidr":"10.2.3.0/24","accountId":2,"domainId":1,"gatewayId":4,"state":"Revoke","vpcId":77,"vlanTag":"2364","gateway":"10.2.3.1","netmask":"255.255.255.0","ipAddress":"10.2.3.2"}],"accessDetails":{"zone.network.type":"Advanced","router.ip":"169.254.2.108","router.name":"r-706-QA"},"wait":0}}] }

      2013-12-13 15:23:13,563 DEBUG [c.c.a.t.Request] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Seq 2-1828592526: Received: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 0, { SetStaticRouteAnswer }

      }
      2013-12-13 15:23:13,563 DEBUG [c.c.n.e.VpcVirtualRouterElement] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applied static routes on vpc [VPC [77-RayeesVPC]
      2013-12-13 15:23:13,563 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Applied static routes for vpc [VPC [77-RayeesVPC]
      2013-12-13 15:23:13,567 DEBUG [c.c.n.NetworkModelImpl] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Service SecurityGroup is not supported in the network id=572
      2013-12-13 15:23:13,574 DEBUG [c.c.n.v.VpcManagerImpl] (Job-Executor-102:ctx-12de6228 ctx-e80dd4b0) Removed route StaticRoute[7c159911-d38a-4dbf-a98d-708b099c7389|10.2.3.0/24|4] from the DB
      2013-12-13 15:23:13,578 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-102:ctx-12de6228) Complete async job-2928, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/

      {"uuidList":[],"errorcode":530,"errortext":"Failed to create static route"}

      2013-12-13 15:23:13,584 DEBUG [c.c.n.NetworkModelImpl] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Service SecurityGroup is not supported in the network id=572
      2013-12-13 15:23:13,589 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Applying dhcp entry in network Ntwk[572|Guest|156]
      2013-12-13 15:23:13,597 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-102:ctx-12de6228) Done executing org.apache.cloudstack.api.command.user.vpc.CreateStaticRouteCmd for job-2928
      2013-12-13 15:23:13,601 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-102:ctx-12de6228) Remove job-2928 from job monitoring
      2013-12-13 15:23:13,603 DEBUG [c.c.a.t.Request] (Job-Executor-15:ctx-f757750b ctx-4a7c2e5e) Seq 1-1935872252: Sending { Cmd , MgmtId: 29066118877352, via: 1(Rack2Host11.lab.vmops.com), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:42:c8:00:01","vmIpAddress":"10.1.1.122","vmName":"QA-2ac35f13-8ced-4c76-b983-5bdea778e276","defaultRouter":"10.1.1.1","duid":"00:03:00:01:02:00:42:c8:00:01","isDefault":true,"executeInSequence":false,"accessDetails":

      {"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-732-QA","router.ip":"169.254.1.101"}

      ,"wait":0}}] }

      Router Log
      -------------
      if you look the message.log in side the VPC router, you can see Private Gateway itself not programmed (attaching all log, please check router log r-706-QA)

      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160401] pci 0000:00:0b.0: BAR 6: assigned [mem 0x08040000-0x0804ffff pref]
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160406] pci 0000:00:0b.0: BAR 1: assigned [mem 0x08013000-0x08013fff]
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160443] pci 0000:00:0b.0: BAR 0: assigned [io 0x1060-0x107f]
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160483] pci 0000:00:00.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160486] pci 0000:00:00.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160536] pci 0000:00:01.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160539] pci 0000:00:01.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160584] ata_piix 0000:00:01.1: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160587] ata_piix 0000:00:01.1: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160637] uhci_hcd 0000:00:01.2: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160641] uhci_hcd 0000:00:01.2: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160818] piix4_smbus 0000:00:01.3: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160822] piix4_smbus 0000:00:01.3: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160872] pci 0000:00:02.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.160875] pci 0000:00:02.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163389] virtio-pci 0000:00:03.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163393] virtio-pci 0000:00:03.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163430] virtio-pci 0000:00:04.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163433] virtio-pci 0000:00:04.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163488] virtio-pci 0000:00:05.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163491] virtio-pci 0000:00:05.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163532] virtio-pci 0000:00:06.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163535] virtio-pci 0000:00:06.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163581] virtio-pci 0000:00:07.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163584] virtio-pci 0000:00:07.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163640] virtio-pci 0000:00:08.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163644] virtio-pci 0000:00:08.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163795] virtio-pci 0000:00:09.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163798] virtio-pci 0000:00:09.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163855] pci 0000:00:0b.0: no hotplug settings from platform
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.163858] pci 0000:00:0b.0: using default PCI settings
      Dec 13 23:45:39 r-706-QA kernel: [ 2365.166368] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)
      Dec 13 23:45:39 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 0 seconds
      Dec 13 23:45:40 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 1 seconds
      Dec 13 23:45:41 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 2 seconds
      Dec 13 23:45:42 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 3 seconds
      Dec 13 23:45:43 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 4 seconds
      Dec 13 23:45:44 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 5 seconds
      Dec 13 23:45:45 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 6 seconds
      Dec 13 23:45:46 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 7 seconds
      Dec 13 23:45:47 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 8 seconds
      Dec 13 23:45:48 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 9 seconds
      Dec 13 23:45:49 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 10 seconds
      Dec 13 23:45:50 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 11 seconds
      Dec 13 23:45:51 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 12 seconds
      Dec 13 23:45:52 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 13 seconds
      Dec 13 23:45:53 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 14 seconds
      Dec 13 23:45:54 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 15 seconds
      Dec 13 23:45:55 r-706-QA cloud: vpc_ipassoc.sh:Waiting for interface ethnull to appear, 16 seconds
      Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:interface ethnull never appeared
      Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:Adding ip 10.2.3.2 on interface ethnull
      Dec 13 23:45:56 r-706-QA cloud: vpc_ipassoc.sh:Add routing 10.2.3.2 on interface ethnull
      Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: enter apply acl rules on private gateway interface : eth4, inbound::all:0:0:0.0.0.0/0
      Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: exit apply acl rules for private gw interface : eth4
      Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: successful in applying acl rules on private gateway interface : eth4
      Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: enter apply acl rules on private gateway interface : eth4, inbound::all:0:0:0.0.0.0/0
      Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: exit apply acl rules for private gw interface : eth4
      Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: successful in applying acl rules on private gateway interface : eth4
      Dec 13 23:45:57 r-706-QA cloud: vpc_privategw_acl.sh: deleting backup on private gateway interface : eth4

      Attachments

        1. CLOUDSTACK-5505.rar
          6.13 MB
          Rayees Namathponnan

        Activity

          People

            jayapal Jayapal
            rayeesn Rayees Namathponnan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: