CloudStack
  1. CloudStack
  2. CLOUDSTACK-1228

Unable to Create System Vm's in the VMware Hypervisor setup.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 4.1.0, 4.2.0
    • Fix Version/s: 4.1.0, 4.2.0
    • Component/s: Install and Setup
    • Security Level: Public (Anyone can view this level - this is the default.)
    • Labels:
      None

      Description

      Created a Advanced Zone setup using the Latest 4.1 build (http://repo-ccp.citrix.com/releases/ASF/rhel/6.3/4.1/CloudStack-non-OSS-11-rhel6.3.tar.gz)
      While trying to create the system VM's error messages are obserevd.

      Attaching the Management server logs.

      Error message Observed:

      Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://10.147.28.7/export/home/kiran/s, templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
      2013-02-11 23:03:41,530 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.47bfef59
      2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.648a617f
      2013-02-11 23:03:41,531 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.2257f5b0
      2013-02-11 23:03:41,532 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.3e22e9af
      2013-02-11 23:03:41,533 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.1e46e9bc
      2013-02-11 23:03:41,534 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.49fee4a0
      2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.c1b6a94
      2013-02-11 23:03:41,535 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.5604ba42
      2013-02-11 23:03:41,536 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.3632e0de
      2013-02-11 23:03:41,537 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount: /var/lib/cloud/mnt/VM/7313725325399.22475368
      2013-02-11 23:03:41,537 WARN [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create a mount point
      2013-02-11 23:03:41,538 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-2:10.147.40.18) Unable to create mount point for nfs://10.147.28.7/export/home/kiran/s
      2013-02-11 23:03:41,538 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Secondary storage mount point: /mnt/sec
      2013-02-11 23:03:41,538 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Executing command: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
      2013-02-11 23:03:41,539 DEBUG [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Executing: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
      2013-02-11 23:03:41,603 WARN [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-2:10.147.40.18) Exception: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
      java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or directory
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
      at com.cloud.utils.script.Script.execute(Script.java:183)
      at com.cloud.utils.script.Script.execute(Script.java:161)
      at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:443)
      at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138)
      at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587)
      at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401)
      at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
      Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
      at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
      at java.lang.ProcessImpl.start(ProcessImpl.java:81)
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
      ... 15 more

      1. catalina.2013-02-11.zip
        0.9 kB
        Kiran Koneti
      2. catalina.zip
        3.81 MB
        Kiran Koneti
      3. management-server.zip
        1.10 MB
        Kiran Koneti
      4. management-server-18-2-2013.zip
        86 kB
        Kiran Koneti

        Activity

        Hide
        Chip Childers added a comment -

        Closing this record, since 4.1.0 is now released.

        Show
        Chip Childers added a comment - Closing this record, since 4.1.0 is now released.
        Hide
        Sudha Ponnaganti added a comment -

        Looks lik the changes done in Kiran's setup for permission issue are causing lot of of other issues down the execution. Try with the build that chandan has created and see if that helps. Then will check in Pradeep's env what is causing these permission issues. Looks more like a build issue

        Show
        Sudha Ponnaganti added a comment - Looks lik the changes done in Kiran's setup for permission issue are causing lot of of other issues down the execution. Try with the build that chandan has created and see if that helps. Then will check in Pradeep's env what is causing these permission issues. Looks more like a build issue
        Hide
        Chandan Purushothama added a comment -

        Sudha,

        On my setup, the "root" user owns the templates folder and it still works for me. I didn't have to change the owner of the templates folder to "cloud" user as Kiran did inorder to work with the setup.

        On my setup:
        [root@asfmgmt 7200344900649.4910c1c6]# ls -lh
        total 32K
        drwxrwxrwx. 3 root root 4.0K Mar 13 16:07 snapshots
        drwxrwxr-x. 2 cloud cloud 4.0K Mar 11 18:53 systemvm
        drwxr-xr-x. 3 root root 4.0K Sep 11 2012 template
        drwxrwxrwx. 2 root root 4.0K Mar 11 18:54 volumes
        [root@asfmgmt 7200344900649.4910c1c6]#

        mysql> select id,name,state from vm_instance where vm_type != "User";
        ---------------------

        id name state

        ---------------------

        1 s-1-VMSNAP Running
        2 v-2-VMSNAP Running
        4 r-4-VMSNAP Running

        ---------------------
        3 rows in set (0.00 sec)

        Show
        Chandan Purushothama added a comment - Sudha, On my setup, the "root" user owns the templates folder and it still works for me. I didn't have to change the owner of the templates folder to "cloud" user as Kiran did inorder to work with the setup. On my setup: [root@asfmgmt 7200344900649.4910c1c6] # ls -lh total 32K drwxrwxrwx. 3 root root 4.0K Mar 13 16:07 snapshots drwxrwxr-x. 2 cloud cloud 4.0K Mar 11 18:53 systemvm drwxr-xr-x. 3 root root 4.0K Sep 11 2012 template drwxrwxrwx. 2 root root 4.0K Mar 11 18:54 volumes [root@asfmgmt 7200344900649.4910c1c6] # mysql> select id,name,state from vm_instance where vm_type != "User"; --- ---------- -------- id name state --- ---------- -------- 1 s-1-VMSNAP Running 2 v-2-VMSNAP Running 4 r-4-VMSNAP Running --- ---------- -------- 3 rows in set (0.00 sec)
        Hide
        Sudha Ponnaganti added a comment -

        chandan is taking a look at kiran's setup as some are seeing this issue and some are not so need to figure out. But problem seem to be with the permission

        Show
        Sudha Ponnaganti added a comment - chandan is taking a look at kiran's setup as some are seeing this issue and some are not so need to figure out. But problem seem to be with the permission
        Hide
        Sudha Ponnaganti added a comment -

        Kiran,

        is this a different issue than originally logged??
        Can you close this one and log a new one.

        Should Pradeep be looking at this??

        thanks
        /sudha

        Show
        Sudha Ponnaganti added a comment - Kiran, is this a different issue than originally logged?? Can you close this one and log a new one. Should Pradeep be looking at this?? thanks /sudha
        Hide
        Kiran Koneti added a comment -

        The permission Issue is always seen in the VMware setup both in the Master and 4.1 even if we create it in the new machine(Freshly installed RHEL).Hence Re opening this issue.
        The folder "/var/cloudstack/mnt/VM/6950196609054.51e6a86f/ contains the template folder with the root as the owner and changing it to cloud should be done manually.This should be done Automatically while installing the CS.
        Once the owner permissions are changed the CS starts installing the System Vm's.

        Show
        Kiran Koneti added a comment - The permission Issue is always seen in the VMware setup both in the Master and 4.1 even if we create it in the new machine(Freshly installed RHEL).Hence Re opening this issue. The folder "/var/cloudstack/mnt/VM/6950196609054.51e6a86f/ contains the template folder with the root as the owner and changing it to cloud should be done manually.This should be done Automatically while installing the CS. Once the owner permissions are changed the CS starts installing the System Vm's.
        Hide
        Venkata Siva Vijayendra Bhamidipati added a comment -

        Identified as a setup issue.

        Show
        Venkata Siva Vijayendra Bhamidipati added a comment - Identified as a setup issue.
        Hide
        Venkata Siva Vijayendra Bhamidipati added a comment -

        Since the original issue of mnt creation has been gotten over and is a setup issue, and since Chandan isn't running into it, I'm closing this bug. We'll track the outstanding issues in a different bug or as part of 1252.

        Show
        Venkata Siva Vijayendra Bhamidipati added a comment - Since the original issue of mnt creation has been gotten over and is a setup issue, and since Chandan isn't running into it, I'm closing this bug. We'll track the outstanding issues in a different bug or as part of 1252.
        Hide
        Kiran Koneti added a comment -

        I tried using a new secondary Storage and deleted the Cloud user and installed the new setup.

        Then I observed the following error message.

        "2013-02-27 05:09:43,843 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs:daryStorage: template/tmpl/1/8/, templateName: routing-8
        2013-02-27 05:09:43,848 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Secondary storage mount point: /var/cloudstack/mnt/VM/6903220404277.
        2013-02-27 05:09:43,848 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Executing command: tar --no-same-owner -xf /var/cloudstack/mnt/VM/69
        2013-02-27 05:09:43,848 DEBUG [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Executing: tar --no-same-owner -xf /var/cloudstack/mnt/VM/6903220404
        2013-02-27 05:09:43,899 WARN [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Exception: tar --no-same-owner -xf /var/cloudstack/mnt/VM/6903220404
        java.io.IOException: Cannot run program "tar" (in directory "/var/cloudstack/mnt/VM/6903220404277.73e986c3/template/tmpl/1/8"): java.io.IOException: error=13, Permissio
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
        at com.cloud.utils.script.Script.execute(Script.java:183)
        at com.cloud.utils.script.Script.execute(Script.java:161)"

        Now I logged in the machine as cloud user and navigate to the path then I was able to navigate till the "/var/cloudstack/mnt/VM/7614406590488.1f2f172a" path but was not able to navigate to the template folder as the folder owner was root.
        Then I change the owner permissions to cloud and I haven't observed the issue with the permissions.

        Then I observed another error message related systemvm.iso this looks like the issue CS-1262.

        2013-02-27 05:20:14,580 DEBUG [utils.script.Script] (DirectAgent-4:10.147.40.8) environment.properties says scripts should be in /usr/share/cloudstack-common
        2013-02-27 05:20:14,580 DEBUG [utils.script.Script] (DirectAgent-4:10.147.40.8) Looking for vms/systemvm.iso in /usr/share/cloudstack-common/vms/systemvm.iso
        2013-02-27 05:20:14,580 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-4:10.147.40.8) Executing: /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso
        2013-02-27 05:20:14,703 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-4:10.147.40.8) Execution is successful.
        2013-02-27 05:20:14,709 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-4:10.147.40.8) Unexpected exception
        java.io.FileNotFoundException: /usr/lib/cloud/common/vms/systemvm.iso (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at com.cloud.utils.FileUtil.copyfile(FileUtil.java:68)
        at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:555)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)

        Here the systemvm.iso is copied to "/usr/share/cloudstack-common/vms/" but while fetching the iso it is looking for it in "/usr/lib/cloud/common/vms/"

        Then I manually copied the systemvm.iso to "/usr/lib/cloud/common/vms/" from the "/usr/share/cloudstack-common/vms/".Then the system VM's came up.

        But still I see the issue with the builtin template CLOUDSTACK-1252 .

        Show
        Kiran Koneti added a comment - I tried using a new secondary Storage and deleted the Cloud user and installed the new setup. Then I observed the following error message. "2013-02-27 05:09:43,843 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs:daryStorage: template/tmpl/1/8/, templateName: routing-8 2013-02-27 05:09:43,848 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Secondary storage mount point: /var/cloudstack/mnt/VM/6903220404277. 2013-02-27 05:09:43,848 INFO [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Executing command: tar --no-same-owner -xf /var/cloudstack/mnt/VM/69 2013-02-27 05:09:43,848 DEBUG [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Executing: tar --no-same-owner -xf /var/cloudstack/mnt/VM/6903220404 2013-02-27 05:09:43,899 WARN [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-4:10.147.40.8) Exception: tar --no-same-owner -xf /var/cloudstack/mnt/VM/6903220404 java.io.IOException: Cannot run program "tar" (in directory "/var/cloudstack/mnt/VM/6903220404277.73e986c3/template/tmpl/1/8"): java.io.IOException: error=13, Permissio at java.lang.ProcessBuilder.start(ProcessBuilder.java:475) at com.cloud.utils.script.Script.execute(Script.java:183) at com.cloud.utils.script.Script.execute(Script.java:161)" Now I logged in the machine as cloud user and navigate to the path then I was able to navigate till the "/var/cloudstack/mnt/VM/7614406590488.1f2f172a" path but was not able to navigate to the template folder as the folder owner was root. Then I change the owner permissions to cloud and I haven't observed the issue with the permissions. Then I observed another error message related systemvm.iso this looks like the issue CS-1262. 2013-02-27 05:20:14,580 DEBUG [utils.script.Script] (DirectAgent-4:10.147.40.8) environment.properties says scripts should be in /usr/share/cloudstack-common 2013-02-27 05:20:14,580 DEBUG [utils.script.Script] (DirectAgent-4:10.147.40.8) Looking for vms/systemvm.iso in /usr/share/cloudstack-common/vms/systemvm.iso 2013-02-27 05:20:14,580 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-4:10.147.40.8) Executing: /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso 2013-02-27 05:20:14,703 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-4:10.147.40.8) Execution is successful. 2013-02-27 05:20:14,709 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-4:10.147.40.8) Unexpected exception java.io.FileNotFoundException: /usr/lib/cloud/common/vms/systemvm.iso (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:137) at com.cloud.utils.FileUtil.copyfile(FileUtil.java:68) at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:555) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) Here the systemvm.iso is copied to "/usr/share/cloudstack-common/vms/" but while fetching the iso it is looking for it in "/usr/lib/cloud/common/vms/" Then I manually copied the systemvm.iso to "/usr/lib/cloud/common/vms/" from the "/usr/share/cloudstack-common/vms/".Then the system VM's came up. But still I see the issue with the builtin template CLOUDSTACK-1252 .
        Hide
        Sangeetha Hariharan added a comment -

        In my case , I delete the existing cloud user in my management sever:

        userdel cloud

        Then attempted a fresh installation.
        I did not face issue with Systemvm keypairs generation anymore.

        Kiran , you could try deleting the cloud user and attempting a fresh install on the same box to see if this issue you are facing is resolved.

        Note - I was testing a KVM set up.

        Show
        Sangeetha Hariharan added a comment - In my case , I delete the existing cloud user in my management sever: userdel cloud Then attempted a fresh installation. I did not face issue with Systemvm keypairs generation anymore. Kiran , you could try deleting the cloud user and attempting a fresh install on the same box to see if this issue you are facing is resolved. Note - I was testing a KVM set up.
        Hide
        Chandan Purushothama added a comment -

        I used a brand new rhel 6.3 VM and deployed my management server on it.

        Kiran, did you use the RHEL 6.3 server on which your ASF management server is currently running for older release builds?

        Show
        Chandan Purushothama added a comment - I used a brand new rhel 6.3 VM and deployed my management server on it. Kiran, did you use the RHEL 6.3 server on which your ASF management server is currently running for older release builds?
        Hide
        Chandan Purushothama added a comment -

        I am able to successfully deploy a VMWare Setup with System VMs running on them.

        Show
        Chandan Purushothama added a comment - I am able to successfully deploy a VMWare Setup with System VMs running on them.
        Hide
        Venkata Siva Vijayendra Bhamidipati added a comment -

        Sangeetha just spoke with me regarding this bug. She pointed out that packaging scripts being used here in the QA teams have been changed recently to use /var/cloudstack/ instead of /var/lib/cloud/.

        On Kiran's server, I do see:

        [root@KiranRhel63-3 vijay]# ls /var/cloudstack/mnt/VM/
        7614406590488.32f8324
        [root@KiranRhel63-3 vijay]#

        So it looks like something needs to be additionally changed in the QA packaging scripts.

        Sangeetha has a workaround for this - she will update this bug with the workaround. I'm pasting the stack trace she ran into with ssh keypairs below - but I don't see a similar issue in Krian's server's logs - probably, Kiran missed modifying a part of the scripts.

        013-02-26 11:52:53,705 INFO [cloud.server.ConfigurationServerImpl] (Timer-1:null) Stored SSL keystore to database.
        2013-02-26 11:52:53,707 INFO [cloud.server.ConfigurationServerImpl] (Timer-1:null) Processing updateKeyPairs
        2013-02-26 11:52:53,708 INFO [cloud.server.ConfigurationServerImpl] (Timer-1:null) Systemvm keypairs not found in database. Need to store them in the database
        2013-02-26 11:52:53,708 DEBUG [utils.script.Script] (Timer-1:null) Executing: /bin/bash -c if [ -f /var/lib/cloud/management/.ssh/id_rsa ]; then rm -f /var/lib/cloud/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/lib/cloud/management/.ssh/id_rsa -q
        2013-02-26 11:52:54,335 DEBUG [utils.script.Script] (Timer-1:null) Exit value is 1
        2013-02-26 11:52:54,336 DEBUG [utils.script.Script] (Timer-1:null) Could not create directory '/var/lib/cloud/management/.ssh'.open /var/lib/cloud/management/.ssh/id_rsa failed: No such file or directory.Saving the key failed: /var/lib/cloud/management/.ssh/id_rsa.
        2013-02-26 11:52:54,337 ERROR [cloud.server.ConfigurationServerImpl] (Timer-1:null) Cannot read the private key file
        java.io.FileNotFoundException: /var/lib/cloud/management/.ssh/id_rsa (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at com.cloud.server.ConfigurationServerImpl.updateKeyPairs(ConfigurationServerImpl.java:617)
        at com.cloud.server.ConfigurationServerImpl.persistDefaultValues(ConfigurationServerImpl.java:275)
        at com.cloud.server.ConfigurationServerImpl.configure(ConfigurationServerImpl.java:144)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
        at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at $Proxy233.configure(Unknown Source)
        at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:110)
        at com.cloud.servlet.CloudStartupServlet$1.run(CloudStartupServlet.java:50)
        at java.util.TimerThread.mainLoop(Timer.java:534)
        at java.util.TimerThread.run(Timer.java:484)

        Show
        Venkata Siva Vijayendra Bhamidipati added a comment - Sangeetha just spoke with me regarding this bug. She pointed out that packaging scripts being used here in the QA teams have been changed recently to use /var/cloudstack/ instead of /var/lib/cloud/. On Kiran's server, I do see: [root@KiranRhel63-3 vijay] # ls /var/cloudstack/mnt/VM/ 7614406590488.32f8324 [root@KiranRhel63-3 vijay] # So it looks like something needs to be additionally changed in the QA packaging scripts. Sangeetha has a workaround for this - she will update this bug with the workaround. I'm pasting the stack trace she ran into with ssh keypairs below - but I don't see a similar issue in Krian's server's logs - probably, Kiran missed modifying a part of the scripts. 013-02-26 11:52:53,705 INFO [cloud.server.ConfigurationServerImpl] (Timer-1:null) Stored SSL keystore to database. 2013-02-26 11:52:53,707 INFO [cloud.server.ConfigurationServerImpl] (Timer-1:null) Processing updateKeyPairs 2013-02-26 11:52:53,708 INFO [cloud.server.ConfigurationServerImpl] (Timer-1:null) Systemvm keypairs not found in database. Need to store them in the database 2013-02-26 11:52:53,708 DEBUG [utils.script.Script] (Timer-1:null) Executing: /bin/bash -c if [ -f /var/lib/cloud/management/.ssh/id_rsa ]; then rm -f /var/lib/cloud/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/lib/cloud/management/.ssh/id_rsa -q 2013-02-26 11:52:54,335 DEBUG [utils.script.Script] (Timer-1:null) Exit value is 1 2013-02-26 11:52:54,336 DEBUG [utils.script.Script] (Timer-1:null) Could not create directory '/var/lib/cloud/management/.ssh'.open /var/lib/cloud/management/.ssh/id_rsa failed: No such file or directory.Saving the key failed: /var/lib/cloud/management/.ssh/id_rsa. 2013-02-26 11:52:54,337 ERROR [cloud.server.ConfigurationServerImpl] (Timer-1:null) Cannot read the private key file java.io.FileNotFoundException: /var/lib/cloud/management/.ssh/id_rsa (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:137) at com.cloud.server.ConfigurationServerImpl.updateKeyPairs(ConfigurationServerImpl.java:617) at com.cloud.server.ConfigurationServerImpl.persistDefaultValues(ConfigurationServerImpl.java:275) at com.cloud.server.ConfigurationServerImpl.configure(ConfigurationServerImpl.java:144) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy233.configure(Unknown Source) at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:110) at com.cloud.servlet.CloudStartupServlet$1.run(CloudStartupServlet.java:50) at java.util.TimerThread.mainLoop(Timer.java:534) at java.util.TimerThread.run(Timer.java:484)
        Hide
        Venkata Siva Vijayendra Bhamidipati added a comment -

        Hi Kiran,

        Anthony and I logged into 10.147.38.142 and took a look at the mgmt server process and the directory structure.

        [root@KiranRhel63-3 init.d]# service cloudstack-management status
        cloudstack-management (pid 2941) is running...
        [root@KiranRhel63-3 init.d]#

        [root@KiranRhel63-3 init.d]# ps -ef | grep 2941
        cloud 2941 1 3 Feb22 ? 03:59:27 /usr/lib/jvm/jre/bin/java -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=45219 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M -XX:MaxPermSize=800m -classpath :::/etc/cloudstack/management:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management/bin/bootstrap.jar:/usr/share/cloudstack-management/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/cloudstack-management -Dcatalina.home=/usr/share/cloudstack-management -Djava.endorsed.dirs= -Djava.io.tmpdir=/usr/share/cloudstack-management/temp -Djava.util.logging.config.file=/usr/share/cloudstack-management/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
        root 28753 28547 0 12:12 pts/1 00:00:00 grep 2941
        [root@KiranRhel63-3 init.d]#

        The mgmt server is running under the user "cloud".

        Next, logged in as cloud user:

        [root@KiranRhel63-3 init.d]# su - cloud
        -sh-4.1$

        -sh-4.1$ ls -ld /var/lib/cloud/mnt/
        drwxr-xr-x. 3 cloud cloud 4096 Feb 27 12:06 /var/lib/cloud/mnt/
        -sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/
        drwxr-xr-x. 3 root root 4096 Feb 27 11:56 /var/lib/cloud/mnt/VM/
        -sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/7313725325399.503a5754/
        drwxr-xr-x. 2 root root 4096 Feb 27 11:56 /var/lib/cloud/mnt/VM/7313725325399.503a5754/
        -sh-4.1$

        Things looked fine till now, but when we tried:

        -sh-4.1$ mkdir -p /var/lib/cloud/mnt/VM/7313725325399.503a5767
        mkdir: cannot create directory `/var/lib/cloud/mnt/VM/7313725325399.503a5767': Permission denied
        -sh-4.1$

        We just changed the last two digits to create a new directory, but are hitting a permission denied error. Checked whether /var/lib/cloud/mnt/VM or any of the directories in the chain was already mounted, and they weren't:

        -sh-4.1$ df -kh
        Filesystem Size Used Avail Use% Mounted on
        /dev/mapper/vg_kiranrhel633-lv_root
        18G 7.8G 8.6G 48% /
        tmpfs 499M 0 499M 0% /dev/shm
        /dev/vda1 485M 33M 428M 8% /boot
        /mnt/ISO/rhel-server-6.3-x86_64-dvd.iso
        3.5G 3.5G 0 100% /media
        10.147.28.7:/export/home/kiran
        5.4T 2.5T 3.0T 46% /tmp/sec
        10.147.28.7:/export/home/kiran/secondary
        5.4T 2.5T 3.0T 46% /var/cloudstack/mnt/VM/7614406590488.32f8324
        -sh-4.1$

        I am not sure what is causing this issue on your server, but it looks like a setup issue. The mgmt server isn't able to create the mount point under /var/lib/cloud/mnt/ and even manual creation is failing.

        The workaround I can think of to immediately unblock this is for you to create a /mnt/sec directory as the cloud user, and then mount the nfs secondary directory to /mnt/sec/, and see how it goes. I believe the mgmt server will proceed beyond the current point of failure then.

        Also, can you please confirm what happens when you run the mgmt server as root? Do you still see the above issue? I'm guessing that you won't. I know that you should be able to run the mgmt server as the cloud user - I'm just asking about root to see if it can unblock your testing user independent functionality. In any case, you must look at why the cloud user cannot create directories in a directory chain owned by it.

        Regards,
        Vijay

        Show
        Venkata Siva Vijayendra Bhamidipati added a comment - Hi Kiran, Anthony and I logged into 10.147.38.142 and took a look at the mgmt server process and the directory structure. [root@KiranRhel63-3 init.d] # service cloudstack-management status cloudstack-management (pid 2941) is running... [root@KiranRhel63-3 init.d] # [root@KiranRhel63-3 init.d] # ps -ef | grep 2941 cloud 2941 1 3 Feb22 ? 03:59:27 /usr/lib/jvm/jre/bin/java -Djava.awt.headless=true -Dcom.sun.management.jmxremote.port=45219 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M -XX:MaxPermSize=800m -classpath :::/etc/cloudstack/management:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management/bin/bootstrap.jar:/usr/share/cloudstack-management/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/cloudstack-management -Dcatalina.home=/usr/share/cloudstack-management -Djava.endorsed.dirs= -Djava.io.tmpdir=/usr/share/cloudstack-management/temp -Djava.util.logging.config.file=/usr/share/cloudstack-management/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start root 28753 28547 0 12:12 pts/1 00:00:00 grep 2941 [root@KiranRhel63-3 init.d] # The mgmt server is running under the user "cloud". Next, logged in as cloud user: [root@KiranRhel63-3 init.d] # su - cloud -sh-4.1$ -sh-4.1$ ls -ld /var/lib/cloud/mnt/ drwxr-xr-x. 3 cloud cloud 4096 Feb 27 12:06 /var/lib/cloud/mnt/ -sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/ drwxr-xr-x. 3 root root 4096 Feb 27 11:56 /var/lib/cloud/mnt/VM/ -sh-4.1$ ls -ld /var/lib/cloud/mnt/VM/7313725325399.503a5754/ drwxr-xr-x. 2 root root 4096 Feb 27 11:56 /var/lib/cloud/mnt/VM/7313725325399.503a5754/ -sh-4.1$ Things looked fine till now, but when we tried: -sh-4.1$ mkdir -p /var/lib/cloud/mnt/VM/7313725325399.503a5767 mkdir: cannot create directory `/var/lib/cloud/mnt/VM/7313725325399.503a5767': Permission denied -sh-4.1$ We just changed the last two digits to create a new directory, but are hitting a permission denied error. Checked whether /var/lib/cloud/mnt/VM or any of the directories in the chain was already mounted, and they weren't: -sh-4.1$ df -kh Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_kiranrhel633-lv_root 18G 7.8G 8.6G 48% / tmpfs 499M 0 499M 0% /dev/shm /dev/vda1 485M 33M 428M 8% /boot /mnt/ISO/rhel-server-6.3-x86_64-dvd.iso 3.5G 3.5G 0 100% /media 10.147.28.7:/export/home/kiran 5.4T 2.5T 3.0T 46% /tmp/sec 10.147.28.7:/export/home/kiran/secondary 5.4T 2.5T 3.0T 46% /var/cloudstack/mnt/VM/7614406590488.32f8324 -sh-4.1$ I am not sure what is causing this issue on your server, but it looks like a setup issue. The mgmt server isn't able to create the mount point under /var/lib/cloud/mnt/ and even manual creation is failing. The workaround I can think of to immediately unblock this is for you to create a /mnt/sec directory as the cloud user, and then mount the nfs secondary directory to /mnt/sec/, and see how it goes. I believe the mgmt server will proceed beyond the current point of failure then. Also, can you please confirm what happens when you run the mgmt server as root? Do you still see the above issue? I'm guessing that you won't. I know that you should be able to run the mgmt server as the cloud user - I'm just asking about root to see if it can unblock your testing user independent functionality. In any case, you must look at why the cloud user cannot create directories in a directory chain owned by it. Regards, Vijay
        Hide
        prashant kumar mishra added a comment -

        i changed permission and manually mounted .. then ran into another exception;
        Checked in vcenter server around 12 system vms got created(all are in power off state) within 30 minute , and CS keep deploying new system vms as time passes.CS UI don't show them but you can see them in vCenter server.

        snippet of log
        -------------------
        ------------------

        013-02-26 07:50:25,006 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) Search file ROOT-15-15.vmdk on [d65fe2b3054b37289a07f4af32167d63]
        2013-02-26 07:50:26,304 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) File [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15.vmdk does not exist on datastore
        2013-02-26 07:50:26,519 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) Search file ROOT-15-15-flat.vmdk on [d65fe2b3054b37289a07f4af32167d63]
        2013-02-26 07:50:27,832 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) File [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15-flat.vmdk does not exist on datastore
        2013-02-26 07:50:28,044 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) Search file ROOT-15-15-delta.vmdk on [d65fe2b3054b37289a07f4af32167d63]
        2013-02-26 07:50:29,358 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) File [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15-delta.vmdk does not exist on datastore
        2013-02-26 07:50:32,115 INFO [vmware.resource.VmwareResource] (DirectAgent-36:10.147.40.8) Destroy dummy VM after volume creation
        2013-02-26 07:50:34,015 ERROR [vmware.resource.VmwareResource] (DirectAgent-36:10.147.40.8) CreateCommand failed due to Exception: java.lang.Exception
        Message: Unable to create disk [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15.vmdk due to Invalid configuration for device '0'. com.vmware.vim25.InvalidDeviceSpec

        java.lang.Exception: Unable to create disk [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15.vmdk due to Invalid configuration for device '0'. com.vmware.vim25.InvalidDeviceSpec
        at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createDisk(VirtualMachineMO.java:937)
        at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createDisk(VirtualMachineMO.java:855)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3915)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:342)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
        2013-02-26 07:50:34,018 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-36:null) Seq 1-745013312: Response Received:

        ==========
        2
        ==========
        2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso
        2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Current binaries reside at /usr/share/cloudstack-management/webapps/client/WEB-INF/lib
        2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/vms/systemvm.iso
        2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/vms/systemvm.iso
        2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/WEB-INF/vms/systemvm.iso
        2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/vms/systemvm.iso
        2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/vms/systemvm.iso
        2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/vms/systemvm.iso
        2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/vms/systemvm.iso
        2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/vms/systemvm.iso
        2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /vms/systemvm.iso
        2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Searching in environment.properties
        2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) environment.properties says scripts should be in /usr/share/cloudstack-common
        2013-02-26 10:12:31,489 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-common/vms/systemvm.iso
        2013-02-26 10:12:31,489 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-15:10.147.40.7) Executing: /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso
        2013-02-26 10:12:31,599 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-15:10.147.40.7) Execution is successful.
        2013-02-26 10:12:31,600 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-15:10.147.40.7) Unexpected exception
        java.io.FileNotFoundException: /usr/lib/cloud/common/vms/systemvm.iso (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at com.cloud.utils.FileUtil.copyfile(FileUtil.java:68)
        at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:555)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:43)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2145)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:426)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
        2013-02-26 10:12:31,607 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-15:10.147.40.7) Unable to copy systemvm ISO on secondary storage. src location: /usr/lib/cloud/common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/6903220404277.6defdeab/systemvm/systemvm-4.2.0-SNAPSHOT.iso
        2013-02-26 10:12:31,609 WARN [vmware.resource.VmwareResource] (DirectAgent-15:10.147.40.7) StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
        Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/lib/cloud/common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/6903220404277.6defdeab/systemvm/systemvm-4.2.0-SNAPSHOT.iso

        com.cloud.utils.exception.CloudRuntimeException: Unable to copy systemvm ISO on secondary storage. src location: /usr/lib/cloud/common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/6903220404277.6defdeab/systemvm/systemvm-4.2.0-SNAPSHOT.iso
        at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:561)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:43)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2145)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:426)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
        2013-02-26 10:12:31,616 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-15:null) Seq 1-845021211: Cancelling because one of the answers is false and it is stop on error.

        Show
        prashant kumar mishra added a comment - i changed permission and manually mounted .. then ran into another exception; Checked in vcenter server around 12 system vms got created(all are in power off state) within 30 minute , and CS keep deploying new system vms as time passes.CS UI don't show them but you can see them in vCenter server. snippet of log ------------------- ------------------ 013-02-26 07:50:25,006 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) Search file ROOT-15-15.vmdk on [d65fe2b3054b37289a07f4af32167d63] 2013-02-26 07:50:26,304 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) File [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15.vmdk does not exist on datastore 2013-02-26 07:50:26,519 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) Search file ROOT-15-15-flat.vmdk on [d65fe2b3054b37289a07f4af32167d63] 2013-02-26 07:50:27,832 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) File [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15-flat.vmdk does not exist on datastore 2013-02-26 07:50:28,044 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) Search file ROOT-15-15-delta.vmdk on [d65fe2b3054b37289a07f4af32167d63] 2013-02-26 07:50:29,358 INFO [vmware.mo.DatastoreMO] (DirectAgent-36:10.147.40.8) File [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15-delta.vmdk does not exist on datastore 2013-02-26 07:50:32,115 INFO [vmware.resource.VmwareResource] (DirectAgent-36:10.147.40.8) Destroy dummy VM after volume creation 2013-02-26 07:50:34,015 ERROR [vmware.resource.VmwareResource] (DirectAgent-36:10.147.40.8) CreateCommand failed due to Exception: java.lang.Exception Message: Unable to create disk [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15.vmdk due to Invalid configuration for device '0'. com.vmware.vim25.InvalidDeviceSpec java.lang.Exception: Unable to create disk [d65fe2b3054b37289a07f4af32167d63] ROOT-15-15.vmdk due to Invalid configuration for device '0'. com.vmware.vim25.InvalidDeviceSpec at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createDisk(VirtualMachineMO.java:937) at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createDisk(VirtualMachineMO.java:855) at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3915) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:342) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 2013-02-26 07:50:34,018 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-36:null) Seq 1-745013312: Response Received: ========== 2 ========== 2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso 2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Current binaries reside at /usr/share/cloudstack-management/webapps/client/WEB-INF/lib 2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/vms/systemvm.iso 2013-02-26 10:12:31,486 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/WEB-INF/lib/vms/systemvm.iso 2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/WEB-INF/vms/systemvm.iso 2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/client/vms/systemvm.iso 2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/webapps/vms/systemvm.iso 2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-management/vms/systemvm.iso 2013-02-26 10:12:31,487 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/vms/systemvm.iso 2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/vms/systemvm.iso 2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /vms/systemvm.iso 2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Searching in environment.properties 2013-02-26 10:12:31,488 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) environment.properties says scripts should be in /usr/share/cloudstack-common 2013-02-26 10:12:31,489 DEBUG [utils.script.Script] (DirectAgent-15:10.147.40.7) Looking for vms/systemvm.iso in /usr/share/cloudstack-common/vms/systemvm.iso 2013-02-26 10:12:31,489 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-15:10.147.40.7) Executing: /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso 2013-02-26 10:12:31,599 DEBUG [cloud.server.ConfigurationServerImpl] (DirectAgent-15:10.147.40.7) Execution is successful. 2013-02-26 10:12:31,600 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-15:10.147.40.7) Unexpected exception java.io.FileNotFoundException: /usr/lib/cloud/common/vms/systemvm.iso (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:137) at com.cloud.utils.FileUtil.copyfile(FileUtil.java:68) at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:555) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:43) at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2145) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:426) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 2013-02-26 10:12:31,607 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-15:10.147.40.7) Unable to copy systemvm ISO on secondary storage. src location: /usr/lib/cloud/common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/6903220404277.6defdeab/systemvm/systemvm-4.2.0-SNAPSHOT.iso 2013-02-26 10:12:31,609 WARN [vmware.resource.VmwareResource] (DirectAgent-15:10.147.40.7) StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/lib/cloud/common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/6903220404277.6defdeab/systemvm/systemvm-4.2.0-SNAPSHOT.iso com.cloud.utils.exception.CloudRuntimeException: Unable to copy systemvm ISO on secondary storage. src location: /usr/lib/cloud/common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/6903220404277.6defdeab/systemvm/systemvm-4.2.0-SNAPSHOT.iso at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:561) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:43) at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2145) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:426) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 2013-02-26 10:12:31,616 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-15:null) Seq 1-845021211: Cancelling because one of the answers is false and it is stop on error.
        Hide
        prashant kumar mishra added a comment -

        i am also facing same issue with the ASF master build

        Show
        prashant kumar mishra added a comment - i am also facing same issue with the ASF master build
        Hide
        Kiran Koneti added a comment -

        Observed the same issue with the Master Build(4.2) also.It is having the same error message.

        Show
        Kiran Koneti added a comment - Observed the same issue with the Master Build(4.2) also.It is having the same error message.
        Hide
        Animesh Chaturvedi added a comment -

        Vijay, please check Kiran / Prashanth's setup.

        Show
        Animesh Chaturvedi added a comment - Vijay, please check Kiran / Prashanth's setup.
        Hide
        Kiran Koneti added a comment -

        Hi Vijay,

        I tried it with a newly deployed machine and observed the same error message.Not only me even Prashanth Mishra tried it in a different setup.the umask value of the machines are 0022 and we were manually able to create the directory "/var/lib/cloud/mnt/ " with root user and even tried it again by changing the owner permission to cloud as rayees suggested but nothing worked.

        Regards,
        Kiran.

        Show
        Kiran Koneti added a comment - Hi Vijay, I tried it with a newly deployed machine and observed the same error message.Not only me even Prashanth Mishra tried it in a different setup.the umask value of the machines are 0022 and we were manually able to create the directory "/var/lib/cloud/mnt/ " with root user and even tried it again by changing the owner permission to cloud as rayees suggested but nothing worked. Regards, Kiran.
        Hide
        Rayees Namathponnan added a comment -

        This issue observed in Kiran's setup

        Show
        Rayees Namathponnan added a comment - This issue observed in Kiran's setup
        Hide
        Animesh Chaturvedi added a comment -

        Vijay

        Did you check Rayees's setup? Does Rayees setup also has this issue?

        Show
        Animesh Chaturvedi added a comment - Vijay Did you check Rayees's setup? Does Rayees setup also has this issue?
        Hide
        Fang Wang added a comment -

        is there a limit set on the path length? I am wondering if the /mnt/secondary got chopped.

        Show
        Fang Wang added a comment - is there a limit set on the path length? I am wondering if the /mnt/secondary got chopped.
        Hide
        Venkata Siva Vijayendra Bhamidipati added a comment -

        Hi Kiran,

        The mgmt server is attempting to do an mkdir /var/lib/cloud/mnt/VM/7313725325399.47bfef59 and that is failing for some reason. The mgmt server doesn't log that reason currently. If it fails, the mgmt server code returns a default value of /mnt/sec/ instead. I don't know why it should be doing that. Ideally it should throw an exception and cry foul. Going by the CS installation guide, it should atleast return /mnt/secondary/ and not /mnt/sec/.

        Anyway, if the creation of the /var/lib/cloud/mnt/VM directory had gone through correctly, the mgmt server then would have basically mounted the remote NFS mount point (where the ova files exist) to this directory, and would pick up the system template from there. If it fails to locate that system template ova file (which it definitely will fail to because the /var/lib/cloud/mnt/VM directory never got created, so /var/lib/cloud/mnt/VM/xyz/template/tmpl/1/8/<uuidxyz>.ova would not be found), it defaults the template name to routing-8.ova. So looking for a file by name routing-8.ova is futile - it won't exist unless we copy over the <longname>.ova system template under /mnt/sec/template/tmpl/1/8/ as routing-8.ova.

        In this case, it's trying to pick it up from /mnt/sec/ which itself doesn't exist and so that will definitely fail anyway.

        In short the root cause is that the secondary mount point is not getting created on the mgmt server node. That needs to be investigated. It does not look like a code issue to me at all - it's most likely some config issue on the mgmt server box.

        So -

        1. Why did the mkdir /var/lib/cloud/mnt/ fail? I strongly suggest starting with a fresh setup like Pradeep suggested, and as root user, checking whether all the nfs settings/options have been set correctly as described in the CS installation guide, with umask set such that umask -S shows: u=rwx,g=rx,o=rx (so set it to 0022), checking whether an mkdir of this directory (/var/lib/cloud/mnt/) by the root user works, and if mounting the nfs directory on this mount point works like suggested by Sateesh, and then removing this directory and starting up the mgmt server to see whether it can create the directory.
        2. Why is the mgmt server returning /mnt/sec? I'd remove that. I don't see how it can be of any use. Anyway this won't resolve this issue and isn't a cause. It's a side issue.

        Please try out the above and let me know how it goes.

        Regards,
        Vijay

        Show
        Venkata Siva Vijayendra Bhamidipati added a comment - Hi Kiran, The mgmt server is attempting to do an mkdir /var/lib/cloud/mnt/VM/7313725325399.47bfef59 and that is failing for some reason. The mgmt server doesn't log that reason currently. If it fails, the mgmt server code returns a default value of /mnt/sec/ instead. I don't know why it should be doing that. Ideally it should throw an exception and cry foul. Going by the CS installation guide, it should atleast return /mnt/secondary/ and not /mnt/sec/. Anyway, if the creation of the /var/lib/cloud/mnt/VM directory had gone through correctly, the mgmt server then would have basically mounted the remote NFS mount point (where the ova files exist) to this directory, and would pick up the system template from there. If it fails to locate that system template ova file (which it definitely will fail to because the /var/lib/cloud/mnt/VM directory never got created, so /var/lib/cloud/mnt/VM/xyz/template/tmpl/1/8/<uuidxyz>.ova would not be found), it defaults the template name to routing-8.ova. So looking for a file by name routing-8.ova is futile - it won't exist unless we copy over the <longname>.ova system template under /mnt/sec/template/tmpl/1/8/ as routing-8.ova. In this case, it's trying to pick it up from /mnt/sec/ which itself doesn't exist and so that will definitely fail anyway. In short the root cause is that the secondary mount point is not getting created on the mgmt server node. That needs to be investigated. It does not look like a code issue to me at all - it's most likely some config issue on the mgmt server box. So - 1. Why did the mkdir /var/lib/cloud/mnt/ fail? I strongly suggest starting with a fresh setup like Pradeep suggested, and as root user, checking whether all the nfs settings/options have been set correctly as described in the CS installation guide, with umask set such that umask -S shows: u=rwx,g=rx,o=rx (so set it to 0022), checking whether an mkdir of this directory (/var/lib/cloud/mnt/) by the root user works, and if mounting the nfs directory on this mount point works like suggested by Sateesh, and then removing this directory and starting up the mgmt server to see whether it can create the directory. 2. Why is the mgmt server returning /mnt/sec? I'd remove that. I don't see how it can be of any use. Anyway this won't resolve this issue and isn't a cause. It's a side issue. Please try out the above and let me know how it goes. Regards, Vijay
        Hide
        Sudha Ponnaganti added a comment -

        Vijay,

        Rayees can help US timezone if you need help

        thanks
        /sudha

        Show
        Sudha Ponnaganti added a comment - Vijay, Rayees can help US timezone if you need help thanks /sudha
        Hide
        Chip Childers added a comment -

        Venkata,

        Can you perhaps collaborate in real-time (sit down with or chat with) Kiran to get this sorted out?

        Show
        Chip Childers added a comment - Venkata, Can you perhaps collaborate in real-time (sit down with or chat with) Kiran to get this sorted out?
        Hide
        Kiran Koneti added a comment -

        The problem here is the specified directory ("/var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8") there is no file with the name "routing-8.ova" So the error message says it is not able to tar that particular file.

        I suspect that the problem is with this particular file either this file is missing in the tempalte which we seed in the secondary storage or else it is looking for a wrong file.

        Show
        Kiran Koneti added a comment - The problem here is the specified directory ("/var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8") there is no file with the name "routing-8.ova" So the error message says it is not able to tar that particular file. I suspect that the problem is with this particular file either this file is missing in the tempalte which we seed in the secondary storage or else it is looking for a wrong file.
        Hide
        Pradeep Soundararajan added a comment -

        commit 5594b1d622adb07203d4549aee426268ecee092d
        There are no more mount issues, i guess it has been resolved with the below commit already or by the workaround. Any way it has to be tested separately on a fresh setup:

        Author: Hugo Trippaers <htrippaers@schubergphilis.com>
        Date: Tue Feb 12 16:16:36 2013 +0100

        The new error messages were observerd. We have pointed this already. Please let us know what are these?

        "java.io.IOException: Cannot run program "tar" (in directory "/var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8"): java.io.IOException: error=13, Permission denied
        ......
        ......
        2013-02-18 21:30:39,800 ERROR [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-1:10.147.40.18) Unable to unpack snapshot OVA file at: /var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8/routing-8.ova

        The new error messages were observerd.

        Show
        Pradeep Soundararajan added a comment - commit 5594b1d622adb07203d4549aee426268ecee092d There are no more mount issues, i guess it has been resolved with the below commit already or by the workaround. Any way it has to be tested separately on a fresh setup: Author: Hugo Trippaers <htrippaers@schubergphilis.com> Date: Tue Feb 12 16:16:36 2013 +0100 The new error messages were observerd. We have pointed this already. Please let us know what are these? "java.io.IOException: Cannot run program "tar" (in directory "/var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8"): java.io.IOException: error=13, Permission denied ...... ...... 2013-02-18 21:30:39,800 ERROR [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-1:10.147.40.18) Unable to unpack snapshot OVA file at: /var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8/routing-8.ova The new error messages were observerd.
        Hide
        Kiran Koneti added a comment -

        I created the folder as rayees suggested and changed the owner to cloud user but still observe the same issue.The work around didnt work in my setup.below mentioned are the details about the same.

        [root@Kiran-Rhel63 cloud]# pwd
        /var/lib/cloud
        [root@Kiran-Rhel63 cloud]# ls -l
        total 8
        drwxr-xr-x. 4 cloud cloud 4096 Feb 11 20:26 management
        drwxr-xr-x. 2 cloud root 4096 Feb 18 19:59 mnt

        And regading the umask my system umask is 0022.

        [root@Kiran-Rhel63 cloud]# umask
        0022

        Please resolve this ASAP as this is blocking the VMware cases.

        Show
        Kiran Koneti added a comment - I created the folder as rayees suggested and changed the owner to cloud user but still observe the same issue.The work around didnt work in my setup.below mentioned are the details about the same. [root@Kiran-Rhel63 cloud] # pwd /var/lib/cloud [root@Kiran-Rhel63 cloud] # ls -l total 8 drwxr-xr-x. 4 cloud cloud 4096 Feb 11 20:26 management drwxr-xr-x. 2 cloud root 4096 Feb 18 19:59 mnt And regading the umask my system umask is 0022. [root@Kiran-Rhel63 cloud] # umask 0022 Please resolve this ASAP as this is blocking the VMware cases.
        Hide
        Rayees Namathponnan added a comment -

        mount point "/var/lib/cloud/mnt" was not available in my machine, then i created this manually and set owner is "cloud"

        after this i didn't faced the issue reported by Kiran

        Show
        Rayees Namathponnan added a comment - mount point "/var/lib/cloud/mnt" was not available in my machine, then i created this manually and set owner is "cloud" after this i didn't faced the issue reported by Kiran
        Hide
        Venkata Siva Vijayendra Bhamidipati added a comment -

        How was the mounting issue resolved? What is the umask on your system? Can you please set it to 0022 if it isn't so and retry?

        Show
        Venkata Siva Vijayendra Bhamidipati added a comment - How was the mounting issue resolved? What is the umask on your system? Can you please set it to 0022 if it isn't so and retry?
        Hide
        Animesh Chaturvedi added a comment -

        Since we have a workaround pointed by Rayees why should this be a blocker?

        Show
        Animesh Chaturvedi added a comment - Since we have a workaround pointed by Rayees why should this be a blocker?
        Hide
        Rohit Yadav added a comment -

        Assigning to Vijay, QA suggests that template for vmware was changed. If not, pl. assign to someone who would know vmware systemvm workflow better.

        Show
        Rohit Yadav added a comment - Assigning to Vijay, QA suggests that template for vmware was changed. If not, pl. assign to someone who would know vmware systemvm workflow better.
        Hide
        Pradeep Soundararajan added a comment -

        Mounting issues have been resolved. Right now it is a different issue, the issue seems to be it is not finding routing-8.ova.

        Whether template-simulator.sql has to be updated to look for routing-8?

        Show
        Pradeep Soundararajan added a comment - Mounting issues have been resolved. Right now it is a different issue, the issue seems to be it is not finding routing-8.ova. Whether template-simulator.sql has to be updated to look for routing-8?
        Hide
        Kiran Koneti added a comment -

        Attached the new Management server Logs.New error message is found

        "java.io.IOException: Cannot run program "tar" (in directory "/var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8"): java.io.IOException: error=13, Permission denied
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
        at com.cloud.utils.script.Script.execute(Script.java:183)
        at com.cloud.utils.script.Script.execute(Script.java:161)
        at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:443)
        at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
        Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
        at java.lang.ProcessImpl.start(ProcessImpl.java:81)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
        ... 15 more
        2013-02-18 21:30:39,800 ERROR [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-1:10.147.40.18) Unable to unpack snapshot OVA file at: /var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8/routing-8.ova
        2013-02-18 21:30:39,801 ERROR [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-1:10.147.40.18) Unable to execute PrimaryStorageDownloadCommand due to exception
        java.lang.Exception: Unable to unpack snapshot OVA file at: /var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8/routing-8.ova
        at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:447)
        at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)"

        Show
        Kiran Koneti added a comment - Attached the new Management server Logs.New error message is found "java.io.IOException: Cannot run program "tar" (in directory "/var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8"): java.io.IOException: error=13, Permission denied at java.lang.ProcessBuilder.start(ProcessBuilder.java:475) at com.cloud.utils.script.Script.execute(Script.java:183) at com.cloud.utils.script.Script.execute(Script.java:161) at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:443) at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138) at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied at java.lang.UNIXProcess.<init>(UNIXProcess.java:164) at java.lang.ProcessImpl.start(ProcessImpl.java:81) at java.lang.ProcessBuilder.start(ProcessBuilder.java:468) ... 15 more 2013-02-18 21:30:39,800 ERROR [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-1:10.147.40.18) Unable to unpack snapshot OVA file at: /var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8/routing-8.ova 2013-02-18 21:30:39,801 ERROR [vmware.manager.VmwareStorageManagerImpl] (DirectAgent-1:10.147.40.18) Unable to execute PrimaryStorageDownloadCommand due to exception java.lang.Exception: Unable to unpack snapshot OVA file at: /var/cloudstack/mnt/VM/7313725325399.4970b63d/template/tmpl/1/8/routing-8.ova at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.copyTemplateFromSecondaryToPrimary(VmwareStorageManagerImpl.java:447) at com.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:138) at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3587) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:401) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679)"
        Hide
        Kiran Koneti added a comment -

        I still see the same issue.. Please resolve this ASAP as this is blocking.

        Show
        Kiran Koneti added a comment - I still see the same issue.. Please resolve this ASAP as this is blocking.
        Hide
        Pradeep Soundararajan added a comment - - edited

        Right now I am not seeing that duplicate entries. There is a check-in went into both the branches master/4.1 to remove that:

        commit 5594b1d622adb07203d4549aee426268ecee092d
        Author: Hugo Trippaers <htrippaers@schubergphilis.com>
        Date: Tue Feb 12 16:16:36 2013 +0100

        Don't forget the environment.properties when packaging

        packaging/centos63/cloud.spec | 3 ++-
        packaging/centos63/replace.properties | 3 +--
        2 files changed, 3 insertions, 3 deletions

        Could you please verify this using the latest build.

        Show
        Pradeep Soundararajan added a comment - - edited Right now I am not seeing that duplicate entries. There is a check-in went into both the branches master/4.1 to remove that: commit 5594b1d622adb07203d4549aee426268ecee092d Author: Hugo Trippaers <htrippaers@schubergphilis.com> Date: Tue Feb 12 16:16:36 2013 +0100 Don't forget the environment.properties when packaging packaging/centos63/cloud.spec | 3 ++- packaging/centos63/replace.properties | 3 +-- 2 files changed, 3 insertions , 3 deletions Could you please verify this using the latest build.
        Hide
        Sateesh Chodapuneedi added a comment -

        Hi Kiran,
        This seems package/install script issue.
        Can you please upload install log file.

        Show
        Sateesh Chodapuneedi added a comment - Hi Kiran, This seems package/install script issue. Can you please upload install log file.
        Hide
        Pradeep Soundararajan added a comment -

        I am seeing these duplicate entries for "MSMNTDIR" in packaging/centos63/replace.properties, This is called during packaging and it is replacing the one present in build/.

        MSMNTDIR=/mnt
        MSMNTDIR=/var/lib/cloud/mnt

        Show
        Pradeep Soundararajan added a comment - I am seeing these duplicate entries for "MSMNTDIR" in packaging/centos63/replace.properties, This is called during packaging and it is replacing the one present in build/. MSMNTDIR=/mnt MSMNTDIR=/var/lib/cloud/mnt
        Hide
        Animesh Chaturvedi added a comment -

        Also since you are looking into it, please mark it "In progress"

        Show
        Animesh Chaturvedi added a comment - Also since you are looking into it, please mark it "In progress"
        Hide
        Animesh Chaturvedi added a comment -

        Sateesh any updates?

        Show
        Animesh Chaturvedi added a comment - Sateesh any updates?
        Hide
        Rayees Namathponnan added a comment -

        Here workaround

        I created directory "/var/lib/cloud/mnt" manually with owner is user "cloud", after this i diddnt faced this issue,

        Kiran - Can you please try this ?

        Show
        Rayees Namathponnan added a comment - Here workaround I created directory "/var/lib/cloud/mnt" manually with owner is user "cloud", after this i diddnt faced this issue, Kiran - Can you please try this ?
        Hide
        Kiran Koneti added a comment -

        mount -t nfs 10.147.28.7:/export/home/kiran/s /var/lib/cloud/mnt/VM/7313725325399.75eba671
        mount.nfs: mount point /var/lib/cloud/mnt/VM/7313725325399.75eba671 does not exist

        There is no folder existing but when i checked the path i could fine the following path exsisting "/var/lib/cloud/management/systemvm_mnt"

        Show
        Kiran Koneti added a comment - mount -t nfs 10.147.28.7:/export/home/kiran/s /var/lib/cloud/mnt/VM/7313725325399.75eba671 mount.nfs: mount point /var/lib/cloud/mnt/VM/7313725325399.75eba671 does not exist There is no folder existing but when i checked the path i could fine the following path exsisting "/var/lib/cloud/management/systemvm_mnt"
        Hide
        Sateesh Chodapuneedi added a comment -

        Hi Kiran,
        Can you please run following command in management server and let me know the result?
        mount -t nfs 10.147.28.7:/export/home/kiran/s /var/lib/cloud/mnt/VM/7313725325399.75eba671

        Show
        Sateesh Chodapuneedi added a comment - Hi Kiran, Can you please run following command in management server and let me know the result? mount -t nfs 10.147.28.7:/export/home/kiran/s /var/lib/cloud/mnt/VM/7313725325399.75eba671
        Hide
        Kiran Koneti added a comment -

        There is no such folder named /mnt/sec in my management serever.

        "mount.nfs: mount point /mnt/sec does not exist" this is the output.

        Show
        Kiran Koneti added a comment - There is no such folder named /mnt/sec in my management serever. "mount.nfs: mount point /mnt/sec does not exist" this is the output.
        Hide
        Sateesh Chodapuneedi added a comment -

        Hi Kiran,
        Can you please run following command in management server and let me know the result?
        mount -t nfs 10.147.28.7:/export/home/kiran/s /mnt/sec

        Show
        Sateesh Chodapuneedi added a comment - Hi Kiran, Can you please run following command in management server and let me know the result? mount -t nfs 10.147.28.7:/export/home/kiran/s /mnt/sec
        Hide
        Kiran Koneti added a comment -

        Hi Sateesh

        Attached the catalina.out files of yeaterday and today whcih might help you in debugging.Yesterday's log file might contain the details you are looking for.

        Show
        Kiran Koneti added a comment - Hi Sateesh Attached the catalina.out files of yeaterday and today whcih might help you in debugging.Yesterday's log file might contain the details you are looking for.
        Hide
        Sateesh Chodapuneedi added a comment -

        Can we get catalina.out log file from this setup?

        Show
        Sateesh Chodapuneedi added a comment - Can we get catalina.out log file from this setup?
        Hide
        Kiran Koneti added a comment -

        I have checked all the permissions of my NFS and it has read write permissions.And also it has enough space also i tried the same even in the dev enviroment and see the same error message.

        Show
        Kiran Koneti added a comment - I have checked all the permissions of my NFS and it has read write permissions.And also it has enough space also i tried the same even in the dev enviroment and see the same error message.
        Hide
        Rohit Yadav added a comment -

        From logs and ls listing from Kiran, the file is not there. Is this a NFS issue, Kiran can you confirm if your nfs share is read-able, check permission and also that there was enough space (so that the ova could be copied there)?
        Assigning this to Sateesh looks like a VMWare issue, the ova failed to get copied.

        Show
        Rohit Yadav added a comment - From logs and ls listing from Kiran, the file is not there. Is this a NFS issue, Kiran can you confirm if your nfs share is read-able, check permission and also that there was enough space (so that the ova could be copied there)? Assigning this to Sateesh looks like a VMWare issue, the ova failed to get copied.
        Hide
        Sateesh Chodapuneedi added a comment -

        Mount operation itself failed. I suspect permissions issue
        2013-02-11 23:12:44,203 WARN [vmware.manager.VmwareManagerImpl] (DirectAgent-83:10.147.40.18) Unable to create a mount point
        2013-02-11 23:12:44,203 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-83:10.147.40.18) Unable to create mount point for nfs://10.147.28.7/export/home/kiran/s

        Show
        Sateesh Chodapuneedi added a comment - Mount operation itself failed. I suspect permissions issue 2013-02-11 23:12:44,203 WARN [vmware.manager.VmwareManagerImpl] (DirectAgent-83:10.147.40.18) Unable to create a mount point 2013-02-11 23:12:44,203 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-83:10.147.40.18) Unable to create mount point for nfs://10.147.28.7/export/home/kiran/s
        Hide
        Kiran Koneti added a comment -

        The File routing-8.ova is not available.in the directory.I think it is searching for a template with a different name.
        I see the below files in that directory
        500123cf-f814-45b6-838c-788d282a3ff5.ova burbank-systemvm-08012012.ova-disk1.vmdk burbank-systemvm-08012012.ova.mf burbank-systemvm-08012012.ova.ovf template.properties

        There is no file with the name "routing-8.ova ".

        Show
        Kiran Koneti added a comment - The File routing-8.ova is not available.in the directory.I think it is searching for a template with a different name. I see the below files in that directory 500123cf-f814-45b6-838c-788d282a3ff5.ova burbank-systemvm-08012012.ova-disk1.vmdk burbank-systemvm-08012012.ova.mf burbank-systemvm-08012012.ova.ovf template.properties There is no file with the name "routing-8.ova ".
        Hide
        Abhinandan Prateek added a comment -

        Rohit,
        This file is not there /mnt/sec/template/tmpl/1/8/routing-8.ova. Can you take a initial look ?

        Show
        Abhinandan Prateek added a comment - Rohit, This file is not there /mnt/sec/template/tmpl/1/8/routing-8.ova. Can you take a initial look ?
        Hide
        Chip Childers added a comment -

        This line is the interesting one to me:

        java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or directory

        But I realized that I asked a silly question. The better question is:

        Does this file exist?

        /mnt/sec/template/tmpl/1/8/routing-8.ova

        Show
        Chip Childers added a comment - This line is the interesting one to me: java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or directory But I realized that I asked a silly question. The better question is: Does this file exist? /mnt/sec/template/tmpl/1/8/routing-8.ova
        Hide
        Kiran Koneti added a comment -

        Chip Can you let me know which tar you are asking about?

        Show
        Kiran Koneti added a comment - Chip Can you let me know which tar you are asking about?
        Hide
        Chip Childers added a comment -

        Is tar installed on the effected system?

        Show
        Chip Childers added a comment - Is tar installed on the effected system?
        Hide
        Animesh Chaturvedi added a comment -

        Vijay didn't you run into similar issue

        Show
        Animesh Chaturvedi added a comment - Vijay didn't you run into similar issue
        Hide
        Sateesh Chodapuneedi added a comment -

        Observed this when permissions are not sufficient to mont secondary storage.

        Show
        Sateesh Chodapuneedi added a comment - Observed this when permissions are not sufficient to mont secondary storage.

          People

          • Assignee:
            Pradeep Soundararajan
            Reporter:
            Kiran Koneti
          • Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development