Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-7951

cloudstack-agent jsvc gets too large virtual memory space.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 4.3.0
    • Fix Version/s: 4.5.0, 4.4.3, 4.3.2, 4.6.0, Future
    • Component/s: KVM
    • Security Level: Public (Anyone can view this level - this is the default.)
    • Labels:
      None
    • Environment:

      Description

      cloudstack-agent jsvc gets too large virtual memory space on huge
      memory equipped machine.

      We have 155 physical machines that each is equipped 128GB RAM and
      is installed CentOS6.6 (64bit). On these physical machines, I use
      KVM as hypervisor.

      I create Compute Offering with 120GB RAM and I deploy VM with this
      Compute Offering on this environment.

      Now, I have a problem that qemu-kvm process often fails deploying VM
      in the above conditions. I confirm that the following message is
      outputted in /var/log/libvirt/qemu/i-X-X-VM.log.

      "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
      

      I find that cloudstack-agent jsvc gets virtual memory about 35GB.

      $ top -a
      (snip)
         PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
        5787 root      20   0 34.9g 667m  18m S  0.0  0.5   8:45.20 jsvc
      (snip)
      

      I suspect that this failure is caused by what cloudstack-agent jsvc
      gets too large virtual memory.

      I try to patch /etc/init.d/cloudstack-agent as follows:

      @@ -66,7 +66,7 @@
       start() {
           echo -n $"Starting $PROGNAME: "
           if hostname --fqdn >/dev/null 2>&1 ; then
      -        $JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
      +        $JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
                   -errfile $LOGDIR/cloudstack-agent.err -outfile $LOGDIR/cloudstack-agent.out $CLASS
               RETVAL=$?
               echo
      

      Then, I restart cloudstack-agent.

      As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
      Also, qemu-kvm process does not fail deploying VM for now.

      $ top -a
      (snip)
         PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
      141405 root      20   0 6290m 393m  18m S  0.0  0.3   1:11.44 jsvc
      (snip)
      

      If that helps, our CloudStack environment is as follows:

      - Builds up with 155 physical machines and a cloudstack server.
      - Each physical machine equips these.
       + Xeon E5-2680 (10 cores, 20 threads)
       + 128GB memory (and is not set swap space).
       + 600GB Solid State Drive.
      - All physical machines are networked by InfiniBand and 10GBase-T.
      - Using CentOS6.6 (64bit)
      

        Attachments

          Activity

            People

            • Assignee:
              bhaisaab Rohit Yadav
              Reporter:
              yusa Keiichi Yusa
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: