[CLOUDSTACK-7951] cloudstack-agent jsvc gets too large virtual memory space.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.3.0
• Fix Version/s: 4.5.0, 4.4.3, 4.3.2, 4.6.0, Future
• Component/s: KVM
• Security Level: Public (Anyone can view this level - this is the default.)

Environment

- Builds up with 155 physical machines and a cloudstack server.
- Each physical machine equips these.
 + Xeon E5-2680 (10 cores, 20 threads).
 + 128GB memory (and is not set swap space).
 + 600GB Solid State Drive.
- All physical machines are networked by InfiniBand and 10GBase-T.
- Using CentOS6.6 (64bit)

• Assignee: Rohit Yadav
• Reporter: Keiichi Yusa
• Created: 20/Nov/14 07:45
• Updated: 11/Mar/15 11:56
• Resolved: 26/Nov/14 18:46

Description

cloudstack-agent jsvc gets too large virtual memory space on huge
memory equipped machine.

We have 155 physical machines that each is equipped 128GB RAM and
is installed CentOS6.6 (64bit). On these physical machines, I use
KVM as hypervisor.

I create Compute Offering with 120GB RAM and I deploy VM with this
Compute Offering on this environment.

Now, I have a problem that qemu-kvm process often fails deploying VM
in the above conditions. I confirm that the following message is
outputted in /var/log/libvirt/qemu/i-X-X-VM.log.

"Cannot set up guest memory 'pc.ram': Cannot allocate memory"

I find that cloudstack-agent jsvc gets virtual memory about 35GB.

$ top -a
(snip)
   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  5787 root      20   0 34.9g 667m  18m S  0.0  0.5   8:45.20 jsvc
(snip)

I suspect that this failure is caused by what cloudstack-agent jsvc
gets too large virtual memory.

I try to patch /etc/init.d/cloudstack-agent as follows:

@@ -66,7 +66,7 @@
 start() {
     echo -n $"Starting $PROGNAME: "
     if hostname --fqdn >/dev/null 2>&1 ; then
-        $JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
+        $JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
             -errfile $LOGDIR/cloudstack-agent.err -outfile $LOGDIR/cloudstack-agent.out $CLASS
         RETVAL=$?
         echo

Then, I restart cloudstack-agent.

As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
Also, qemu-kvm process does not fail deploying VM for now.

$ top -a
(snip)
   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
141405 root      20   0 6290m 393m  18m S  0.0  0.3   1:11.44 jsvc
(snip)

If that helps, our CloudStack environment is as follows:

- Builds up with 155 physical machines and a cloudstack server.
- Each physical machine equips these.
 + Xeon E5-2680 (10 cores, 20 threads)
 + 128GB memory (and is not set swap space).
 + 600GB Solid State Drive.
- All physical machines are networked by InfiniBand and 10GBase-T.
- Using CentOS6.6 (64bit)