[AMBARI-21528] Zookeeper server has incorrect memory setting, missing m in Xmx value

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.5.2
• Fix Version/s: trunk, 2.5.2
• Component/s: stacks
• Labels: AMBARI-21348
• Assignee: Alejandro Fernandez
• Reporter: Alejandro Fernandez
• Created: 19/Jul/17 22:59
• Updated: 20/Jul/17 22:51
• Resolved: 20/Jul/17 21:51

Description

Repro Steps:

• Installed BI 4.2.0 cluster on IBM Ambari 2.2.2 with Zookeeper
• Upgraded Ambari to 2.5.2.0-146
• Registered HDP 2.6.2.0 repo, installed packages
• Ran service checks
• Started Express Upgrade

Result: Service Check ZooKeeper step failed with KeeperErrorCode = ConnectionLoss for /zk_smoketest

This was caused by Zookeeper dying immediately during restart:

Error occurred during initialization of VM
Too small initial heap

zookeeper-env.sh before upgrade

export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
export ZOOKEEPER_HOME=/usr/iop/current/zookeeper-server
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
export SERVER_JVMFLAGS=-Xmx1024m
export JAVA=$JAVA_HOME/bin/java
export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*

zookeeper-env.sh after upgrade

export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-client
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
export SERVER_JVMFLAGS=-Xmx1024
export JAVA=$JAVA_HOME/bin/java
export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*

Note missing "m" in memory setting.

zookeeper-env template contains,

export SERVER_JVMFLAGS={{zk_server_heapsize}}

In this cluster, zookeeper-env contains,
zk_server_heapsize: "1024"

While the params_linux.py file has some inconsistencies with appending the letter "m".

zk_server_heapsize_value = str(default('configurations/zookeeper-env/zk_server_heapsize', "1024m"))
zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")

Instead, it should be,

zk_server_heapsize_value = str(default('configurations/zookeeper-env/zk_server_heapsize', "1024"))
zk_server_heapsize_value = zk_server_heapsize_value.strip()
if len(zk_server_heapsize_value) > 0 and not zk_server_heapsize_value[-1].isdigit():
  zk_server_heapsize_value = zk_server_heapsize_value + "m"
zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")

Issue Links

• links to: Review Board patch