[AMBARI-21239] Ambari leaves a lot of processes from Hive alert on the machine

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5.2
• Assignee: Andrew Onischuk
• Reporter: Andrew Onischuk
• Created: 13/Jun/17 13:57
• Updated: 13/Jun/17 14:50
• Resolved: 13/Jun/17 14:01

Description

I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
the database VM storing the metadata after the cluster had been runing for
almost 2 weeks without issues. What I observed is that the instance in a few
hours got really slow. Upon investigation found hundreds of Beeline processes
like this:

ambari-+ 3439770 3439769 0 06:19 ? 00:00:00 -bash -c export PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' -e '' 2>&1| awk '

{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3439814 3439770 0 06:19 ? 00:00:42 /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary -e
ambari-+ 3439816 3439770 0 06:19 ? 00:00:00 awk {print}

ambari-+ 3439817 3439770 0 06:19 ? 00:00:00 grep -i -e Connection refused -e Invalid URL
ambari-+ 3440645 3440644 0 06:22 ? 00:00:00 -bash -c export PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' -e '' 2>&1| awk '

{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3440685 3440645 0 06:22 ? 00:00:42 /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary -e
ambari-+ 3440686 3440645 0 06:22 ? 00:00:00 awk {print}

ambari-+ 3440687 3440645 0 06:22 ? 00:00:00 grep -i -e Connection refused -e Invalid URL
ambari-+ 3441185 3441184 0 06:25 ? 00:00:00 -bash -c export PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' -e '' 2>&1| awk '

{print}

'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3441224 3441185 0 06:25 ? 00:00:46 /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary -e

Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in
them and owned by ambari found 309 processes.
The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't
accessible as the DB host serving the metadata went down.

Issue Links

• links to: ReviewBoard