[HIVE-20548] Can not start llap via yarn service

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: llap
• Assignee: Unassigned
• Reporter: Butao Zhang
• Created: 13/Sep/18 02:51
• Updated: 24/Oct/19 23:57

Description

We start llap through yarn service instead of slider, and some problems happen as follows:

2018-09-12 19:32:48,629 - LLAP start command: /usr/bch/current/hive-server2/bin/hive --service llap --size 10930m --startImmediately --name llap0 --cache 0m --xmx 8m --loglevel INFO --output /var/lib/ambari-agent/tmp/llap-yarn-service_2018-09-12_11-32-48 --service-placement 4 --skiphadoopversion --skiphbasecp --instances 1 --logger query-routing --args " -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:InitiatingHeapOccupancyPercent=70 -XX:+UnlockExperimentalVMOptions -XX:G1MaxNewSizePercent=40 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=200 -XX:MetaspaceSize=1024m"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/bch/3.0.0/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/bch/3.0.0/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist
WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute does not exist
WARN cli.LlapServiceDriver: Ignoring unknown llap server parameter: [hive.aux.jars.path]
WARN cli.LlapServiceDriver: Java versions might not match : JAVA_HOME=[/usr/jdk64/jdk1.8.0_112],process jre=[/usr/jdk64/jdk1.8.0_112/jre]
WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist
WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute does not exist
WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist
WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute does not exist
11:32:54 Running as a child of LlapServiceDriver
11:32:54 Prepared the files
11:33:13 Packaged the files
WARN curator.CuratorZookeeperClient: session timeout [10000] is less than connection timeout [15000]
ERROR client.ServiceClient: Error on destroy 'llap0': not found.
WARN client.ServiceClient: Property yarn.service.framework.path has a value /bch/apps/3.0.0/yarn/service-dep.tar.gz, but is not a valid file
2018-09-12 19:33:17,385 - 





2018-09-12 19:33:17,385 - LLAP status command : /usr/bch/current/hive-server2/bin/hive --service llapstatus -w -r 0.8 -i 2 -t 400
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/bch/3.0.0/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/bch/3.0.0/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist
WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
WARN conf.HiveConf: HiveConf of name hive.druid.select.distribute does not exist

LLAPSTATUS WatchMode with timeout=400 s
--------------------------------------------------------------------------------
LLAP Starting up with AppId=application_1536745653378_0002.
--------------------------------------------------------------------------------
WARN cli.LlapStatusServiceDriver: COMPLETE state reached while waiting for RUNNING state. Failing.
Final diagnostics: null
LLAP Application already complete. ApplicationId=application_1536745653378_0002
--------------------------------------------------------------------------------




{
  "amInfo" : {
    "appName" : "llap0",
    "appType" : "yarn-service",
    "appId" : "application_1536745653378_0002"
  },
  "state" : "COMPLETE",
  "appStartTime" : 1536751995975,
  "appFinishTime" : 1536752013756,
  "runningThresholdAchieved" : false
}
2018-09-12 19:33:34,521 - LLAP app 'llap0' current state is COMPLETE.
2018-09-12 19:33:34,521 - LLAP app 'llap0' current state is COMPLETE.
2018-09-12 19:33:34,521 - LLAP app 'llap0' deployment unsuccessful.

Command failed after 1 tries

We find the error logs:

ERROR client.ServiceClient: Error on destroy 'llap0': not found.
WARN client.ServiceClient: Property yarn.service.framework.path has a value /bch/apps/3.0.0/yarn/service-dep.tar.gz, but is not a valid file

but we can't  make sure if the yarn service or llap has some problems. In addition, hadoop version is 3.1.0.