[AMBARI-18215] HiveServer2 dies since ZK unavailable, Ambari 2.0.2 with Ranger + Kerberos is upgraded to Ambari 2.4.0.0

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: ambari-upgrade
• Assignee: Dmytro Sen
• Reporter: Dmytro Sen
• Created: 19/Aug/16 15:51
• Updated: 19/Aug/16 19:09
• Resolved: 19/Aug/16 16:48

Description

STR:
1) Install old version (Add ranger)
2) Enable Mit security
2) Make ambari only upgrade
3) Add host

Actual result:
Hive Service 2 becomes stopped by it self after Upgrade from 2.0.2 to 2.4.0.0

This is the fatal error that caused HS2 to die,

2016-05-17 04:07:03,028 FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(204)) - Unable to create a znode for this server instance
java.lang.IllegalStateException: Not started
        at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
        at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode.waitForInitialCreate(PersistentEphemeralNode.java:256)
        at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:189)
        at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:334)
        at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:70)
        at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:581)
        at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:454)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

This error is actually ignored by HS2.

2016-05-16 08:06:05,501 INFO  [HiveServer2-Handler-Pool: Thread-53]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(720)) - 2: Shutting down the object store...
2016-05-16 08:06:05,502 INFO  [HiveServer2-Handler-Pool: Thread-53]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(346)) - ugi=hive/[e-mail]	ip=unknown-ip-addr	cmd=Shutting down the object store...	
2016-05-16 08:06:05,502 INFO  [HiveServer2-Handler-Pool: Thread-53]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(720)) - 2: Metastore shutdown complete.
2016-05-16 08:06:05,502 INFO  [HiveServer2-Handler-Pool: Thread-53]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(346)) - ugi=hive/[e-mail]	ip=unknown-ip-addr	cmd=Metastore shutdown complete.	
2016-05-16 08:06:05,504 ERROR [HiveServer2-Handler-Pool: Thread-53]: metadata.Hive (Hive.java:getDelegationToken(2890)) - java.lang.UnsupportedOperationException: getDelegationToken() can be called only in thrift (non local) mode
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:1670)
	at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
	at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2888)
	at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:468)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.getDelegationToken(ThriftCLIService.java:340)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:321)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:235)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:679)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: Review Board