Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-18215

HiveServer2 dies since ZK unavailable, Ambari 2.0.2 with Ranger + Kerberos is upgraded to Ambari 2.4.0.0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: ambari-upgrade
    • Labels:
      None

      Description

      STR:
      1) Install old version (Add ranger)
      2) Enable Mit security
      2) Make ambari only upgrade
      3) Add host

      Actual result:
      Hive Service 2 becomes stopped by it self after Upgrade from 2.0.2 to 2.4.0.0

      This is the fatal error that caused HS2 to die,

      2016-05-17 04:07:03,028 FATAL [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(204)) - Unable to create a znode for this server instance
      java.lang.IllegalStateException: Not started
              at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
              at org.apache.curator.framework.recipes.nodes.PersistentEphemeralNode.waitForInitialCreate(PersistentEphemeralNode.java:256)
              at org.apache.hive.service.server.HiveServer2.addServerInstanceToZooKeeper(HiveServer2.java:189)
              at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:334)
              at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:70)
              at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:581)
              at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:454)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      

      This error is actually ignored by HS2.

      2016-05-16 08:06:05,501 INFO  [HiveServer2-Handler-Pool: Thread-53]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(720)) - 2: Shutting down the object store...
      2016-05-16 08:06:05,502 INFO  [HiveServer2-Handler-Pool: Thread-53]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(346)) - ugi=hive/host1@EXAMPLE.COM	ip=unknown-ip-addr	cmd=Shutting down the object store...	
      2016-05-16 08:06:05,502 INFO  [HiveServer2-Handler-Pool: Thread-53]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(720)) - 2: Metastore shutdown complete.
      2016-05-16 08:06:05,502 INFO  [HiveServer2-Handler-Pool: Thread-53]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(346)) - ugi=hive/host1@EXAMPLE.COM	ip=unknown-ip-addr	cmd=Metastore shutdown complete.	
      2016-05-16 08:06:05,504 ERROR [HiveServer2-Handler-Pool: Thread-53]: metadata.Hive (Hive.java:getDelegationToken(2890)) - java.lang.UnsupportedOperationException: getDelegationToken() can be called only in thrift (non local) mode
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:1670)
      	at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
      	at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
      	at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2888)
      	at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:468)
      	at org.apache.hive.service.cli.thrift.ThriftCLIService.getDelegationToken(ThriftCLIService.java:340)
      	at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:321)
      	at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:235)
      	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
      	at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
      	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:679)
      	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      

        Attachments

        1. AMBARI-18215.patch
          2 kB
          Dmytro Sen

          Issue Links

            Activity

              People

              • Assignee:
                dsen Dmytro Sen
                Reporter:
                dsen Dmytro Sen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: