Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-14377

HiveServer start fails after enabling security post EU from 2.1 to 2.3 on non-HA cluster

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.0
    • Component/s: ambari-server
    • Labels:
      None

      Description

      Steps:

      1. Try EU from HDP 2.1 to 2.3.4 on an unsecure and non-HA cluster
      2. Let the EU succeed
      3. Enable security on the cluster

      Result:
      After enabling security HiveServer2 goes down

      Looked at the value of three properties:
      hive.cluster.delegation.token.store.zookeeper.connectString": "localhost:2181"
      hive.zookeeper.quorum": "localhost:2181"
      hive.cluster.delegation.token.store.class:org.apache.hadoop.hive.thrift.ZooKeeperTokenStore

      It appears that values of first and second property are wrongly set (compared this with a fresh non-HA cluster where Kerberos was enabled after install)

      Logs on HiveServer node:

      2015-12-14 17:41:48,495 FATAL [Thread-9]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
      veserorg.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegationHIVESERVER2/keys
              at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
              at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
              at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
              at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server.startDelegationTokenSecretManager(HadoopThriftAuthBridge.java:444)
              at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:124)
              at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hive/cluster/delegationHIVESERVER2/keys
              at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
              at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
              at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
              at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691)
              at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675)
              at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
              at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:672)
              at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
              at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
              at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
              at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
              at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
              at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
              ... 6 more
      2015-12-14 17:41:48,497 INFO  [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(709)) - SHUTDOWN_MSG:
      /************************************************************
      SHUTDOWN_MSG: Shutting down HiveServer2 at os-r6-udmbts-baikaltom20unsecr-re-3.novalocal/172.22.76.200
      ************************************************************/
      2015-12-14 17:41:58,398 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
      2015-12-14 17:41:58,398 INFO  [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped.
      2015-12-14 17:41:58,403 INFO  [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper.
      2015-12-14 17:41:58,403 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:stop(371)) - Shutting down HiveServer2
      2015-12-14 17:41:58,403 INFO  [Thread-7]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(338)) - Server instance removed from ZooKeeper.
      

      As it turns out, the installation wizard sets these at Hive installation time, even though they are only used for a Kerberized Hive. Therefore, the Kerberization wizard doesn't set these at all.

      When upgrading from HDP 2.1 Hive, the installation wizard hasn't set these since they first appeared in HDP 2.2. Therefore, they are never set. The right way to fix this is to have the Kerberos Wizard determine that it needs to calculate and set them. But that's an architectural change mostly. The faster fix is to make Hive consistent - just set them on upgrade from HDP 2.1

        Attachments

        1. AMBARI-14377.patch
          24 kB
          Jonathan Hurley

          Issue Links

            Activity

              People

              • Assignee:
                jonathan.hurley Jonathan Hurley
                Reporter:
                jonathan.hurley Jonathan Hurley
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: