Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-13133

Hive Metastore did not start when Kerberized

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.3.0, 2.1.2
    • Component/s: None
    • Labels:
      None

      Description

      When starting up HiveMetastore under a Kerberized cluster, the following error occurs:

      resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/hive.service.keytab hive/host1.company.com@REALM; ' returned 1. kinit: Keytab contains no suitable keys for hive/host1.company.com@REALM while getting initial credentials
      

      This happens when Hive Metastore and HiveServer2 principals are set up distinct from each other.
      Hive Metastore is not using hive.metastore.kerberos.principal, but instead it uses hive.server2.authentication.kerberos.principal

      Also, the following references hive_conf_dir:
      https://github.com/apache/ambari/blob/release-2.1.1/ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py#L119-L120

      In HDP2.3+ the following file content becomes UNSECURED
      /var/lib/ambari-agent/data/structured-out-status.json

      We need to either reference hive_server_conf_dir or set hive_conf_dir as hive_server_conf_dir somewhere:
      https://github.com/apache/ambari/blob/release-2.1.1/ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py#L90-L101

      Solution
      Since a kinit call here is unnecessary and the relevant configuration files are being created properly. Simply removing the kinit call (and related variabled) will fix the kinit failure issue.

      For the hive_conf_dir issue, setting hive_conf_dir = hive_server_conf_dir in status_params.py, solves the issue.

      1. AMBARI-13133_branch2.1_01.patch
        5 kB
        Robert Levas
      2. AMBARI-13133_trunk_01.patch
        5 kB
        Robert Levas

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12757124/AMBARI-13133_branch2.1_01.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in ambari-server.

          Test results: https://builds.apache.org/job/Ambari-trunk-test-patch/3804//testReport/
          Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/3804//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12757124/AMBARI-13133_branch2.1_01.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in ambari-server. Test results: https://builds.apache.org/job/Ambari-trunk-test-patch/3804//testReport/ Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/3804//console This message is automatically generated.
          Hide
          rlevas Robert Levas added a comment -

          Committed to trunk

          commit 7bc58ac7c76bd5550d5794ec679860c7c0531e3c
          Author: Robert Levas <rlevas@hortonworks.com>
          Date:   Fri Sep 18 09:08:31 2015 -0400
          

          Committed to branch-2.1

          commit 8d59cb56494bd03a2da5f6acca3e7d41e89c254a
          Author: Robert Levas <rlevas@hortonworks.com>
          Date:   Fri Sep 18 10:51:03 2015 -0400
          

          Committed to branch-2.1.2

          commit 12a65e6c5f1fe33d5e9717e9a140b929140dcf33
          Author: Robert Levas <rlevas@hortonworks.com>
          Date:   Fri Sep 18 10:52:29 2015 -0400
          
          Show
          rlevas Robert Levas added a comment - Committed to trunk commit 7bc58ac7c76bd5550d5794ec679860c7c0531e3c Author: Robert Levas <rlevas@hortonworks.com> Date: Fri Sep 18 09:08:31 2015 -0400 Committed to branch-2.1 commit 8d59cb56494bd03a2da5f6acca3e7d41e89c254a Author: Robert Levas <rlevas@hortonworks.com> Date: Fri Sep 18 10:51:03 2015 -0400 Committed to branch-2.1.2 commit 12a65e6c5f1fe33d5e9717e9a140b929140dcf33 Author: Robert Levas <rlevas@hortonworks.com> Date: Fri Sep 18 10:52:29 2015 -0400
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Ambari-trunk-Commit #3463 (See https://builds.apache.org/job/Ambari-trunk-Commit/3463/)
          AMBARI-13133. Hive Metastore did not start when Kerberized (rlevas) (rlevas: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=7bc58ac7c76bd5550d5794ec679860c7c0531e3c)

          • ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py
          • ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py
          • ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py
          • ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
          • ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Ambari-trunk-Commit #3463 (See https://builds.apache.org/job/Ambari-trunk-Commit/3463/ ) AMBARI-13133 . Hive Metastore did not start when Kerberized (rlevas) (rlevas: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=7bc58ac7c76bd5550d5794ec679860c7c0531e3c ) ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Ambari-branch-2.1 #554 (See https://builds.apache.org/job/Ambari-branch-2.1/554/)
          AMBARI-13133. Hive Metastore did not start when Kerberized (rlevas) (rlevas: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=8d59cb56494bd03a2da5f6acca3e7d41e89c254a)

          • ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
          • ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py
          • ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py
          • ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py
          • ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Ambari-branch-2.1 #554 (See https://builds.apache.org/job/Ambari-branch-2.1/554/ ) AMBARI-13133 . Hive Metastore did not start when Kerberized (rlevas) (rlevas: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=8d59cb56494bd03a2da5f6acca3e7d41e89c254a ) ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py

            People

            • Assignee:
              rlevas Robert Levas
              Reporter:
              rlevas Robert Levas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development