Hive
  1. Hive
  2. HIVE-7443

Fix HiveConnection to communicate with Kerberized Hive JDBC server and alternative JDKs

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.12.0, 0.13.1
    • Fix Version/s: None
    • Component/s: JDBC, Security
    • Labels:
      None
    • Environment:

      Kerberos
      Run Hive server2 and client with IBM JDK7.1

      Description

      Hive Kerberos authentication has been enabled in my cluster. I ran kinit to initialize the current login user's ticket cache successfully, and then tried to use beeline to connect to Hive Server2, but failed. After I manually added some logging to catch the failure exception, this is what I got that caused the failure:

      beeline> !connect jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM org.apache.hive.jdbc.HiveDriver
      scan complete in 2ms
      Connecting to jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM
      Enter password for jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM:
      14/07/17 15:12:45 ERROR jdbc.HiveConnection: Failed to open client transport
      javax.security.sasl.SaslException: Failed to open client transport [Caused by java.io.IOException: Could not instantiate SASL transport]
      at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:78)
      at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:342)
      at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:200)
      at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:178)
      at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
      at java.sql.DriverManager.getConnection(DriverManager.java:582)
      at java.sql.DriverManager.getConnection(DriverManager.java:198)
      at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
      at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186)
      at org.apache.hive.beeline.Commands.connect(Commands.java:959)
      at org.apache.hive.beeline.Commands.connect(Commands.java:880)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
      at java.lang.reflect.Method.invoke(Method.java:619)
      at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44)
      at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:801)
      at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659)
      at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368)
      at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
      at java.lang.reflect.Method.invoke(Method.java:619)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.io.IOException: Could not instantiate SASL transport
      at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:177)
      at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:74)
      ... 24 more
      Caused by: javax.security.sasl.SaslException: Failure to initialize security context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0
      major string: Invalid credentials
      minor string: SubjectCredFinder: no JAAS Subject]
      at com.ibm.security.sasl.gsskerb.GssKrb5Client.<init>(GssKrb5Client.java:131)
      at com.ibm.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:53)
      at javax.security.sasl.Sasl.createSaslClient(Sasl.java:362)
      at org.apache.thrift.transport.TSaslClientTransport.<init>(TSaslClientTransport.java:72)
      at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:169)
      ... 25 more
      Caused by: org.ietf.jgss.GSSException, major code: 13, minor code: 0
      major string: Invalid credentials
      minor string: SubjectCredFinder: no JAAS Subject
      at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:83)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential$SubjectCredFinder.run(Krb5Credential.java:1126)
      at java.security.AccessController.doPrivileged(AccessController.java:330)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.getClientCredsFromSubject(Krb5Credential.java:816)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.getCredentials(Krb5Credential.java:388)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.init(Krb5Credential.java:196)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.<init>(Krb5Credential.java:168)
      at com.ibm.security.jgss.mech.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:123)
      at com.ibm.security.jgss.GSSManagerImpl.createMechCredential(GSSManagerImpl.java:294)
      at com.ibm.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:137)
      at com.ibm.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:69)
      at com.ibm.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:169)
      at com.ibm.security.jgss.GSSContextImpl.init(GSSContextImpl.java:157)
      at com.ibm.security.jgss.GSSContextImpl.<init>(GSSContextImpl.java:102)
      at com.ibm.security.jgss.GSSManagerImpl.createContext(GSSManagerImpl.java:183)
      at com.ibm.security.sasl.gsskerb.GssKrb5Client.<init>(GssKrb5Client.java:110)
      ... 29 more
      Error: Invalid URL: jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM (state=08S01,code=0)

        Activity

        Hide
        Yu Gao added a comment -

        Also tried with a Java client which does keytab login - UserGroupInformation.loginUserFromKeytab(client_principal, client_keytab") - before calls DriverManager.getConnection to make the connection. It failed with the same exception as that when using beeline. (The environment was set up correctly, jars, confs, kerberos and keytabs, etc.)

        Show
        Yu Gao added a comment - Also tried with a Java client which does keytab login - UserGroupInformation.loginUserFromKeytab(client_principal, client_keytab") - before calls DriverManager.getConnection to make the connection. It failed with the same exception as that when using beeline. (The environment was set up correctly, jars, confs, kerberos and keytabs, etc.)
        Hide
        Yu Gao added a comment -

        This is caused by no kerberos login behavior in HiveConnection class when opening transport to kerberized Hive server2: IBM JDK requires valid kerberos credentials in place when creating Sasl client, so adding UserGroupInformation.getCurrentUser() call in there, which in turn invokes UserGroupInformation.getLoginUser(). The login user is the one who holds kerberos credentials, either via ticket cache or via keytab login.

        After this change, to access Hive server2 using beeline, what client needs to do is a kinit;
        While for java client with keytab login, before make JDBC connection, one needs to call Hadoop UGI API to login (UGI.loginUserFromKeytab())

        Show
        Yu Gao added a comment - This is caused by no kerberos login behavior in HiveConnection class when opening transport to kerberized Hive server2: IBM JDK requires valid kerberos credentials in place when creating Sasl client, so adding UserGroupInformation.getCurrentUser() call in there, which in turn invokes UserGroupInformation.getLoginUser(). The login user is the one who holds kerberos credentials, either via ticket cache or via keytab login. After this change, to access Hive server2 using beeline, what client needs to do is a kinit; While for java client with keytab login, before make JDBC connection, one needs to call Hadoop UGI API to login (UGI.loginUserFromKeytab())
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12656397/HIVE-7443.patch

        ERROR: -1 due to 2 failed/errored test(s), 5740 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-844/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12656397

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12656397/HIVE-7443.patch ERROR: -1 due to 2 failed/errored test(s), 5740 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-844/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12656397
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12657492/HIVE-7443.patch

        ERROR: -1 due to 3 failed/errored test(s), 5756 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
        org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-34/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 3 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12657492

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12657492/HIVE-7443.patch ERROR: -1 due to 3 failed/errored test(s), 5756 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-34/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12657492
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12658595/HIVE-7443.patch

        ERROR: -1 due to 2 failed/errored test(s), 5838 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
        org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-108/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12658595

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12658595/HIVE-7443.patch ERROR: -1 due to 2 failed/errored test(s), 5838 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-108/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12658595
        Hide
        Yu Gao added a comment -

        The test failures are not related to the patch.

        Show
        Yu Gao added a comment - The test failures are not related to the patch.
        Hide
        He Zhang added a comment -

        Is this patch available now?

        Show
        He Zhang added a comment - Is this patch available now?

          People

          • Assignee:
            Yu Gao
            Reporter:
            Yu Gao
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development