Hive
  1. Hive
  2. HIVE-7443

Fix HiveConnection to communicate with Kerberized Hive JDBC server and alternative JDKs

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, 0.13.1
    • Fix Version/s: 2.2.0
    • Component/s: JDBC, Security
    • Labels:
      None
    • Environment:

      Kerberos
      Run Hive server2 and client with IBM JDK7.1

      Description

      Hive Kerberos authentication has been enabled in my cluster. I ran kinit to initialize the current login user's ticket cache successfully, and then tried to use beeline to connect to Hive Server2, but failed. After I manually added some logging to catch the failure exception, this is what I got that caused the failure:

      beeline> !connect jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM org.apache.hive.jdbc.HiveDriver
      scan complete in 2ms
      Connecting to jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM
      Enter password for jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM:
      14/07/17 15:12:45 ERROR jdbc.HiveConnection: Failed to open client transport
      javax.security.sasl.SaslException: Failed to open client transport [Caused by java.io.IOException: Could not instantiate SASL transport]
      at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:78)
      at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:342)
      at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:200)
      at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:178)
      at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
      at java.sql.DriverManager.getConnection(DriverManager.java:582)
      at java.sql.DriverManager.getConnection(DriverManager.java:198)
      at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
      at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186)
      at org.apache.hive.beeline.Commands.connect(Commands.java:959)
      at org.apache.hive.beeline.Commands.connect(Commands.java:880)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
      at java.lang.reflect.Method.invoke(Method.java:619)
      at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44)
      at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:801)
      at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659)
      at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368)
      at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
      at java.lang.reflect.Method.invoke(Method.java:619)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.io.IOException: Could not instantiate SASL transport
      at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:177)
      at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:74)
      ... 24 more
      Caused by: javax.security.sasl.SaslException: Failure to initialize security context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0
      major string: Invalid credentials
      minor string: SubjectCredFinder: no JAAS Subject]
      at com.ibm.security.sasl.gsskerb.GssKrb5Client.<init>(GssKrb5Client.java:131)
      at com.ibm.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:53)
      at javax.security.sasl.Sasl.createSaslClient(Sasl.java:362)
      at org.apache.thrift.transport.TSaslClientTransport.<init>(TSaslClientTransport.java:72)
      at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Client.createClientTransport(HadoopThriftAuthBridge20S.java:169)
      ... 25 more
      Caused by: org.ietf.jgss.GSSException, major code: 13, minor code: 0
      major string: Invalid credentials
      minor string: SubjectCredFinder: no JAAS Subject
      at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:83)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential$SubjectCredFinder.run(Krb5Credential.java:1126)
      at java.security.AccessController.doPrivileged(AccessController.java:330)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.getClientCredsFromSubject(Krb5Credential.java:816)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.getCredentials(Krb5Credential.java:388)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.init(Krb5Credential.java:196)
      at com.ibm.security.jgss.mech.krb5.Krb5Credential.<init>(Krb5Credential.java:168)
      at com.ibm.security.jgss.mech.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:123)
      at com.ibm.security.jgss.GSSManagerImpl.createMechCredential(GSSManagerImpl.java:294)
      at com.ibm.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:137)
      at com.ibm.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:69)
      at com.ibm.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:169)
      at com.ibm.security.jgss.GSSContextImpl.init(GSSContextImpl.java:157)
      at com.ibm.security.jgss.GSSContextImpl.<init>(GSSContextImpl.java:102)
      at com.ibm.security.jgss.GSSManagerImpl.createContext(GSSManagerImpl.java:183)
      at com.ibm.security.sasl.gsskerb.GssKrb5Client.<init>(GssKrb5Client.java:110)
      ... 29 more
      Error: Invalid URL: jdbc:hive2://<hiveserver.host>:10000/default;principal=hive/<hiveserver.host>@REALM.COM (state=08S01,code=0)

      1. HIVE-7443.2.patch
        3 kB
        Aihua Xu
      2. HIVE-7443.3.patch
        3 kB
        Aihua Xu
      3. HIVE-7443.patch
        3 kB
        Yu Gao

        Activity

        Hide
        Yu Gao added a comment -

        Also tried with a Java client which does keytab login - UserGroupInformation.loginUserFromKeytab(client_principal, client_keytab") - before calls DriverManager.getConnection to make the connection. It failed with the same exception as that when using beeline. (The environment was set up correctly, jars, confs, kerberos and keytabs, etc.)

        Show
        Yu Gao added a comment - Also tried with a Java client which does keytab login - UserGroupInformation.loginUserFromKeytab(client_principal, client_keytab") - before calls DriverManager.getConnection to make the connection. It failed with the same exception as that when using beeline. (The environment was set up correctly, jars, confs, kerberos and keytabs, etc.)
        Hide
        Yu Gao added a comment -

        This is caused by no kerberos login behavior in HiveConnection class when opening transport to kerberized Hive server2: IBM JDK requires valid kerberos credentials in place when creating Sasl client, so adding UserGroupInformation.getCurrentUser() call in there, which in turn invokes UserGroupInformation.getLoginUser(). The login user is the one who holds kerberos credentials, either via ticket cache or via keytab login.

        After this change, to access Hive server2 using beeline, what client needs to do is a kinit;
        While for java client with keytab login, before make JDBC connection, one needs to call Hadoop UGI API to login (UGI.loginUserFromKeytab())

        Show
        Yu Gao added a comment - This is caused by no kerberos login behavior in HiveConnection class when opening transport to kerberized Hive server2: IBM JDK requires valid kerberos credentials in place when creating Sasl client, so adding UserGroupInformation.getCurrentUser() call in there, which in turn invokes UserGroupInformation.getLoginUser(). The login user is the one who holds kerberos credentials, either via ticket cache or via keytab login. After this change, to access Hive server2 using beeline, what client needs to do is a kinit; While for java client with keytab login, before make JDBC connection, one needs to call Hadoop UGI API to login (UGI.loginUserFromKeytab())
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12656397/HIVE-7443.patch

        ERROR: -1 due to 2 failed/errored test(s), 5740 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
        org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-844/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12656397

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12656397/HIVE-7443.patch ERROR: -1 due to 2 failed/errored test(s), 5740 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/844/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-844/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12656397
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12657492/HIVE-7443.patch

        ERROR: -1 due to 3 failed/errored test(s), 5756 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
        org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-34/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 3 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12657492

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12657492/HIVE-7443.patch ERROR: -1 due to 3 failed/errored test(s), 5756 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/34/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-34/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12657492
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12658595/HIVE-7443.patch

        ERROR: -1 due to 2 failed/errored test(s), 5838 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
        org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-108/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12658595

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12658595/HIVE-7443.patch ERROR: -1 due to 2 failed/errored test(s), 5838 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/108/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-108/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12658595
        Hide
        Yu Gao added a comment -

        The test failures are not related to the patch.

        Show
        Yu Gao added a comment - The test failures are not related to the patch.
        Hide
        He Zhang added a comment -

        Is this patch available now?

        Show
        He Zhang added a comment - Is this patch available now?
        Hide
        Greg Senia added a comment -

        Yu Gao did this fix ever make it into Hive? If it didn't by applying https://issues.apache.org/jira/browse/HADOOP-9969 this issue with beeline is gone with Hive 1.2.0 and the following fixes... https://issues.apache.org/jira/browse/TEZ-3105, https://issues.apache.org/jira/browse/HIVE-13020

        Show
        Greg Senia added a comment - Yu Gao did this fix ever make it into Hive? If it didn't by applying https://issues.apache.org/jira/browse/HADOOP-9969 this issue with beeline is gone with Hive 1.2.0 and the following fixes... https://issues.apache.org/jira/browse/TEZ-3105 , https://issues.apache.org/jira/browse/HIVE-13020
        Hide
        Aihua Xu added a comment -

        Greg Senia So do you mean HIVE-13020 also fixes this issue? Or do we need the patch from HIVE-7443?

        Show
        Aihua Xu added a comment - Greg Senia So do you mean HIVE-13020 also fixes this issue? Or do we need the patch from HIVE-7443 ?
        Hide
        Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12658595/HIVE-7443.patch

        ERROR: -1 due to build exiting with an error

        Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/38/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/38/console
        Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-38/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
        + export JAVA_HOME=/usr/java/jdk1.8.0_25
        + JAVA_HOME=/usr/java/jdk1.8.0_25
        + export PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-38/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ git = \s\v\n ]]
        + [[ git = \g\i\t ]]
        + [[ -z master ]]
        + [[ -d apache-github-source-source ]]
        + [[ ! -d apache-github-source-source/.git ]]
        + [[ ! -d apache-github-source-source ]]
        + cd apache-github-source-source
        + git fetch origin
        From https://github.com/apache/hive
           0e975f0..52c7c04  branch-1   -> origin/branch-1
        + git reset --hard HEAD
        HEAD is now at ef7dc77 HIVE-13954: Parquet logs should go to STDERR (Prasanth Jayachandran reviewed by Gunther Hagleitner)
        + git clean -f -d
        + git checkout master
        Already on 'master'
        + git reset --hard origin/master
        HEAD is now at ef7dc77 HIVE-13954: Parquet logs should go to STDERR (Prasanth Jayachandran reviewed by Gunther Hagleitner)
        + git merge --ff-only origin/master
        Already up-to-date.
        + git gc
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12658595 - PreCommit-HIVE-MASTER-Build

        Show
        Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12658595/HIVE-7443.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/38/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/38/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-38/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]] + export JAVA_HOME=/usr/java/jdk1.8.0_25 + JAVA_HOME=/usr/java/jdk1.8.0_25 + export PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.8.0_25/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-38/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin From https://github.com/apache/hive 0e975f0..52c7c04 branch-1 -> origin/branch-1 + git reset --hard HEAD HEAD is now at ef7dc77 HIVE-13954: Parquet logs should go to STDERR (Prasanth Jayachandran reviewed by Gunther Hagleitner) + git clean -f -d + git checkout master Already on 'master' + git reset --hard origin/master HEAD is now at ef7dc77 HIVE-13954: Parquet logs should go to STDERR (Prasanth Jayachandran reviewed by Gunther Hagleitner) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12658595 - PreCommit-HIVE-MASTER-Build
        Hide
        Aihua Xu added a comment -

        Yu Gao Thanks for working on that. Seems you need to rebase. Do you know if HIVE-13020 would also fix the issue? I don' have enough knowledge on that.

        Show
        Aihua Xu added a comment - Yu Gao Thanks for working on that. Seems you need to rebase. Do you know if HIVE-13020 would also fix the issue? I don' have enough knowledge on that.
        Hide
        Aihua Xu added a comment -

        Seems we still need the fix for HIVE-7443 since HIVE-13020 is forzookeeper.

        Show
        Aihua Xu added a comment - Seems we still need the fix for HIVE-7443 since HIVE-13020 is forzookeeper.
        Hide
        Aihua Xu added a comment -

        Yu Gao I tried the patch but seems it doesn't fix the issue with the slightly different exception. Are you still working on the issue? If not, do you mind I take over to investigate?

        Caused by: org.ietf.jgss.GSSException, major code: 13, minor code: 0
                major string: Invalid credentials
                minor string: Cannot get credential from JAAS Subject for principal: default principal 
                at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:27)
                at com.ibm.security.jgss.mech.krb5.y.c(y.java:38)
                at com.ibm.security.jgss.mech.krb5.y.a(y.java:44)
                at com.ibm.security.jgss.mech.krb5.y.a(y.java:218)
                at com.ibm.security.jgss.mech.krb5.y.<init>(y.java:360)
        
        Show
        Aihua Xu added a comment - Yu Gao I tried the patch but seems it doesn't fix the issue with the slightly different exception. Are you still working on the issue? If not, do you mind I take over to investigate? Caused by: org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials minor string: Cannot get credential from JAAS Subject for principal: default principal at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:27) at com.ibm.security.jgss.mech.krb5.y.c(y.java:38) at com.ibm.security.jgss.mech.krb5.y.a(y.java:44) at com.ibm.security.jgss.mech.krb5.y.a(y.java:218) at com.ibm.security.jgss.mech.krb5.y.<init>(y.java:360)
        Hide
        Aihua Xu added a comment -

        Yu Gao I will continue the investigation. Just take over. If you have updates, feel free to take it back.

        Show
        Aihua Xu added a comment - Yu Gao I will continue the investigation. Just take over. If you have updates, feel free to take it back.
        Hide
        Aihua Xu added a comment -

        Attached the patch-2. Seems the first patch should work as well. I tested with oracle kinit and then ibm java, which won't work. But both IBM kinit and java will work. I moved the similar logic to hive-shim.

        Show
        Aihua Xu added a comment - Attached the patch-2. Seems the first patch should work as well. I tested with oracle kinit and then ibm java, which won't work. But both IBM kinit and java will work. I moved the similar logic to hive-shim.
        Hide
        Chaoyu Tang added a comment -

        Will this IBM JDK issue affect the case where kerberosAuthType is fromSubject? Also should we separate the logic for IBM JDK vs. others?

        Show
        Chaoyu Tang added a comment - Will this IBM JDK issue affect the case where kerberosAuthType is fromSubject? Also should we separate the logic for IBM JDK vs. others?
        Hide
        Aihua Xu added a comment -

        Thanks Chaoyu for reviewing the code.

        I changed to use UserGroupInformation.getCurrentUser() instead of getLoginUser() which also works for the case of fromSubject. So if the JDBC client does the login, then JDBC will get the credential from the subject. We don't separate logic for different JDKs, the same logic works for both of them.

        Show
        Aihua Xu added a comment - Thanks Chaoyu for reviewing the code. I changed to use UserGroupInformation.getCurrentUser() instead of getLoginUser() which also works for the case of fromSubject. So if the JDBC client does the login, then JDBC will get the credential from the subject. We don't separate logic for different JDKs, the same logic works for both of them.
        Hide
        Chaoyu Tang added a comment -

        LGTM, +1 pending on testing

        Show
        Chaoyu Tang added a comment - LGTM, +1 pending on testing
        Hide
        Aihua Xu added a comment -

        Reattach the patch-2 to trigger the build.

        Show
        Aihua Xu added a comment - Reattach the patch-2 to trigger the build.
        Hide
        Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12812169/HIVE-7443.2.patch

        ERROR: -1 due to no test(s) being added or modified.

        ERROR: -1 due to 31 failed/errored test(s), 10251 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
        org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
        org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
        org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
        org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.testHookContexts
        org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testConnection
        org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testIsValid
        org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testIsValidNeg
        org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNegativeProxyAuth
        org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNegativeTokenAuth
        org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testProxyAuth
        org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testTokenAuth
        org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection
        org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid
        org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg
        org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth
        org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
        org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth
        org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth
        org.apache.hive.minikdc.TestJdbcWithMiniKdc.testConnection
        org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValid
        org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValidNeg
        org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeProxyAuth
        org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
        org.apache.hive.minikdc.TestJdbcWithMiniKdc.testProxyAuth
        org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth
        org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookie
        org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.testAuthorization1
        org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.testAuthorization1
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/210/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/210/console
        Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-210/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 31 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12812169 - PreCommit-HIVE-MASTER-Build

        Show
        Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12812169/HIVE-7443.2.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 31 failed/errored test(s), 10251 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd org.apache.hive.minikdc.TestHs2HooksWithMiniKdc.testHookContexts org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testConnection org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testIsValid org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testIsValidNeg org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNegativeProxyAuth org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNegativeTokenAuth org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testProxyAuth org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testTokenAuth org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth org.apache.hive.minikdc.TestJdbcWithMiniKdc.testConnection org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValid org.apache.hive.minikdc.TestJdbcWithMiniKdc.testIsValidNeg org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeProxyAuth org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth org.apache.hive.minikdc.TestJdbcWithMiniKdc.testProxyAuth org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookie org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.testAuthorization1 org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthHttp.testAuthorization1 Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/210/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/210/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-210/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 31 tests failed This message is automatically generated. ATTACHMENT ID: 12812169 - PreCommit-HIVE-MASTER-Build
        Hide
        Aihua Xu added a comment -

        Attached patch-3: missing the return which causes the test failures.

        Show
        Aihua Xu added a comment - Attached patch-3: missing the return which causes the test failures.
        Hide
        Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12812483/HIVE-7443.3.patch

        ERROR: -1 due to no test(s) being added or modified.

        ERROR: -1 due to 6 failed/errored test(s), 10254 tests executed
        Failed tests:

        TestSchedulerQueue - did not produce a TEST-*.xml file
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
        org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
        org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
        org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/227/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/227/console
        Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-227/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 6 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12812483 - PreCommit-HIVE-MASTER-Build

        Show
        Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12812483/HIVE-7443.3.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 6 failed/errored test(s), 10254 tests executed Failed tests: TestSchedulerQueue - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/227/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/227/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-227/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12812483 - PreCommit-HIVE-MASTER-Build
        Hide
        Aihua Xu added a comment -

        The tests are not related. TestSchedulerQueue was failing with hdfs issue and it passed on my local test.

        Show
        Aihua Xu added a comment - The tests are not related. TestSchedulerQueue was failing with hdfs issue and it passed on my local test.
        Hide
        Aihua Xu added a comment -

        Pushed to master. Thanks Yu Gao for the original patch and Chaoyu for reviewing.

        Show
        Aihua Xu added a comment - Pushed to master. Thanks Yu Gao for the original patch and Chaoyu for reviewing.

          People

          • Assignee:
            Aihua Xu
            Reporter:
            Yu Gao
          • Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development