Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14383

SparkClientImpl should pass principal and keytab to spark-submit instead of calling kinit explicitely

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: Spark
    • Labels:
      None

      Description

      In https://github.com/apache/hive/blob/master/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java#L331, kinit is manually called before spark-submit is called. This is problematic if the spark session outlives the delegation token expiry period or the kerberos ticket expiry period. A better way to handle this would be to pass the keytab file location and principal name as parameters to the spark-submit command when kerberos auth is in use.

        Issue Links

          Activity

          Hide
          ctang.ma Chaoyu Tang added a comment - - edited

          The failed tests seem not related to this patch. Commit patch to 2.2.0 and 2.1.1. Thanks Xuefu Zhang for review.

          Show
          ctang.ma Chaoyu Tang added a comment - - edited The failed tests seem not related to this patch. Commit patch to 2.2.0 and 2.1.1. Thanks Xuefu Zhang for review.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12821048/HIVE-14383.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 9 failed/errored test(s), 10418 tests executed
          Failed tests:

          TestMsgBusConnection - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union
          org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
          org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns
          org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/711/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/711/console
          Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-711/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 9 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12821048 - PreCommit-HIVE-MASTER-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12821048/HIVE-14383.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 9 failed/errored test(s), 10418 tests executed Failed tests: TestMsgBusConnection - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_avro_non_nullable_union org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.stringifyValidTxns org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.testTxnRange Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/711/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/711/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-711/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed This message is automatically generated. ATTACHMENT ID: 12821048 - PreCommit-HIVE-MASTER-Build
          Hide
          xuefuz Xuefu Zhang added a comment -

          I think the old version of Spark doesn't do kinit for the client, and that's why kinit is called explicitly. If the bug is fixed in new version of Spark, this is certainly a welcome fix for Hive on Spark.

          Other than that, code looks good to me. +1

          Show
          xuefuz Xuefu Zhang added a comment - I think the old version of Spark doesn't do kinit for the client, and that's why kinit is called explicitly. If the bug is fixed in new version of Spark, this is certainly a welcome fix for Hive on Spark. Other than that, code looks good to me. +1
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Did some manual tests in kerberized cluster and the patch works fine.
          Xuefu Zhang, Mubashir Kazia, Chao Sun Could you review the patch? Thanks

          Show
          ctang.ma Chaoyu Tang added a comment - Did some manual tests in kerberized cluster and the patch works fine. Xuefu Zhang , Mubashir Kazia , Chao Sun Could you review the patch? Thanks

            People

            • Assignee:
              ctang.ma Chaoyu Tang
              Reporter:
              mkazia Mubashir Kazia
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development