Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15485

Investigate the DoAs failure in HoS

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: Spark
    • Labels:
      None

      Description

      With DoAs enabled, HoS failed with following errors:

      Exception in thread "main" org.apache.hadoop.security.AccessControlException: systest tries to renew a token with renewer hive
      	at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:484)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:7543)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewDelegationToken(NameNodeRpcServer.java:555)
      	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.renewDelegationToken(AuthorizationProviderProxyClientProtocol.java:674)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:999)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2141)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1783)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2135)
      

      It is related to the change from HIVE-14383. It looks like that SparkSubmit logs in Kerberos with passed in hive principal/keytab and then tries to create a hdfs delegation token for user systest with renewer hive.

      1. HIVE-15485.1.patch
        3 kB
        Chaoyu Tang
      2. HIVE-15485.2.patch
        3 kB
        Chaoyu Tang
      3. HIVE-15485.patch
        3 kB
        Chaoyu Tang

        Issue Links

          Activity

          Hide
          ctang.ma Chaoyu Tang added a comment -

          HIVE-14383 is the right way to renew the delegation token for a long running HoS session. Spark needs the principal/keytab passed in via --principal and --keytab options, and does the renewal by copying the keytab to the cluster and handling login to kerberos inside the application.
          But the option --principal, --keytab could not work with --proxy-user in spark-submit.sh as suggested by Marcelo Vanzin, so at this moment we could support either the token renewal or the impersonation, but not both.

          Show
          ctang.ma Chaoyu Tang added a comment - HIVE-14383 is the right way to renew the delegation token for a long running HoS session. Spark needs the principal/keytab passed in via --principal and --keytab options, and does the renewal by copying the keytab to the cluster and handling login to kerberos inside the application. But the option --principal, --keytab could not work with --proxy-user in spark-submit.sh as suggested by Marcelo Vanzin , so at this moment we could support either the token renewal or the impersonation, but not both.
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Given that we are not able to support both doAs and delegation token renewal in Spark at this moment (see comments in SPARK-5493, SPARK-19143 etc), and doAs is more common case in Hive, so when doAs is enabled, we will use kinit instead of passing the principal/keytab to Spark. I could not thought of other ways to make both work. Xuefu Zhang, do you have any thought? If you agree on that, could you help review the patch? Thanks.

          Show
          ctang.ma Chaoyu Tang added a comment - Given that we are not able to support both doAs and delegation token renewal in Spark at this moment (see comments in SPARK-5493 , SPARK-19143 etc), and doAs is more common case in Hive, so when doAs is enabled, we will use kinit instead of passing the principal/keytab to Spark. I could not thought of other ways to make both work. Xuefu Zhang , do you have any thought? If you agree on that, could you help review the patch? Thanks.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12849217/HIVE-15485.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 7 failed/errored test(s), 10983 tests executed
          Failed tests:

          TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
          TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=98)
          	[ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,order2.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q]
          org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer] (batchId=151)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part] (batchId=149)
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3169/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3169/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3169/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12849217 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849217/HIVE-15485.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 7 failed/errored test(s), 10983 tests executed Failed tests: TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=98) [ptf_general_queries.q,auto_join_reordering_values.q,sample2.q,join1.q,decimal_join.q,mapjoin_subquery2.q,join32_lessSize.q,mapjoin1.q,order2.q,skewjoinopt18.q,union_remove_18.q,join25.q,groupby9.q,bucketsortoptimize_insert_6.q,ctas.q] org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part] (batchId=149) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3169/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3169/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3169/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12849217 - PreCommit-HIVE-Build
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Xuefu Zhang, Chao Sun, Jimmy Xiang, could you review the patch to see if it makes sense, so that we can at lease unblock the doAs issue? Thanks

          Show
          ctang.ma Chaoyu Tang added a comment - Xuefu Zhang , Chao Sun , Jimmy Xiang , could you review the patch to see if it makes sense, so that we can at lease unblock the doAs issue? Thanks
          Hide
          jxiang Jimmy Xiang added a comment -

          Could you put the two changes in your patch in the same place to make it a little easier to understand?

          Show
          jxiang Jimmy Xiang added a comment - Could you put the two changes in your patch in the same place to make it a little easier to understand?
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Thanks, Jimmy, for looking into this. When doAs is enabled, we use kinit to login Kerberos and this kinit command need be put before the spark-submit.sh, but when doAs is disabled, the principal/keytab should be after spark-submit.sh as its parameters. I was also wondering how to combine these two changes into one, but have not found a good way. Any suggestion?

          Show
          ctang.ma Chaoyu Tang added a comment - Thanks, Jimmy, for looking into this. When doAs is enabled, we use kinit to login Kerberos and this kinit command need be put before the spark-submit.sh, but when doAs is disabled, the principal/keytab should be after spark-submit.sh as its parameters. I was also wondering how to combine these two changes into one, but have not found a good way. Any suggestion?
          Hide
          jxiang Jimmy Xiang added a comment -

          For doAs, add kinit etc to the beginning of the list; for the other add principal etc at the end. If you are concerned with performance, will LinkedList be better than ArrayList here?

          By the way, should keyTabFile + ";" be two argvs?

          Show
          jxiang Jimmy Xiang added a comment - For doAs, add kinit etc to the beginning of the list; for the other add principal etc at the end. If you are concerned with performance, will LinkedList be better than ArrayList here? By the way, should keyTabFile + ";" be two argvs?
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Thanks Jimmy Xiang. Please take a look to see if it is you suggested. The keytab + ";" do not have to be in two argv. Thanks

          Show
          ctang.ma Chaoyu Tang added a comment - Thanks Jimmy Xiang . Please take a look to see if it is you suggested. The keytab + ";" do not have to be in two argv. Thanks
          Hide
          jxiang Jimmy Xiang added a comment -

          Thanks for making the change. Looks good to me. +1

          Show
          jxiang Jimmy Xiang added a comment - Thanks for making the change. Looks good to me. +1
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12849642/HIVE-15485.1.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 43 failed/errored test(s), 10419 tests executed
          Failed tests:

          TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
          org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159)
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=100)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=101)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=102)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=103)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=104)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=105)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=106)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=107)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=108)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=109)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=110)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=111)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=112)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=113)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=114)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=115)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=116)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=117)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=118)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=119)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=120)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=121)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=122)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=123)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=124)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=125)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=126)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=127)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=128)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=129)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=130)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=131)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=132)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=133)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=95)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=96)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=97)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=98)
          org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=99)
          org.apache.hive.spark.client.TestSparkClient.testRemoteClient (batchId=278)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3226/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3226/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3226/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 43 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12849642 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849642/HIVE-15485.1.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 43 failed/errored test(s), 10419 tests executed Failed tests: TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=100) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=101) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=102) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=103) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=104) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=105) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=106) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=107) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=108) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=109) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=110) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=111) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=112) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=113) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=114) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=115) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=116) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=117) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=118) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=119) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=120) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=121) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=122) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=123) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=124) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=125) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=126) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=127) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=128) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=129) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=130) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=131) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=132) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=133) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=95) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=96) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=97) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=98) org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=99) org.apache.hive.spark.client.TestSparkClient.testRemoteClient (batchId=278) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3226/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3226/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3226/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 43 tests failed This message is automatically generated. ATTACHMENT ID: 12849642 - PreCommit-HIVE-Build
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Fixed the test failures.

          Show
          ctang.ma Chaoyu Tang added a comment - Fixed the test failures.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12849796/HIVE-15485.2.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 8 failed/errored test(s), 10973 tests executed
          Failed tests:

          TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
          TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=109)
          	[union_remove_1.q,ppd_outer_join2.q,date_udf.q,groupby1_noskew.q,join20.q,smb_mapjoin_13.q,groupby_rollup1.q,temp_table_gb1.q,vector_string_concat.q,smb_mapjoin_6.q,metadata_only_queries.q,auto_sortmerge_join_12.q,groupby_bigdata.q,groupby3_map_multi_distinct.q,innerjoin.q]
          TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=125)
          	[table_access_keys_stats.q,bucketmapjoin11.q,auto_join4.q,mapjoin_decimal.q,join34.q,nullgroup.q,mergejoins_mixed.q,sort.q,stats8.q,auto_join28.q,join17.q,union17.q,skewjoinopt11.q,groupby1_map.q,load_dyn_part11.q]
          org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple] (batchId=153)
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3240/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3240/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3240/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12849796 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849796/HIVE-15485.2.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 8 failed/errored test(s), 10973 tests executed Failed tests: TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=109) [union_remove_1.q,ppd_outer_join2.q,date_udf.q,groupby1_noskew.q,join20.q,smb_mapjoin_13.q,groupby_rollup1.q,temp_table_gb1.q,vector_string_concat.q,smb_mapjoin_6.q,metadata_only_queries.q,auto_sortmerge_join_12.q,groupby_bigdata.q,groupby3_map_multi_distinct.q,innerjoin.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=125) [table_access_keys_stats.q,bucketmapjoin11.q,auto_join4.q,mapjoin_decimal.q,join34.q,nullgroup.q,mergejoins_mixed.q,sort.q,stats8.q,auto_join28.q,join17.q,union17.q,skewjoinopt11.q,groupby1_map.q,load_dyn_part11.q] org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple] (batchId=153) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3240/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3240/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3240/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12849796 - PreCommit-HIVE-Build
          Hide
          ctang.ma Chaoyu Tang added a comment -

          The test failures are not related to this patch.

          Show
          ctang.ma Chaoyu Tang added a comment - The test failures are not related to this patch.
          Hide
          xuefuz Xuefu Zhang added a comment -

          Sorry for my late reply. (I'm currently OOO.) The patch looks good to me too. While these test failures are caused by something else, the fact that some Spark tests didn't actually run is a little concern. Is there a way to validate these tests locally?

          Show
          xuefuz Xuefu Zhang added a comment - Sorry for my late reply. (I'm currently OOO.) The patch looks good to me too. While these test failures are caused by something else, the fact that some Spark tests didn't actually run is a little concern. Is there a way to validate these tests locally?
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Thanks Xuefu Zhang for looking into this patch. I have attached a new patch to trigger the build test. The patch has been verified in my local environment. The TestSparkCliDriver tests ran successfully. I also manually validated cases (e.g. kerberos w/ or w/o doAs) in my local cluster, they all passed.

          Show
          ctang.ma Chaoyu Tang added a comment - Thanks Xuefu Zhang for looking into this patch. I have attached a new patch to trigger the build test. The patch has been verified in my local environment. The TestSparkCliDriver tests ran successfully. I also manually validated cases (e.g. kerberos w/ or w/o doAs) in my local cluster, they all passed.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12849876/HIVE-15485.2.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 7 failed/errored test(s), 11003 tests executed
          Failed tests:

          TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
          org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple] (batchId=147)
          org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=140)
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] (batchId=93)
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93)
          org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223)
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3249/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3249/console
          Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3249/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12849876 - PreCommit-HIVE-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849876/HIVE-15485.2.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 7 failed/errored test(s), 11003 tests executed Failed tests: TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=140) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] (batchId=93) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3249/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3249/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3249/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12849876 - PreCommit-HIVE-Build
          Hide
          ctang.ma Chaoyu Tang added a comment -

          The failed tests are not related to this patch.

          Show
          ctang.ma Chaoyu Tang added a comment - The failed tests are not related to this patch.
          Hide
          xuefuz Xuefu Zhang added a comment -

          +1. Thanks for looking into this, Chaoyu Tang.

          Show
          xuefuz Xuefu Zhang added a comment - +1. Thanks for looking into this, Chaoyu Tang .
          Hide
          ctang.ma Chaoyu Tang added a comment -

          Committed to 2.2.0. Thanks Xuefu Zhang and Jimmy Xiang for review.

          Show
          ctang.ma Chaoyu Tang added a comment - Committed to 2.2.0. Thanks Xuefu Zhang and Jimmy Xiang for review.

            People

            • Assignee:
              ctang.ma Chaoyu Tang
              Reporter:
              ctang.ma Chaoyu Tang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development