Hive
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7503

Support Hive's multi-table insert query with Spark [Spark Branch]

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:

      Description

      For Hive's multi insert query (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML), there may be an MR job for each insert. When we achieve this with Spark, it would be nice if all the inserts can happen concurrently.

      It seems that this functionality isn't available in Spark. To make things worse, the source of the insert may be re-computed unless it's staged. Even with this, the inserts will happen sequentially, making the performance suffer.

      This task is to find out what takes in Spark to enable this without requiring staging the source and sequential insertion. If this has to be solved in Hive, find out an optimum way to do this.

      1. HIVE-7503.1-spark.patch
        24 kB
        Chao Sun
      2. HIVE-7503.2-spark.patch
        26 kB
        Chao Sun
      3. HIVE-7503.3-spark.patch
        25 kB
        Chao Sun
      4. HIVE-7503.4-spark.patch
        27 kB
        Chao Sun
      5. HIVE-7503.5-spark.patch
        27 kB
        Chao Sun
      6. HIVE-7503.6-spark.patch
        34 kB
        Chao Sun
      7. HIVE-7503.7-spark.patch
        33 kB
        Chao Sun
      8. HIVE-7503.8-spark.patch
        61 kB
        Chao Sun
      9. HIVE-7503.9-spark.patch
        61 kB
        Chao Sun

        Issue Links

          Activity

          Hide
          Xuefu Zhang added a comment -

          Assigned to myself for initial research.

          Show
          Xuefu Zhang added a comment - Assigned to myself for initial research.
          Hide
          Xuefu Zhang added a comment - - edited

          As it's unlikely to get SPARK-2688 in a short period of time and since HIVE-7525 has shown that we can submit Spark job concurrently, I'd like to propose the following backup plan:

          1. The multi-insert plan can be decomposed into 1 + N plans, where N is the number of inserts.
          2. The 1 plan is the one generating the data source for all the inserts. The plan may not be necessary if the source is a table, but in general, the source coming from a job.
          3. For each insert, there will be one plan to insert the data. Thus, N inserts equals to N plans. The source to the plans is the data generated from the 1 job.
          4. We first run the 1 plan as a Spark job, which emits the data source.
          5. Then, we call checkpoint on the rdd from #4.
          6. Lastly, we launch N jobs concurrently, each with the above checkpointed rdd as input.

          While not ideal and probably not efficient, this approach should performs better than running 1 + N job sequentially.

          The idea was from Sean Owen, and verified by Chao Sun via HIVE-7525.

          Reynold Xin, what's your thought on this approach? Do you have any other suggestions?

          Show
          Xuefu Zhang added a comment - - edited As it's unlikely to get SPARK-2688 in a short period of time and since HIVE-7525 has shown that we can submit Spark job concurrently, I'd like to propose the following backup plan: 1. The multi-insert plan can be decomposed into 1 + N plans, where N is the number of inserts. 2. The 1 plan is the one generating the data source for all the inserts. The plan may not be necessary if the source is a table, but in general, the source coming from a job. 3. For each insert, there will be one plan to insert the data. Thus, N inserts equals to N plans. The source to the plans is the data generated from the 1 job. 4. We first run the 1 plan as a Spark job, which emits the data source. 5. Then, we call checkpoint on the rdd from #4. 6. Lastly, we launch N jobs concurrently, each with the above checkpointed rdd as input. While not ideal and probably not efficient, this approach should performs better than running 1 + N job sequentially. The idea was from Sean Owen , and verified by Chao Sun via HIVE-7525 . Reynold Xin , what's your thought on this approach? Do you have any other suggestions?
          Hide
          Xuefu Zhang added a comment -

          Unassigned it from me, as I will not be able to work on this in the next few weeks. (I will be in the discussions, though.) Please feel free to take it if anyone likes to work on this.

          Show
          Xuefu Zhang added a comment - Unassigned it from me, as I will not be able to work on this in the next few weeks. (I will be in the discussions, though.) Please feel free to take it if anyone likes to work on this.
          Hide
          Xuefu Zhang added a comment -

          I just chatted with Reynold Xin about above proposal, and we are in the same page. Please proceed with the proposal.

          Show
          Xuefu Zhang added a comment - I just chatted with Reynold Xin about above proposal, and we are in the same page. Please proceed with the proposal.
          Hide
          Chao Sun added a comment -

          Initial patch for this JIRA. Currently it only works for simple queries like:

          from src
          insert overwrite table tgt1 select key group by key
          insert overwrite table tgt2 select value group by value
          

          if the from_statement contains complicated queries like union, then
          it doesn't work. The case for union is a little bit tricky.

          Also, I need to feed it through related *.q files.

          Show
          Chao Sun added a comment - Initial patch for this JIRA. Currently it only works for simple queries like: from src insert overwrite table tgt1 select key group by key insert overwrite table tgt2 select value group by value if the from_statement contains complicated queries like union, then it doesn't work. The case for union is a little bit tricky. Also, I need to feed it through related *.q files.
          Hide
          Chao Sun added a comment -

          This patch will only change plan in the multi-table insertion case.
          I also modified a bit on GenSparkUtils::removeUnionOperators, which may overlap with HIVE-7870.

          Show
          Chao Sun added a comment - This patch will only change plan in the multi-table insertion case. I also modified a bit on GenSparkUtils::removeUnionOperators , which may overlap with HIVE-7870 .
          Hide
          Xuefu Zhang added a comment -

          Chao Sun Thanks for the patch which looks good. I left some comments on the RB. One mroe thing: when you create review request, could you also fill in the bug number to the JIRA number and review group to hive? Thanks.

          Show
          Xuefu Zhang added a comment - Chao Sun Thanks for the patch which looks good. I left some comments on the RB. One mroe thing: when you create review request, could you also fill in the bug number to the JIRA number and review group to hive? Thanks.
          Hide
          Chao Sun added a comment -

          Xuefu Zhang OK, I've updated it and will remember next time.

          Show
          Chao Sun added a comment - Xuefu Zhang OK, I've updated it and will remember next time.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12666824/HIVE-7503.2-spark.patch

          ERROR: -1 due to 79 failed/errored test(s), 6291 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_context_ngrams
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_disable_merge_for_bucketing
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join14
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join17
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join25
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join26
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join27
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join7
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join8
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join9
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part10
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part14
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part9
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample7
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union25
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_10
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_11
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_15
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_16
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_17
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_20
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_21
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_24
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_25
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_5
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_7
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_8
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_9
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/113/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/113/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-113/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 79 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12666824

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12666824/HIVE-7503.2-spark.patch ERROR: -1 due to 79 failed/errored test(s), 6291 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_context_ngrams org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join14 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join17 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join25 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join26 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join27 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join7 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join8 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join9 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part14 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part9 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample7 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union25 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_11 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_16 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_17 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_20 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_21 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_24 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_25 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_7 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_8 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_9 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/113/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/113/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-113/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 79 tests failed This message is automatically generated. ATTACHMENT ID: 12666824
          Hide
          Chao Sun added a comment -

          I guess this is because generated plans are not completely identical - some numberings are different.

          Show
          Chao Sun added a comment - I guess this is because generated plans are not completely identical - some numberings are different.
          Hide
          Chao Sun added a comment -

          Addressing comments from RB. Also, trying to make the tester happy by adding a dependency task.

          The task is for multi-table insertion. However, with current implementation, it is generated even in cases that are not multi-table insertion. We may open a separate JIRA for this minor issue.

          Show
          Chao Sun added a comment - Addressing comments from RB. Also, trying to make the tester happy by adding a dependency task. The task is for multi-table insertion. However, with current implementation, it is generated even in cases that are not multi-table insertion. We may open a separate JIRA for this minor issue.
          Hide
          Xuefu Zhang added a comment -

          We may open a separate JIRA for this minor issue.

          Yes, please open a followup JIRA fix the issue if this is the only test failure.

          Show
          Xuefu Zhang added a comment - We may open a separate JIRA for this minor issue. Yes, please open a followup JIRA fix the issue if this is the only test failure.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12666885/HIVE-7503.3-spark.patch

          ERROR: -1 due to 17 failed/errored test(s), 6291 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_udf
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/114/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/114/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-114/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 17 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12666885

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12666885/HIVE-7503.3-spark.patch ERROR: -1 due to 17 failed/errored test(s), 6291 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_udf org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/114/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/114/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-114/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 17 tests failed This message is automatically generated. ATTACHMENT ID: 12666885
          Hide
          Chao Sun added a comment -

          I forgot to put a change on SparkProcessAnalyzeTable.java in the last patch. Also, some test fails are caused by duplicated Stats-Aggr Operators from the issue in HIVE-7870.

          Show
          Chao Sun added a comment - I forgot to put a change on SparkProcessAnalyzeTable.java in the last patch. Also, some test fails are caused by duplicated Stats-Aggr Operators from the issue in HIVE-7870 .
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12666955/HIVE-7503.4-spark.patch

          ERROR: -1 due to 15 failed/errored test(s), 6291 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/115/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/115/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-115/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 15 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12666955

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12666955/HIVE-7503.4-spark.patch ERROR: -1 due to 15 failed/errored test(s), 6291 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/115/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/115/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-115/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 15 tests failed This message is automatically generated. ATTACHMENT ID: 12666955
          Hide
          Chao Sun added a comment -

          All the failed diffs are caused by either duplicated filesinks or multi-table insertion.
          I should probably start updating the related .q files for the latter.

          Show
          Chao Sun added a comment - All the failed diffs are caused by either duplicated filesinks or multi-table insertion. I should probably start updating the related .q files for the latter.
          Hide
          Chao Sun added a comment -

          This patch retains the duplicated filesinks (as an alternative).

          Show
          Chao Sun added a comment - This patch retains the duplicated filesinks (as an alternative).
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12667195/HIVE-7503.5-spark.patch

          ERROR: -1 due to 17 failed/errored test(s), 6343 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/118/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/118/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-118/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 17 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12667195

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12667195/HIVE-7503.5-spark.patch ERROR: -1 due to 17 failed/errored test(s), 6343 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union10 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union28 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union29 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union6 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/118/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/118/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-118/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 17 tests failed This message is automatically generated. ATTACHMENT ID: 12667195
          Hide
          Chao Sun added a comment -

          This patch mainly changes the way for detecting multi-table insertion.

          Show
          Chao Sun added a comment - This patch mainly changes the way for detecting multi-table insertion.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12669756/HIVE-7503.6-spark.patch

          ERROR: -1 due to 7 failed/errored test(s), 6437 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_context_ngrams
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/138/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/138/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-138/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12669756

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12669756/HIVE-7503.6-spark.patch ERROR: -1 due to 7 failed/errored test(s), 6437 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_context_ngrams org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/138/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/138/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-138/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12669756
          Hide
          Chao Sun added a comment -

          All failed spark tests (4 of them) have multi-insertion queries. The difference is in the plans generated, but not the results.
          The other 3 failed tests are not related to this patch, I think.

          Show
          Chao Sun added a comment - All failed spark tests (4 of them) have multi-insertion queries. The difference is in the plans generated, but not the results. The other 3 failed tests are not related to this patch, I think.
          Hide
          Chao Sun added a comment -

          Addressing suggestions from RB.

          Show
          Chao Sun added a comment - Addressing suggestions from RB.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12670148/HIVE-7503.7-spark.patch

          ERROR: -1 due to 6 failed/errored test(s), 6437 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
          org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/139/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/139/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-139/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12670148

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12670148/HIVE-7503.7-spark.patch ERROR: -1 due to 6 failed/errored test(s), 6437 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_6 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/139/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/139/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-139/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12670148
          Hide
          Chao Sun added a comment -

          Added updated qfiles.

          Show
          Chao Sun added a comment - Added updated qfiles.
          Hide
          Chao Sun added a comment -

          I forgot Brock Noland's suggestions in RB. Now addressing them.

          Show
          Chao Sun added a comment - I forgot Brock Noland 's suggestions in RB. Now addressing them.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12670162/HIVE-7503.8-spark.patch

          ERROR: -1 due to 3 failed/errored test(s), 6437 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hive.jdbc.miniHS2.TestMiniHS2.testConfInSession
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/140/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/140/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-140/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12670162

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12670162/HIVE-7503.8-spark.patch ERROR: -1 due to 3 failed/errored test(s), 6437 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hive.jdbc.miniHS2.TestMiniHS2.testConfInSession Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/140/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/140/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-140/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12670162
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12670163/HIVE-7503.9-spark.patch

          ERROR: -1 due to 3 failed/errored test(s), 6437 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/141/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/141/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-141/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12670163

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12670163/HIVE-7503.9-spark.patch ERROR: -1 due to 3 failed/errored test(s), 6437 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/141/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/141/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-141/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12670163
          Hide
          Brock Noland added a comment -

          lgtm, I will let Xuefu make the commit decision.

          Show
          Brock Noland added a comment - lgtm, I will let Xuefu make the commit decision.
          Hide
          Xuefu Zhang added a comment -

          +1

          Show
          Xuefu Zhang added a comment - +1
          Hide
          Xuefu Zhang added a comment -

          Patch committed to Spark branch. Thanks to Chao for the contribution.

          Show
          Xuefu Zhang added a comment - Patch committed to Spark branch. Thanks to Chao for the contribution.

            People

            • Assignee:
              Chao Sun
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development