Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: Spark
    • Labels:

      Description

      My understanding is that we don't need to do anything special for this. However, this needs to be verified and tested.

      1. HIVE-7580.patch
        475 kB
        Chinna Rao Lalam
      2. HIVE-7580.1-spark.patch
        467 kB
        Chinna Rao Lalam

        Issue Links

          Activity

          Hide
          Xuefu Zhang added a comment -

          Patch committed to spark branch. Thanks to Chinna for the contribution.

          Show
          Xuefu Zhang added a comment - Patch committed to spark branch. Thanks to Chinna for the contribution.
          Hide
          Xuefu Zhang added a comment -

          +1, will commit shortly.

          Show
          Xuefu Zhang added a comment - +1, will commit shortly.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12666441/HIVE-7580.1-spark.patch

          ERROR: -1 due to 2 failed/errored test(s), 6291 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/110/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/110/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-110/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12666441

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12666441/HIVE-7580.1-spark.patch ERROR: -1 due to 2 failed/errored test(s), 6291 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/110/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/110/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-110/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12666441
          Hide
          Chinna Rao Lalam added a comment -

          Hi Brock Noland,

          Updated the patch with test properties..

          Show
          Chinna Rao Lalam added a comment - Hi Brock Noland , Updated the patch with test properties..
          Hide
          Brock Noland added a comment -

          Chinna Rao Lalam I think this is missing the test properties update?

          Show
          Brock Noland added a comment - Chinna Rao Lalam I think this is missing the test properties update?
          Hide
          Chinna Rao Lalam added a comment -

          Patch contains passed test cases.

          Show
          Chinna Rao Lalam added a comment - Patch contains passed test cases.
          Hide
          Chinna Rao Lalam added a comment -

          Verified the below tests and all the tests are passed except load_dyn_part1.q, load_dyn_part8.q

          load_dyn_part1.q,
          load_dyn_part2.q,
          load_dyn_part3.q,
          load_dyn_part4.q,
          load_dyn_part5.q,
          load_dyn_part6.q,
          load_dyn_part7.q,
          load_dyn_part8.q,
          load_dyn_part9,
          load_dyn_part10.q,
          load_dyn_part11.q,
          load_dyn_part12.q,
          load_dyn_part13.q,
          load_dyn_part.14,
          load_dyn_part15.q
          

          To enable the tests for dynamic partitions considered below tests(referred from tez)

          load_dyn_part1.q,
          load_dyn_part2.q,
          load_dyn_part3.q,
          dynpart_sort_optimization.q,
          dynpart_sort_opt_vectorization.q
          

          Here the below tests are failing

          load_dyn_part1.q,
          oad_dyn_part8.q,
          dynpart_sort_optimization.q,
          dynpart_sort_opt_vectorization.q 
          

          For these 4 test cases we have issues. I will add these tests in those jira's and i will work.

          load_dyn_part1.q,load_dyn_part8.q both the tests contains multi-inserts. Need to test after HIVE-7503 is fixed.

          dynpart_sort_opt_vectorization.q is related to vectorization. Need to test after HIVE-7794 is fixed.

          dynpart_sort_optimization.q is hitting same exception as HIVE-7843.

          Show
          Chinna Rao Lalam added a comment - Verified the below tests and all the tests are passed except load_dyn_part1.q, load_dyn_part8.q load_dyn_part1.q, load_dyn_part2.q, load_dyn_part3.q, load_dyn_part4.q, load_dyn_part5.q, load_dyn_part6.q, load_dyn_part7.q, load_dyn_part8.q, load_dyn_part9, load_dyn_part10.q, load_dyn_part11.q, load_dyn_part12.q, load_dyn_part13.q, load_dyn_part.14, load_dyn_part15.q To enable the tests for dynamic partitions considered below tests(referred from tez) load_dyn_part1.q, load_dyn_part2.q, load_dyn_part3.q, dynpart_sort_optimization.q, dynpart_sort_opt_vectorization.q Here the below tests are failing load_dyn_part1.q, oad_dyn_part8.q, dynpart_sort_optimization.q, dynpart_sort_opt_vectorization.q For these 4 test cases we have issues. I will add these tests in those jira's and i will work. load_dyn_part1.q,load_dyn_part8.q both the tests contains multi-inserts. Need to test after HIVE-7503 is fixed. dynpart_sort_opt_vectorization.q is related to vectorization. Need to test after HIVE-7794 is fixed. dynpart_sort_optimization.q is hitting same exception as HIVE-7843 .
          Hide
          Brock Noland added a comment -

          Venki hit assertion errors in HIVE-7843.

          Show
          Brock Noland added a comment - Venki hit assertion errors in HIVE-7843 .
          Hide
          Xuefu Zhang added a comment -

          Hi Chinna Rao Lalam, are you activelyt working on this? If not, we'd like to see if someone else has the bandwidth. Thanks.

          Show
          Xuefu Zhang added a comment - Hi Chinna Rao Lalam , are you activelyt working on this? If not, we'd like to see if someone else has the bandwidth. Thanks.
          Hide
          Brock Noland added a comment -

          We need to enable dynpart_sort_opt_vectorization.q,\
          dynpart_sort_optimization.q,\ as part of this work since Tez executes them.

          Show
          Brock Noland added a comment - We need to enable dynpart_sort_opt_vectorization.q,\ dynpart_sort_optimization.q,\ as part of this work since Tez executes them.
          Hide
          Xuefu Zhang added a comment -

          I'd think they are different. Here's Hive's dynamic partitioning:

          https://cwiki.apache.org/confluence/display/Hive/DynamicPartitions.

          HIVE-7657 concerns about determining the number of partitions (reducers) at shuffle.

          Show
          Xuefu Zhang added a comment - I'd think they are different. Here's Hive's dynamic partitioning: https://cwiki.apache.org/confluence/display/Hive/DynamicPartitions . HIVE-7657 concerns about determining the number of partitions (reducers) at shuffle.
          Hide
          Chengxiang Li added a comment -

          Chinna Rao Lalam, Is this task duplicate with HIVE-7567?

          Show
          Chengxiang Li added a comment - Chinna Rao Lalam , Is this task duplicate with HIVE-7567 ?

            People

            • Assignee:
              Chinna Rao Lalam
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development