Hive
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-7528

Support cluster by and distributed by [Spark Branch]

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      clustered by = distributed by + sort by, so this is related to HIVE-7527. If sort by is in place, the assumption is that we don't need to do anything about distributed by or clustered by. Still, we need to confirm and verify.

      1. HIVE-7528.spark.patch
        4 kB
        Rui Li
      2. HIVE-7528.1-spark.patch
        4 kB
        Brock Noland

        Issue Links

          Activity

          Xuefu Zhang created issue -
          Hide
          Xuefu Zhang added a comment -

          Assign it to you, Rui, as you're doing the relevant research on sortby.

          Show
          Xuefu Zhang added a comment - Assign it to you, Rui, as you're doing the relevant research on sortby.
          Xuefu Zhang made changes -
          Field Original Value New Value
          Assignee Rui Li [ lirui ]
          Hide
          Rui Li added a comment -

          I've tried simple distribute/cluster by queries and they can run successfully.

          Show
          Rui Li added a comment - I've tried simple distribute/cluster by queries and they can run successfully.
          Rui Li made changes -
          Attachment HIVE-7528.spark.patch [ 12662460 ]
          Hide
          Rui Li added a comment -

          Distribute/cluster by should work with the sort shuffler in place. This patch is mainly some refinement to the current shuffle code.

          Show
          Rui Li added a comment - Distribute/cluster by should work with the sort shuffler in place. This patch is mainly some refinement to the current shuffle code.
          Rui Li made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rui Li made changes -
          Remote Link This issue links to "RB request (Web Link)" [ 17080 ]
          Hide
          Brock Noland added a comment -

          Re-uploading the same patch under a name which allow pre-commit tests to run.

          Show
          Brock Noland added a comment - Re-uploading the same patch under a name which allow pre-commit tests to run.
          Brock Noland made changes -
          Attachment HIVE-7528.1-spark.patch [ 12662507 ]
          Brock Noland made changes -
          Summary Support cluster by and distributed by Support cluster by and distributed by [Spark Branch]
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12662507/HIVE-7528.1-spark.patch

          ERROR: -1 due to 3 failed/errored test(s), 5915 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/56/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/56/console
          Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-56/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12662507

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662507/HIVE-7528.1-spark.patch ERROR: -1 due to 3 failed/errored test(s), 5915 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2 Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/56/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/56/console Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-56/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12662507
          Hide
          Brock Noland added a comment -

          Thank you Rui Li for your contribution! I have committed this to spark!

          Can you open a follow-on JIRA to enable some tests for these queries?

          Show
          Brock Noland added a comment - Thank you Rui Li for your contribution! I have committed this to spark! Can you open a follow-on JIRA to enable some tests for these queries?
          Brock Noland made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s spark-branch [ 12327352 ]
          Resolution Fixed [ 1 ]
          Rui Li made changes -
          Link This issue relates to HIVE-7772 [ HIVE-7772 ]
          Hide
          Rui Li added a comment -

          Thanks Brock Noland I've created HIVE-7772 for it.

          Show
          Rui Li added a comment - Thanks Brock Noland I've created HIVE-7772 for it.
          Hide
          Brock Noland added a comment -

          Thank you!!

          Show
          Brock Noland added a comment - Thank you!!
          Xuefu Zhang made changes -
          Fix Version/s 1.1.0 [ 12329363 ]
          Fix Version/s spark-branch [ 12327352 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          21d 10h 44m 1 Rui Li 18/Aug/14 10:22
          Patch Available Patch Available Resolved Resolved
          9h 43m 1 Brock Noland 18/Aug/14 20:05

            People

            • Assignee:
              Rui Li
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development