Hive
  1. Hive
  2. HIVE-6700

In some queries inputs are closed on Tez before the operator pipeline is flushed

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Group by operators won't flush their last row until operator is closed. In Tez it's possible that the input is already closed at this point.

      1. HIVE-6700.2.patch
        4 kB
        Gunther Hagleitner
      2. HIVE-6700.1.patch
        3 kB
        Gunther Hagleitner

        Issue Links

          Activity

          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12635657/HIVE-6700.2.patch

          ERROR: -1 due to 1 failed/errored test(s), 5440 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1919/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1919/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12635657

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12635657/HIVE-6700.2.patch ERROR: -1 due to 1 failed/errored test(s), 5440 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_bucketed_table Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1919/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1919/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12635657
          Hide
          Siddharth Seth added a comment -

          Looks good. +1

          Show
          Siddharth Seth added a comment - Looks good. +1
          Hide
          Gunther Hagleitner added a comment -

          .2 addresses sid's comment. good point need to include more in the try block.

          Show
          Gunther Hagleitner added a comment - .2 addresses sid's comment. good point need to include more in the try block.
          Hide
          Vikram Dixit K added a comment -

          LGTM +1 pending tests run.

          Show
          Vikram Dixit K added a comment - LGTM +1 pending tests run.
          Hide
          Siddharth Seth added a comment -

          Looks good, only question I have is whether the try block needs to enclose setting up Inputs in the ReduceProcessor.

            if(shuffleInputs.size() == 1){
                  //no merging of inputs required
                  kvsReader = (KeyValuesReader) shuffleInputs.get(0).getReader();
                }else {
                  //get a sort merged input
                  kvsReader = new InputMerger(shuffleInputs);
                }
          
          Show
          Siddharth Seth added a comment - Looks good, only question I have is whether the try block needs to enclose setting up Inputs in the ReduceProcessor. if (shuffleInputs.size() == 1){ //no merging of inputs required kvsReader = (KeyValuesReader) shuffleInputs.get(0).getReader(); } else { //get a sort merged input kvsReader = new InputMerger(shuffleInputs); }

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development