Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7926 long-lived daemons for query fragment execution, I/O and caching
  3. HIVE-11016

MiniTez mergejoin test fails with Tez input error (issue in merge join under certain conditions)

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.0.1, 1.2.1, 2.0.0
    • None
    • None

    Description

      Didn't spend a lot of time investigating, but from the code it looks like we shouldn't be calling it after false at least on this path (after false from next, pushRecord returns false, which causes fetchDone to be set for the tag; and fetchOneRow is not called if that is set; should be ok unless tags are messed up?)

      2015-06-15 17:28:17,272 ERROR [main]: SessionState (SessionState.java:printError(984)) - Vertex failed, vertexName=Reducer 2, vertexId=vertex_1434414363282_0002_17_03, diagnostics=[Task failed, taskId=task_1434414363282_0002_17_03_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1434414363282_0002_17_03_000002_0:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.RuntimeException: java.io.IOException: Please check if you are invoking moveToNext() even after it returned false.
      	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:181)
      	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:146)
      	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:349)
      	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
      	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
      	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.RuntimeException: java.io.IOException: Please check if you are invoking moveToNext() even after it returned false.
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:338)
      	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
      	... 14 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.io.IOException: Please check if you are invoking moveToNext() even after it returned false.
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:412)
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:380)
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinFinalLeftData(CommonMergeJoinOperator.java:449)
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.closeOp(CommonMergeJoinOperator.java:389)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:651)
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:314)
      	... 15 more
      Caused by: java.lang.RuntimeException: java.io.IOException: Please check if you are invoking moveToNext() even after it returned false.
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:302)
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:404)
      	... 20 more
      Caused by: java.io.IOException: Please check if you are invoking moveToNext() even after it returned false.
      	at org.apache.tez.runtime.library.common.ValuesIterator.hasCompletedProcessing(ValuesIterator.java:223)
      	at org.apache.tez.runtime.library.common.ValuesIterator.moveToNext(ValuesIterator.java:105)
      	at org.apache.tez.runtime.library.input.OrderedGroupedKVInput$OrderedGroupedKeyValuesReader.next(OrderedGroupedKVInput.java:308)
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:260)
      	... 21 more
      

      Attachments

        1. HIVE-11016.01.patch
          6 kB
          Sergey Shelukhin
        2. HIVE-11016.patch
          15 kB
          Sergey Shelukhin

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sershe Sergey Shelukhin Assign to me
            sershe Sergey Shelukhin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment