Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12947

SMB join in tez has ClassCastException when container reuse is on

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Tez
    • Labels:
      None

      Description

      SMB join in tez has multiple work items that are connected based on input tag followed by input initialization etc. In case of container re-use, what ends up happening is that we try to reconnect the work items and fail. If we try to work around that issue by recognizing somehow that the cache was in play, we will run into other initialization issues with respect to record readers. So the plan is to disable caching of the SMB work items by clearing out during the close phase.

      java.lang.RuntimeException: Map operator initialization failed
              at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:247)
              at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
              at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
              at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
              at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.FileSinkOperator cannot be cast to org.apache.hadoop.hive.ql.exec.DummyStoreOperator
              at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:300)
              at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:302)
              at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:302)
              at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:302)
              at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:302)
              at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:189)
              ... 15 more
      

        Attachments

        1. HIVE-12947.1.patch
          43 kB
          Vikram Dixit K
        2. HIVE-12947.2.patch
          48 kB
          Vikram Dixit K
        3. HIVE-12947.3.patch
          48 kB
          Vikram Dixit K
        4. HIVE-12947.4.patch
          45 kB
          Vikram Dixit K

          Activity

            People

            • Assignee:
              vikram.dixit Vikram Dixit K
              Reporter:
              vikram.dixit Vikram Dixit K
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: