Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4816

Read a null scalar causing a Tez failure

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following script fail:

      a = load 'studenttab10k' as (name:chararray, age:int, gpa:double);
      b = filter a by age > 100;
      c = group b all;
      d = foreach c generate COUNT(b.gpa) as count;
      e = foreach a generate (d.count IS NOT NULL? d.count : 0l);
      dump e;
      

      Error stack:

      Failure while running task:org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.io.IOException: Please check if you are invoking next() even after it returned false. For usage, please refer to KeyValueReader javadocs
      	at org.apache.pig.backend.hadoop.executionengine.tez.plan.udf.ReadScalarsTez.attachInputs(ReadScalarsTez.java:96)
      	at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.initializeInputs(PigProcessor.java:295)
      	at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:183)
      	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Please check if you are invoking next() even after it returned false. For usage, please refer to KeyValueReader javadocs
      	at org.apache.tez.runtime.library.api.KeyValueReader.hasCompletedProcessing(KeyValueReader.java:77)
      	at org.apache.tez.runtime.library.common.readers.UnorderedKVReader.moveToNextInput(UnorderedKVReader.java:190)
      	at org.apache.tez.runtime.library.common.readers.UnorderedKVReader.next(UnorderedKVReader.java:118)
      	at org.apache.pig.backend.hadoop.executionengine.tez.plan.udf.ReadScalarsTez.attachInputs(ReadScalarsTez.java:81)
      	... 15 more
      

        Attachments

        1. PIG-4816-1.patch
          2 kB
          Jianyong Dai

          Activity

            People

            • Assignee:
              daijy Jianyong Dai
              Reporter:
              daijy Jianyong Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: