Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Stack:
      Error: java.lang.NullPointerException
      : at java.io.FileOutputStream.writeBytes(Native Method)
      : at java.io.FileOutputStream.write(FileOutputStream.java:282)
      : at org.apache.tez.runtime.library.common.shuffle.impl.InMemoryReader.dumpOnError(InMemoryReader.java:85)
      : at org.apache.tez.runtime.library.common.shuffle.impl.InMemoryReader.readRawKey(InMemoryReader.java:119)
      : at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawKey(IFile.java:510)
      : at org.apache.tez.runtime.library.common.readers.ShuffledUnorderedKVReader.readNextFromCurrentReader(ShuffledUnorderedKVReader.java:147)
      : at org.apache.tez.runtime.library.common.readers.ShuffledUnorderedKVReader.next(ShuffledUnorderedKVReader.java:108)
      : at org.apache.pig.backend.hadoop.executionengine.tez.POValueInputTez.getNextTuple(POValueInputTez.java:123)
      : at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:300)
      : at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFilter.getNextTuple(POFilter.java:91)
      : at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:300)
      : at org.apache.pig.backend.hadoop.executionengine.tez.POStoreTez.getNextTuple(POStoreTez.java:113)
      : at org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.runPipeline(PigProcessor.java:276)
      : at org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:175)
      : at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)
      : at org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:581)
      : at java.security.AccessController.doPrivileged(Native Method)
      : at javax.security.auth.Subject.doAs(Subject.java:394)
      : at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1495)
      : at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:570)
      : ], Vertex failed as one or more tasks failed. failedTasks:1]
      : Vertex killed, vertexName=scope-97

      The reason is in a vertex, we have two POValueInput, and Pig erroneously assign them to the same input vertex.

        Activity

        Hide
        Daniel Dai added a comment -

        Patch committed to tez. Thanks Rohini for review. Also open PIG-3957 for refactoring.

        Show
        Daniel Dai added a comment - Patch committed to tez. Thanks Rohini for review. Also open PIG-3957 for refactoring.
        Hide
        Daniel Dai added a comment -

        Will do.

        Show
        Daniel Dai added a comment - Will do.
        Hide
        Rohini Palaniswamy added a comment -

        +1. Fine with that. Can you just add a TODO note to refactor it before checking in?

        Show
        Rohini Palaniswamy added a comment - +1. Fine with that. Can you just add a TODO note to refactor it before checking in?
        Hide
        Daniel Dai added a comment -

        Actually the entire logic to reset the input keys in TezDagBuild is confusing. I'd like to refactor it entirely, but should be in a separate Jira.

        Show
        Daniel Dai added a comment - Actually the entire logic to reset the input keys in TezDagBuild is confusing. I'd like to refactor it entirely, but should be in a separate Jira.
        Hide
        Rohini Palaniswamy added a comment -

        Looks good. But isn't there a better way to determine what is the input to POValueInputTez by selecting one that is not a scalar input? Can't we do set the correct input for POValueInputTez in TezCompiler itself?

        Show
        Rohini Palaniswamy added a comment - Looks good. But isn't there a better way to determine what is the input to POValueInputTez by selecting one that is not a scalar input? Can't we do set the correct input for POValueInputTez in TezCompiler itself?

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development