[PIG-4722] [Pig on Tez] NPE while running Combiner

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 31/Oct/15 02:00
• Updated: 08/Jun/16 20:48
• Resolved: 13/Nov/15 19:28

Description

DefaultSorter in Tez calls Combiner from two different threads - during spill in SpillThread and flush in the main thread. If both run the combiner, one ends up with NPE as Reporter is set on only one thread in initialization.

java.lang.NullPointerException
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:366)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNextTuple(POLocalRearrange.java:332)
        at org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POLocalRearrangeTez.getNextTuple(POLocalRearrangeTez.java:128)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.processOnePackageOutput(PigCombiner.java:197)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:175)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:50)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
        at org.apache.tez.mapreduce.combine.MRCombiner.runNewCombiner(MRCombiner.java:191)
        at org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:115)
        at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.runCombineProcessor(ExternalSorter.java:279)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.spill(DefaultSorter.java:854)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.sortAndSpill(DefaultSorter.java:780)
        at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter$SpillThread.run(DefaultSorter.java:708)
Caused by: java.lang.NullPointerException
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:303)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNextTuple(POProject.java:403)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:361)
        ... 12 more

Issue Links

• requires: TEZ-2937 Can Processor.close() be called after closing inputs and outputs?