[PIG-179] On hadoop 0.16, some jobs using combiner fail with an NPE

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.0.0
• Fix Version/s: 0.1.0

Environment

Hadoop 0.16

• Assignee: Alan Gates
• Reporter: Alan Gates
• Created: 02/Apr/08 21:28
• Updated: 24/Mar/10 22:01
• Resolved: 11/Apr/08 13:58

Description

Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:

PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();

Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.