[PIG-71] Support for Hadoop Speculative Execution

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.1.0

Environment

Hadoop

• Assignee: Unassigned
• Reporter: Amir Youssefi
• Created: 23/Jan/08 22:39
• Updated: 24/Mar/10 22:01
• Resolved: 11/Jun/08 21:33

Description

If Speculative Execution is used in Hadoop while creating a data-set then Pig scripts loading this data-set may fail. Reason is temp directories generated in the process.

Pig can filter out these temp directories and problem gets solved. Here is sample error:

[main] ERROR org.apache.pig - Error message from task (map) tip_..._0001_m_002735 java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1524)
at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1590)
at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1626)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
...

Issue Links

• is related to: HADOOP-2391 Speculative Execution race condition with output paths