[PIG-3113] Shell command execution hangs job

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.8.1
• Component/s: impl
• Assignee: Unassigned
• Reporter: James
• Created: 03/Jan/13 21:49
• Updated: 15/May/17 22:12
• Resolved: 15/May/17 22:12

Description

Executing a shell command inside a Pig script has the potential to deadlock the job. For example, the following statement will block when somebigfile.txt is sufficiently large:

%declare input `cat /path/to/somebigfile.txt`

This happens because PreprocessorContext.executeShellCommand(String) incorrectly uses Runtime.exec(). The sub-process's stderr and stdout streams should be read in a separate thread to prevent p.waitFor() from hanging when the sub-process's output is larger than the output buffer.

Per the Java Process class javadoc: "Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock".

See http://www.javaworld.com/jw-12-2000/jw-1229-traps.html for a correct solution.

Issue Links

• duplicates: PIG-5226 PreprocessorContext.java can deadlock forever with large stderr