[PIG-4845] Parallel instantiation of classes in Tez cause tasks to fail

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 21/Mar/16 22:45
• Updated: 08/Jun/16 20:48
• Resolved: 22/Mar/16 10:45

Description

OptionBuilder is not thread-safe

https://commons.apache.org/proper/commons-cli/javadocs/api-release/org/apache/commons/cli/OptionBuilder.html

This class is NOT thread safe. See CLI-209

HBaseStorage, elephantbird SequenceFileConfig, etc use it in their constructor. This leads to NoSuchMethodException, UnrecognizedOptionException etc when processor, inputs and outputs are initialized in parallel in Tez making the task fail. Retry attempts mostly go through and job might succeed. Need to synchronize the initialization. synchronize would make it similar to mapreduce though at a slight performance cost. But that should be lot better than cost of relaunching containers after hitting failures.

Following two kinds of exception where seen when HBaseStorage and SequenceFileStorage of elephantbird were used together.

org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -loadKey

Caused by: java.lang.RuntimeException: Failed to create WritableConverter instance
        at com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:225)
        at com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:101)
        at com.twitter.elephantbird.pig.util.SequenceFileConfig.<init>(SequenceFileConfig.java:115)
        ... 11 more
Caused by: java.lang.NoSuchMethodException: com.twitter.elephantbird.pig.util.TextConverter.<init>(java.lang.String)
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getConstructor(Class.java:1825)
        at com.twitter.elephantbird.pig.util.SequenceFileConfig.getWritableConverter(SequenceFileConfig.java:213)
        ... 13 more