[PIG-3930] "java.io.IOException: Cannot initialize Cluster" in local mode with hadoopversion=23 dependencies

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.13.0
• Component/s: build
• Assignee: Gera Shegalov
• Reporter: Gera Shegalov
• Created: 08/May/14 00:16
• Updated: 07/Jul/14 18:08
• Resolved: 09/May/14 16:39

Description

Steps to reproduce with script wc.pig

A = load 'README.txt';
B = foreach A generate flatten(TOKENIZE((chararray)$0)) as word;
C = group B by word;
D = foreach C generate COUNT(B), group;
store D into 'wc.out';

1. $ cd workspace/pig-trunk
2. $ ant clean -Dhadoopversion=23 jar
3. $ bin/pig -x local ~/wc.pig

The problem is that in the dependency ./build/ivy/lib/Pig/hadoop-mapreduce-client-common-2.0.3-alpha.jar the file META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider lacks the entry org.apache.hadoop.mapred.LocalClientProtocolProvider.

When the dependency is folded into fat pig.jar src/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider is overwritten.

This can be fixed by using duplicate="preserve" in ant jar task.