[PIG-2932] Setting high default_parallel causes IOException in local mode

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.11
• Assignee: Cheolsoo Park
• Reporter: Gianmarco De Francisci Morales
• Created: 26/Sep/12 13:41
• Updated: 24/Jan/15 21:53
• Resolved: 05/Oct/12 21:45

Description

This bug has been confirmed only in local mode.

When setting a high default_parallel, Pig fails on some operations.
The following data and script reproduce the bug.

Data:

grunt> cat file.txt                                  
11	1	qwer
12	2	qwerty
13	3	ert
13	3	ertyu
14	4	zxcv
16	6	fsdfg
16	6	fdfghj
18	8	fjklopi

Script:

SET default_parallel 9
a = load 'file.txt' as (id1:int, id2:int, str:chararray);
b = group a by (id1,id2);
c = foreach b generate flatten(group), a;
d = order c by group::id1 ASC, group::id2 ASC;
dump d

Error:

2012-09-26 15:28:13,230 [Thread-32] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map - Aliases being processed per job phase (AliasName[line,offset]): M: d[12,4] C:  R: 
2012-09-26 15:28:13,232 [Thread-32] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local_0009
java.io.IOException: Illegal partition for Null: false index: 0 (12,2) (1)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1073)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

The script succeeds if default_parallel is set to 2.
I guess it depends on the fact that the default_parallel is higher than the number of unique keys, probably some quirk with ORDER BY.

Issue Links

• relates to: PIG-4392 RANK BY fails when default_parallel is greater than cardinality of field being ranked by