[TEZ-1107] Support increase of parallelism of vertex in case of custom partitioner

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Bikas Saha
• Reporter: Daniel Dai
• Created: 08/May/14 00:35
• Updated: 25/Jul/14 17:42

Description

Current VertexManagerPlugin/EdgeManager mechanism support decrease of parallelism of a vertex, but increase parallelism is not supported. In general, we need to do repartition to increase the parallelism. However, in my simplified case, the proceeding vertex is using a custom partitioner which is able to partition to the final parallelism, repartitioning is not needed. However, I hit an exception from sorter:
: Caused by: java.io.IOException: Illegal partition for Null: false index: 0 53.8 (2), TotalPartitions: 2
: at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.collect(DefaultSorter.java:208)
: at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.write(DefaultSorter.java:185)
: at org.apache.tez.runtime.library.output.OnFileSortedOutput$1.write(OnFileSortedOutput.java:111)
: at org.apache.pig.backend.hadoop.executionengine.tez.POIdentityInOutTez.getNextTuple(POIdentityInOutTez.java:148)
: ... 8 more

While increase parallelism in general is harder, increase parallelism with a custom partitioner might be easier to fix.

Issue Links

• is required by: TEZ-388 Pig on Tez