[TEZ-1151] Vertex should stay in initializing state until custom vertex manager sets the parallelism

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Hadoop Flags: Reviewed
• Assignee: Bikas Saha
• Reporter: Daniel Dai
• Created: 23/May/14 22:13
• Updated: 06/Sep/14 01:35
• Resolved: 30/May/14 23:24

Description

In TEZ-1145, application is allowed to set -1 parallelism on a non 1-1 input vertex, and expect VertexManager to set the actual parallelism dynamically. However, when I does that, I hit the following exception:

org.apache.tez.dag.api.TezUncheckedException: vertex_1400869346120_0005_1_04 has -1 tasks but neither input initializers nor 1-1 uninited sources
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2485)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2431)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2412)
        at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1267)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:158)
        at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1716)
        at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1702)
        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
        at java.lang.Thread.run(Thread.java:695)

Issue Links

• blocks: PIG-3846 Implement automatic reducer parallelism