[TEZ-1190] Allow multiple edges between two vertices

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Zhiyuan Yang
• Reporter: Daniel Dai
• Created: 07/Jun/14 00:33
• Updated: 27/Jun/17 23:45

Description

This will be helpful in some scenario. In particular example, we can merge two small pipelines together in one pair of vertex. Note it is possible the edge type between the two vertexes are different.

Issue Links

• is related to: PIG-4495 Better multi-query planning in case of multiple edges
• is required by: PIG-3876 Handle two outputs from split going to same input in MultiQueryOptimizer
• is required by: PIG-5266 Simplify MultiQueryOptimizerTez for case of multiple edge to same vertex

Sub-Tasks

• Allow user to create named edge: Resolved
• Update EdgePlan proto for named edge: Open
• Edge in AM side should have name: Open
• VertexImpl should separate edges from source/target vertices: Open
• VertexManagerPluginContext should allow user to get source vertex name of a specified edge: Open
• InputSpec/OutputSpec should include source/target edge name: Open
• VertexStatistics and TaskStatistics have to separate input and output: Open
• TezEvent and EventMetaData should use edge name not vertex name for routing: Open
• InputContext/OutputContext should allow user to get source/target edge name: Open
• VertexManagerEvent (in tez-api) should use edge name for routing: Resolved
• LogicalIOProcessorRuntimeTask should use edge name to index IOs: Patch Available
• Support multi-edge case in graphical view of UI: Open
• Per IO counters should use source edge name instead of source vertex name in counter names: Open
• EdgeInfo in history data model should include edge name: Open
• DAG visualization should contains edge name in edge info: Open
• Make existing vertex manager work in multi-edge scenario: Open
• rebase 03/28: Resolved