[PIG-5363] Use 1-1 edge vertex group in union

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Rohini Palaniswamy
• Created: 04/Oct/18 20:23
• Updated: 04/Oct/18 20:23

Description

TEZ-3983 fixes vertex groups to work properly with 1-1 edges. So we should switch to that instead of  using unordered partitioned shuffle edge in https://github.com/apache/pig/blob/5b0a0bfaefad0d57c6c9fbbc09d709d13ea9af93/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/UnionOptimizer.java#L635-L642). The current logic of partitioning based on hashcode of values is slightly expensive. For backward compatibility with older Tez versions, have to support both modes possibly via configuration setting or Tez version check.

Issue Links

• requires: TEZ-3983 VertexGroup ONE_TO_ONE edges do not produce the right graph