[TEZ-388] Pig on Tez

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Rohini Palaniswamy
• Created: 23/Aug/13 18:41
• Updated: 02/Jul/14 23:41

Description

Umbrella jira to track Pig on Tez issues.

Issue Links

• relates to: PIG-3446 Umbrella jira for Pig on Tez
• requires: TEZ-391 SharedEdge - Support for passing same output from a vertex as input to two different vertices
• requires: TEZ-793 Staging directories accumulate and pollute the home directory
• requires: TEZ-606 Fix Tez to work with kerberos security
• requires: TEZ-711 Fix memory leak when not reading from inputs due to caching
• requires: TEZ-1107 Support increase of parallelism of vertex in case of custom partitioner
• requires: TEZ-789 Setting intermediate output compression does not work
• requires: TEZ-668 Allow a Processor to trigger an Input as to when to fetch/prep data.
• requires: TEZ-800 One-one edge with parallelism -1 fails if source vertex parallelism is not -1 as well
• requires: TEZ-984 Improve usability and debuggability
• requires: TEZ-1153 Allow object caching for more than one vertex
• requires: TEZ-661 Implement a non-sorted partitioned output
• requires: TEZ-678 Support for union operations via VertexGroup abstraction
• requires: TEZ-802 Determination of Task Placement for 1-1 Edges

Sub-Tasks

• Ability to retrieve counters of each vertex and track vertex progress: Closed
• Support a Vertex Initializer and Committer: Resolved
• SharedEdge - Support for passing same output from a vertex as input to two different vertices: Open
• Support for passing two or more different outputs from a vertex as input to two or more different vertexes: Closed
• Handle custom partitioners: Resolved
• Better scheduling for uneven DAGs: Patch Available
• MRInputSplitDistributor should set parallelism like MRInputAMSplitGenerator: Open
• Make some statistics available for the Input: Open