[PIG-4856] Optimization for pig on spark

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Unassigned
• Reporter: liyunzhang
• Created: 29/Mar/16 07:30
• Updated: 06/Apr/16 06:00

Description

As we finished unit test fixing for milestone 2, we will enter milestone3 about the optimization for the project.

Sub-Tasks

• Use pigmix to test the performance of pig on spark: Open
• MultiQueryOptimizerSpark doesn't remove all redudant nodes in spark plan: Open
• Re-design Spark plan to optimize the RDD pipeline: Open
• Optimize sort case when data is skewed: Patch Available
• Enable persist/cache mechanism in Pig: Open