[PIG-4856] Optimization for pig on spark

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Unassigned
• Reporter: liyunzhang
• Created: 29/Mar/16 07:30
• Updated: 06/Apr/16 06:00

Description

As we finished unit test fixing for milestone 2, we will enter milestone3 about the optimization for the project.

Sub-Tasks

• Implement FR Join for spark engine: Closed
• Implement Merge join for spark engine: Closed
• Optimization for join/group case for spark mode: Closed
• Use pigmix to test the performance of pig on spark: Open
• MultiQueryOptimizerSpark doesn't remove all redudant nodes in spark plan: Open
• Implement Skewed join for spark engine: Closed
• Re-design Spark plan to optimize the RDD pipeline: Open
• Not use OperatorPlan#forceConnect in MultiQueryOptimizationSpark: Closed
• Implement secondary sort using one shuffle: Closed
• Run pigmix on spark on yarn with multiple nodes: Resolved
• Calculate the value of parallism for spark mode: Closed
• Optimize combine case for spark mode: Closed
• Remove the deserialize and serialization of JobConf in code for spark mode: Closed
• add a physical operator to broadcast small RDDs: Closed
• Optimize sort case when data is skewed: Patch Available
• Create SparkCompiler#getSamplingJob in spark mode: Closed
• support outer join for skewedjoin in spark mode: Closed
• Initialize PigContants.TASK_INDEX in spark mode correctly: Closed
• Initialize MRConfiguration.JOB_ID in spark mode correctly: Closed
• Initialize SchemaTupleBackend correctly in backend in spark mode if spark job has more than 1 stage: Closed
• Set SPARK_REDUCERS by pig.properties not by system configuration: Closed
• Fix TestPigRunner.simpleMultiQueryTest3 unit test failure: Closed
• Enable persist/cache mechanism in Pig: Open
• Support outer join for SkewedJoin in spark mode: Resolved