[PIG-5024] add a physical operator to broadcast small RDDs

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Xianda Ke
• Reporter: Xianda Ke
• Created: 06/Sep/16 12:43
• Updated: 21/Jun/17 09:18
• Resolved: 09/Sep/16 04:51

Description

Currently, when optimize some kinds of JOIN, the indexed or sampling files are saved into HDFS. By setting the replication to a larger number, it serves as distributed cache.

Spark's broadcast mechanism is suitable for this. It seems that we can add a physical operator to broadcast small RDDs.
This will benefit the optimization of some specialized Joins, such as Skewed Join, Replicated Join and so on.

Issue Links

• links to: RB