[SPARK-4849] Pass partitioning information (distribute by) to In-memory caching

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.6.0
• Component/s: SQL
• Assignee: Nong Li
• Reporter: Nitin Goyal
• Created: 15/Dec/14 11:20
• Updated: 04/Nov/15 18:01
• Resolved: 04/Nov/15 17:57

Description

HQL "distribute by <column_name>" partitions data based on specified column values. We can pass this information to in-memory caching for further performance improvements. e..g. in Joins, an extra partition step can be saved based on this information.

Refer - http://apache-spark-user-list.1001560.n3.nabble.com/SchemaRDD-partition-on-specific-column-values-td20350.html

Issue Links

• duplicates: SPARK-5354 Set InMemoryColumnarTableScan's outputPartitioning and outputOrdering
• relates to: SPARK-11410 Add a DataFrame API that provides functionality similar to HiveQL's DISTRIBUTE BY