[HIVE-10550] Dynamic RDD caching optimization for HoS.[Spark Branch]

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Spark
• Assignee: Chengxiang Li
• Reporter: Chengxiang Li
• Created: 30/Apr/15 07:27
• Updated: 16/Feb/16 23:53
• Resolved: 28/May/15 05:30

Description

A Hive query may try to scan the same table multi times, like self-join, self-union, or even share the same subquery, TPC-DS Q39 is an example. As you may know that, Spark support cache RDD data, which mean Spark would put the calculated RDD data in memory and get the data from memory directly for next time, this avoid the calculation cost of this RDD(and all the cost of its dependencies) at the cost of more memory usage. Through analyze the query context, we should be able to understand which part of query could be shared, so that we can reuse the cached RDD in the generated Spark job.

Issue Links

• is related to: HIVE-10850 Followup for HIVE-10550, check performance w.r.t. persistence level [Spark Branch]
• relates to: HIVE-10844 Combine equivalent Works for HoS[Spark Branch]
• links to: https://reviews.apache.org/r/34455/