[SPARK-8890] Reduce memory consumption for dynamic partition insert

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Michael Armbrust
• Reporter: Reynold Xin
• Created: 08/Jul/15 06:49
• Updated: 26/Oct/15 01:02
• Resolved: 07/Aug/15 23:25

Description

Currently, InsertIntoHadoopFsRelation can run out of memory if the number of table partitions is large. The problem is that we open one output writer for each partition, and when data are randomized and when the number of partitions is large, we open a large number of output writers, leading to OOM.

The solution here is to inject a sorting operation once the number of active partitions is beyond a certain point (e.g. 50?)

Issue Links

• blocks: SPARK-9707 Test sort-based fallback mode for dynamic partition insert
• is duplicated by: SPARK-8597 DataFrame partitionBy memory pressure scales extremely poorly
• is duplicated by: SPARK-8968 dynamic partitioning in spark sql performance issue due to the high GC overhead
• links to: [Github] Pull Request #7514 (ilganeli)
• links to: [Github] Pull Request #8010 (marmbrus)