[SPARK-2650] Caching tables larger than memory causes OOMs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.0.1
• Fix Version/s: 1.1.0
• Component/s: SQL
• Target Version/s: 1.1.0
• Assignee: Michael Armbrust
• Reporter: Michael Armbrust
• Created: 23/Jul/14 18:13
• Updated: 12/Aug/14 06:12
• Resolved: 12/Aug/14 03:22

Description

The logic for setting up the initial column buffers is different for Spark SQL compared to Shark and I'm seeing OOMs when caching tables that are larger than available memory (where shark was okay).

Two suspicious things: the intialSize is always set to 0 so we always go with the default. The default looks like it was copied from code like 10 * 1024 * 1024... but in Spark SQL its 10 * 102 * 1024.

Issue Links

• is related to: SPARK-2902 Change default options to be more agressive
• links to: [Github] Pull Request #1769 (liancheng)
• links to: [Github] Pull Request #1880 (marmbrus)
• links to: [Github] Pull Request #1901 (liancheng)