[SPARK-22149] spark.shuffle.memoryFraction (deprecated) in spark 2

• Type: Documentation
• Status: Closed
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 2.1.1
• Component/s: Documentation
• Assignee: Unassigned
• Reporter: regis le bretonnic
• Created: 27/Sep/17 20:47
• Updated: 28/Sep/17 02:19
• Resolved: 28/Sep/17 02:19

Description

Hi

This is not a bug but maybe a lack of documentation.

I have a job that produce a lot of blockmgr files... I do not understand why the shuffle writes so much on disk and not in the heap of nodemanager.

I wanted to increase spark.shuffle.memoryFraction to reduce the part of data on disk, but this parameter is deprecated in the version we use (https://spark.apache.org/docs/2.1.1/configuration.html)
How to increase the memory size allocated to shuffle in spark 2 ? Is there a non documented parameter ?

I do not use an external shuffle service and I'd prefer to avoid it for now...

Thanks in advance