[SPARK-27704] Change default class loader to ParallelGC

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.0.0
• Component/s: Build
• Assignee: Unassigned
• Reporter: Mihaly Toth
• Created: 14/May/19 11:23
• Updated: 16/Aug/19 10:00
• Resolved: 06/Jun/19 16:02

Description

In JDK 11 the default class loader changed from ParallelGC to G1GC. Even though this gc performs better on pause times and interactivity, most of the tasks that need to be processed are more sensitive to throughput and the to the amount of memory. G1 sacrifices these to some extend to avoid the big pauses. As a result the user may perceive a regression compared to JDK 8. Even worse, the regression may not be limited to performance only but some jobs may start failing in case they do not fit into the memory they used to be happy with when running with previous JDK.

Some other kind of apps, like streaming ones, may rather use G1 because of their more interactive, more realtime needs.

With this jira it is proposed to have a configurable default GC for all spark applications. This may be overridable by the user through command line parameters. The default value of the default GC (in case it is not provided in spark-defaults.conf) could be ParallelGC.

I do not see this change required but I think it would benefit to the user experience.