[SPARK-2696] Reduce default spark.serializer.objectStreamReset

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.3, 1.1.0
• Component/s: Spark Core
• Labels: configuration
• Target Version/s: 1.0.3
• Assignee: Hossein Falaki
• Reporter: Hossein Falaki
• Created: 25/Jul/14 22:55
• Updated: 26/Jul/14 08:06
• Resolved: 26/Jul/14 08:06

Description

The current default value of spark.serializer.objectStreamReset is 10,000.
When trying to re-partition (e.g., to 64 partitions) a large file (e.g., 500MB), containing 1MB records, the serializer will cache 10000 x 1MB x 64 = 640 GB which will cause it to go out of memory.

We think 100 would be a more reasonable default value for this configuration parameter.

Issue Links

• links to: [Github] Pull Request #1595 (falaki)