[SPARK-25904] Avoid allocating arrays too large for JVMs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: Spark Core
• Assignee: Imran Rashid
• Reporter: Imran Rashid
• Created: 31/Oct/18 20:44
• Updated: 08/Nov/18 14:52
• Resolved: 07/Nov/18 12:19

Description

In a few places spark can try to allocate arrays as big as Int.MaxValue, but thats actually too big for the JVM. We should consistently use ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH instead.

In some cases this is changing defaults for configs, in some cases its bounds on a config, and others its just improving error msgs for things that still won't work.

Issue Links

• is related to: SPARK-25704 Replication of > 2GB block fails due to bad config default
• links to: [Github] Pull Request #22818 (squito)
• links to: [Github] Pull Request #22983 (squito)
• links to: [Github] Pull Request #22983 (squito)