[SPARK-30198] BytesToBytesMap does not grow internal long array as expected

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.3, 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Spark Core
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 10/Dec/19 06:12
• Updated: 13/Dec/19 07:25
• Resolved: 11/Dec/19 22:59

Description

One Spark job on our cluster hangs forever at BytesToBytesMap.safeLookup. After inspecting, the long array size is 536870912.

Currently in BytesToBytesMap.append, we only grow the internal array if the size of the array is less than its MAX_CAPACITY that is 536870912. So in above case, the array can not be grown up, and safeLookup can not find an empty slot forever.

But it is wrong because we use two array entries per key, so the array size is twice the capacity. We should compare the current capacity of the array, instead of its size.

Issue Links

• links to: GitHub Pull Request #26828