[SPARK-30274] Avoid BytesToBytesMap lookup hang forever when holding keys reaching max capacity

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Spark Core
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 17/Dec/19 01:26
• Updated: 08/Jun/20 20:57
• Resolved: 17/Dec/19 19:46

Description

BytesToBytesMap.append allows to append keys until the number of keys reaches MAX_CAPACITY. But once the the pointer array in the map holds MAX_CAPACITY keys, next time call of lookup will hand forever.

Issue Links

• links to: GitHub Pull Request #26914