[SPARK-18991] Change ContextCleaner.referenceBuffer to ConcurrentHashMap to make it faster

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Spark Core
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 23/Dec/16 17:32
• Updated: 07/Feb/20 17:23
• Resolved: 23/Dec/16 23:39

Description

Right now `ContextCleaner.referenceBuffer` is ConcurrentLinkedQueue and the time complexity of the `remove` action is O ( n ). It can be changed to use ConcurrentHashMap whose `remove` is O(1).

Issue Links

• contains: SPARK-21547 Spark cleaner cost too many time
• links to: [Github] Pull Request #16390 (zsxwing)