[SPARK-25181] Block Manager master and slave thread pools are unbounded

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core
• Target Version/s: 2.4.0
• Assignee: Mukul Murthy
• Reporter: Mukul Murthy
• Created: 21/Aug/18 23:03
• Updated: 22/Aug/18 17:36
• Resolved: 22/Aug/18 17:36

Description

Currently, BlockManagerMasterEndpoint and BlockManagerSlaveEndpoint both have thread pools with unbounded numbers of threads. In certain cases, this can lead to driver OOM errors. We should add an upper bound on the number of threads in these thread pools; this should not break any existing behavior because they still have queues of size Integer.MAX_VALUE.

Issue Links

• is duplicated by: SPARK-25182 Block Manager master and slave thread pools are unbounded
• links to: [Github] Pull Request #22176 (mukulmurthy)