[SPARK-24713] AppMatser of spark streaming kafka OOM if there are hundreds of topics consumed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 2.4.0
• Component/s: Input/Output
• Assignee: Yuanbo Liu
• Reporter: Yuanbo Liu
• Created: 02/Jul/18 03:12
• Updated: 13/Jul/18 13:38
• Resolved: 13/Jul/18 13:38

Description

We have hundreds of kafka topics need to be consumed in one application. The application master will throw OOM exception after hanging for nearly half of an hour.

Issue Links

• links to: [Github] Pull Request #21690 (yuanboliu)