[SPARK-20426] OneForOneStreamManager occupies too much memory.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Shuffle, Spark Core
• Assignee: Jin Xing
• Reporter: Jin Xing
• Created: 21/Apr/17 07:50
• Updated: 17/May/20 18:29
• Resolved: 27/Apr/17 19:07

Description

Spark jobs are running on yarn cluster in my warehouse. We enabled the external shuffle service(--conf spark.shuffle.service.enabled=true). Recently NodeManager runs OOM now and then. Dumping heap memory, we find that OneFroOneStreamManager's footprint is huge. NodeManager is configured with 5G heap memory. While OneForOneManager costs 2.5G and there are 5503233 FileSegmentManagedBuffer objects. Is there any suggestions to avoid this other than just keep increasing NodeManager's memory? Is it possible to stop registerStream in OneForOneStreamManager? Thus we don't need to cache so many metadatas(i.e. StreamState).

Issue Links

• links to: [Github] Pull Request #17744 (jinxing64)