[SPARK-16083] spark HistoryServer memory increases until gets killed by OS.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.1
• Component/s: Spark Submit

Environment

RHEL-6, IOP-4.1.0.0, 10 node cluster

• Assignee: Unassigned
• Reporter: Sudhakar Thota
• Created: 20/Jun/16 22:38
• Updated: 07/Jul/16 19:55
• Resolved: 21/Jun/16 20:59

Description

Spark HistoryServer process consuming memory over few days finally getting killed by operating system. Heapdump analysis of jmap dumps with IBM HeapAnalyzer, found that total heap size is 800M and process size is 11G. HistoryServer is started with 1G. ("history-server -Xms1g -Xmx1g org.apache.spark.deploy.history.HistoryServer")

Issue Links

• duplicates: SPARK-15083 History Server would OOM due to unlimited TaskUIData in some stages