[SPARK-13845] BlockStatus and StreamBlockId keep on growing result driver OOM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.2, 1.6.1
• Fix Version/s: 1.6.2, 2.0.0
• Component/s: Spark Core
• Target Version/s: 1.6.2, 2.0.0
• Assignee: jeanlyn
• Reporter: jeanlyn
• Created: 13/Mar/16 08:52
• Updated: 30/Jun/18 00:26
• Resolved: 01/Apr/16 21:16

Description

We have a streaming job using FlumePollInputStream always driver OOM after few days, here is some driver heap dump before OOM

 num     #instances         #bytes  class name
----------------------------------------------
   1:      13845916      553836640  org.apache.spark.storage.BlockStatus
   2:      14020324      336487776  org.apache.spark.storage.StreamBlockId
   3:      13883881      333213144  scala.collection.mutable.DefaultEntry
   4:          8907       89043952  [Lscala.collection.mutable.HashEntry;
   5:         62360       65107352  [B
   6:        163368       24453904  [Ljava.lang.Object;
   7:        293651       20342664  [C
...

BlockStatus and StreamBlockId keep on growing, and the driver OOM in the end.

Issue Links

• causes: SPARK-16859 History Server storage information is missing
• links to: [Github] Pull Request #11679 (jeanlyn)
• links to: [Github] Pull Request #11779 (jeanlyn)
• links to: [Github] Pull Request #12028 (jeanlyn)