[SPARK-19644] Memory leak in Spark Streaming (Encoder/Scala Reflection)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: DStreams, SQL, Structured Streaming
• Labels: memory_leak performance

Environment

3 AWS EC2 c3.xLarge
Number of cores - 3
Number of executors 3
Memory to each executor 2GB

• Assignee: Shixiong Zhu
• Reporter: Deenbandhu Agarwal
• Created: 17/Feb/17 06:00
• Updated: 10/Nov/17 22:15
• Resolved: 10/Nov/17 22:15

Description

I am using streaming on the production for some aggregation and fetching data from cassandra and saving data back to cassandra.

I see a gradual increase in old generation heap capacity from 1161216 Bytes to 1397760 Bytes over a period of six hours.

After 50 hours of processing instances of class scala.collection.immutable.$colon$colon incresed to 12,811,793 which is a huge number.

I think this is a clear case of memory leak

Updated: The root cause is when creating an encoder object, it leaks several Scala internal objects due to a Scala memory leak issue: https://github.com/scala/bug/issues/8302

Issue Links

• links to: [Github] Pull Request #19687 (zsxwing)
• links to: [Github] Pull Request #19718 (zsxwing)
• links to: Undolog seems to leak memory #8302