After updating Spark from 1.5.0 to 1.6.0, I found that it seems to have a memory leak on my Spark streaming application.
Here is the head of the heap histogram of my application, which has been running about 160 hours:
It shows that scala.collection.mutable.DefaultEntry and java.lang.Long have unexpected big numbers of instances. In fact, the numbers started growing at streaming process began, and keep growing proportional to total number of tasks.
After some further investigation, I found that the problem is caused by some inappropriate memory management in releaseUnrollMemoryForThisTask and unrollSafely method of class org.apache.spark.storage.MemoryStore.
In Spark 1.6.x, a releaseUnrollMemoryForThisTask operation will be processed only with the parameter memoryToRelease > 0:
But in fact, if a task successfully unrolled all its blocks in memory by unrollSafely method, the memory saved in unrollMemoryMap would be set to zero:
So the result is, the memory saved in unrollMemoryMap will be released, but the key of that part of memory will never be removed from the hash map. The hash table will keep increasing, while new tasks keep incoming. Although the speed of increase is comparatively slow (about dozens of bytes per task), it is possible that result into OOM after weeks or months.