[SPARK-9539] Repeated sc.close() in PySpark causes JVM memory leak

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.4.1
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Andrey Zimovnov
• Created: 02/Aug/15 16:09
• Updated: 02/Aug/15 17:10
• Resolved: 02/Aug/15 17:10

Description

Example code in Python:

for i in range(20):
	print i
	conf = SparkConf().setAppName("test")
	sc = SparkContext(conf=conf)
	hivec = HiveContext(sc)
	hivec.sql("select id from details_info limit 1").show()
	sc.stop()
	del hivec
	del sc

Jstat output:

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
196608,0 196608,0 97566,2  0,0   1179648,0 542150,0 3145728,0    120,0    154112,0 153613,2      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 679041,7 3145728,0    120,0    164352,0 164183,3      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 907928,4 3145728,0    120,0    164352,0 164200,3      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 912132,7 3145728,0    120,0    164352,0 164200,5      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 913741,5 3145728,0    120,0    164352,0 164200,8      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 929458,6 3145728,0    120,0    164352,0 164206,0      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 1003138,1 3145728,0    120,0    168960,0 168646,0      4    0,434   0      0,000    0,434
131584,0 196608,0  0,0   109725,6 1179648,0   0,0    3145728,0    128,0    175104,0 174802,1      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 152654,9 3145728,0    128,0    175104,0 174803,3      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 158586,1 3145728,0    128,0    175104,0 174803,3      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 160659,8 3145728,0    128,0    175104,0 174805,7      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 181935,2 3145728,0    128,0    175104,0 174819,7      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 283389,1 3145728,0    128,0    185856,0 185371,0      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 342596,4 3145728,0    128,0    185856,0 185379,3      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 547634,7 3145728,0    128,0    185856,0 185385,8      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 555930,9 3145728,0    128,0    185856,0 185385,8      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 557888,6 3145728,0    128,0    185856,0 185386,0      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 573907,5 3145728,0    128,0    185856,0 185397,5      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 637955,0 3145728,0    128,0    189952,0 189533,1      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 895866,1 3145728,0    128,0    196096,0 195968,5      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 948046,5 3145728,0    128,0    196096,0 195969,4      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 952427,2 3145728,0    128,0    196096,0 195969,4      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 957977,5 3145728,0    128,0    196096,0 195973,4      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 977811,1 3145728,0    128,0    196096,0 195977,7      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 1118722,0 3145728,0    128,0    206848,0 206539,0      5    0,591   0      0,000    0,591
131584,0 144384,0 118692,5  0,0   1284096,0 183470,8 3145728,0    136,0    206848,0 206543,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 189718,5 3145728,0    136,0    206848,0 206543,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 192165,0 3145728,0    136,0    206848,0 206543,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 199848,4 3145728,0    136,0    206848,0 206546,9      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 219687,6 3145728,0    136,0    206848,0 206552,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 358272,4 3145728,0    136,0    217600,0 217100,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 573543,6 3145728,0    136,0    217600,0 217109,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 593569,9 3145728,0    136,0    217600,0 217109,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 593576,3 3145728,0    136,0    217600,0 217109,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 608832,1 3145728,0    136,0    217600,0 217114,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 649072,6 3145728,0    136,0    218624,0 218234,7      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 774293,0 3145728,0    136,0    227840,0 227734,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 971194,9 3145728,0    136,0    227840,0 227739,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 992291,6 3145728,0    136,0    227840,0 227739,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 992297,7 3145728,0    136,0    227840,0 227739,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 1002356,2 3145728,0    136,0    227840,0 227745,1      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 1020944,8 3145728,0    136,0    227840,0 227754,0      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 1169159,0 3145728,0    136,0    238592,0 238332,9      6    0,773   0      0,000    0,773
147456,0 109056,0  0,0   108867,6 1284096,0 110659,1 3145728,0   19167,5   238592,0 238479,1      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 116954,2 3145728,0   19167,5   238592,0 238491,0      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 116960,6 3145728,0   19167,5   238592,0 238491,0      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 132244,5 3145728,0   19167,5   238592,0 238491,9      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 218816,2 3145728,0   19167,5   245760,0 244549,2      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 467076,0 3145728,0   19167,5   249344,0 249189,3      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 508023,8 3145728,0   19167,5   249344,0 249189,8      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 511485,1 3145728,0   19167,5   249344,0 249191,3      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 513094,4 3145728,0   19167,5   249344,0 249191,9      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 527507,0 3145728,0   19167,5   249344,0 249197,0      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 636565,3 3145728,0   19167,5   260096,0 259705,2      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 819041,0 3145728,0   19167,5   260096,0 259749,7      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 917166,3 3145728,0   19167,5   260096,0 259749,7      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 919595,7 3145728,0   19167,5   260096,0 259749,7      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 925158,0 3145728,0   19167,5   260096,0 259750,1      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 946351,0 3145728,0   19167,5   260096,0 259751,1      7    0,942   0      0,000    0,942
151552,0 1024,0  0,0   924,7  1284608,0   0,0    3145728,0   119860,5  262144,0 262143,9      9    1,038   2      2,021    3,059
148480,0 153088,0  0,0    64,0  1262080,0   0,0    3145728,0   84562,9   262144,0 262143,9     17    1,167  10      6,988    8,155
99840,0 93696,0 160,0   0,0   1379328,0   0,0    3145728,0   84035,5   262144,0 262143,6     26    1,296  19     12,022   13,318
50688,0 54272,0  0,0   224,0  1460224,0   0,0    3145728,0   83958,1   262144,0 262143,8     35    1,474  28     16,642   18,116
23552,0 25088,0  0,0    96,0  1520640,0   0,0    3145728,0   84007,6   262144,0 262144,0     45    1,638  38     21,492   23,130

It looks like it hits -XX:MaxPermSize=256m for the driver and dies with OOM: Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly", Py4JJavaError(u'An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.\n', JavaObject id=o753))