[SPARK-15401] Spark Thrift server creates empty directories in tmp directory on the driver

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.6.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Christophe Préaud
• Created: 19/May/16 09:30
• Updated: 17/Jan/18 13:13
• Resolved: 17/Jan/18 13:13

Description

Each connection to the Spark thrift server (e.g. using beeline) creates two empty directories in the tmp directory on the driver which are never removed:

cd <tmp directory>
ls -ltd *_resources | wc -l && /opt/spark/bin/beeline -u jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000 -n kookel -e '!quit' && ls -ltd *_resources | wc -l
9080
Connecting to jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
Connected to: Spark SQL (version 1.6.1)
Driver: Spark Project Core (version 1.6.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Closing: 0: jdbc:hive2://dc1-kdp-prod-hadoop-00.prod.dc1.kelkoo.net:10000
Beeline version 1.6.1 by Apache Hive
9082

Those directories accumulates over time and are not removed:
ls -ld *_resources | wc -l
9064

And they are indeed empty:
find *_resources -type f | wc -l
0

Issue Links

• duplicates: SPARK-22793 Memory leak in Spark Thrift Server
• relates to: SPARK-23130 Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)