[SPARK-8333] Spark failed to delete temp directory created by HiveContext

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.4.0
• Component/s: SQL
• Labels: Hive bulk-closed metastore sparksql

Environment

Windows7 64bit

• External issue ID: SPARK-6907
• Assignee: Unassigned
• Reporter: sheng
• Created: 12/Jun/15 19:38
• Updated: 28/Apr/20 22:01
• Resolved: 21/May/19 04:35

Description

Spark 1.4.0 failed to stop SparkContext.

LocalHiveTest.scala

 val sc = new SparkContext("local", "local-hive-test", new SparkConf())
 val hc = Utils.createHiveContext(sc)
 ... // execute some HiveQL statements
 sc.stop()

sc.stop() failed to execute, it threw the following exception:

15/06/13 03:19:06 INFO Utils: Shutdown hook called
15/06/13 03:19:06 INFO Utils: Deleting directory C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
15/06/13 03:19:06 ERROR Utils: Exception while deleting Spark temp dir: C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
java.io.IOException: Failed to delete: C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:963)
at org.apache.spark.util.Utils$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(Utils.scala:204)
at org.apache.spark.util.Utils$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(Utils.scala:201)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.spark.util.Utils$$anonfun$1.apply$mcV$sp(Utils.scala:201)
at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2292)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Utils.scala:2262)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2262)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2262)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(Utils.scala:2262)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2262)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2262)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.util.SparkShutdownHookManager.runAll(Utils.scala:2262)
at org.apache.spark.util.SparkShutdownHookManager$$anon$6.run(Utils.scala:2244)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

It seems this bug is introduced by this SPARK-6907. In SPARK-6907, a local hive metastore is created in a temp directory. The problem is the local hive metastore is not shut down correctly. At the end of application, if SparkContext.stop() is called, it tries to delete the temp directory which is still used by the local hive metastore, and throws an exception.

Issue Links

• links to: [Github] Pull Request #6840 (navis)
• links to: [Github] Pull Request #6858 (navis)