Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8333

Spark failed to delete temp directory created by HiveContext

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: SQL
    • Environment:

      Windows7 64bit

      Description

      Spark 1.4.0 failed to stop SparkContext.

      LocalHiveTest.scala
       val sc = new SparkContext("local", "local-hive-test", new SparkConf())
       val hc = Utils.createHiveContext(sc)
       ... // execute some HiveQL statements
       sc.stop()
      

      sc.stop() failed to execute, it threw the following exception:

      15/06/13 03:19:06 INFO Utils: Shutdown hook called
      15/06/13 03:19:06 INFO Utils: Deleting directory C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
      15/06/13 03:19:06 ERROR Utils: Exception while deleting Spark temp dir: C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
      java.io.IOException: Failed to delete: C:\Users\moshangcheng\AppData\Local\Temp\spark-d6d3c30e-512e-4693-a436-485e2af4baea
      at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:963)
      at org.apache.spark.util.Utils$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(Utils.scala:204)
      at org.apache.spark.util.Utils$$anonfun$1$$anonfun$apply$mcV$sp$5.apply(Utils.scala:201)
      at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
      at org.apache.spark.util.Utils$$anonfun$1.apply$mcV$sp(Utils.scala:201)
      at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2292)
      at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Utils.scala:2262)
      at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2262)
      at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2262)
      at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
      at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(Utils.scala:2262)
      at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2262)
      at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2262)
      at scala.util.Try$.apply(Try.scala:161)
      at org.apache.spark.util.SparkShutdownHookManager.runAll(Utils.scala:2262)
      at org.apache.spark.util.SparkShutdownHookManager$$anon$6.run(Utils.scala:2244)
      at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

      It seems this bug is introduced by this SPARK-6907. In SPARK-6907, a local hive metastore is created in a temp directory. The problem is the local hive metastore is not shut down correctly. At the end of application, if SparkContext.stop() is called, it tries to delete the temp directory which is still used by the local hive metastore, and throws an exception.

        Attachments

        1. test.tar
          46 kB
          Sudhakar Thota

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              moshangcheng sheng
            • Votes:
              11 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: