Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1407

EventLogging to HDFS doesn't work properly on yarn

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Not A Problem
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When running on spark on yarn and accessing an HDFS file (like in the SparkHdfsLR example) while using the event logging configured to write logs to HDFS, it throws an exception at the end of the application.

      SPARK_JAVA_OPTS=-Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs:///history/spark/

      14/04/03 13:41:31 INFO yarn.ApplicationMaster$$anon$1: Invoking sc stop from shutdown hook
      Exception in thread "Thread-41" java.io.IOException: Filesystem closed
      at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:398)
      at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1465)
      at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1450)
      at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:116)
      at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
      at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.util.FileLogger.flush(FileLogger.scala:137)
      at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:69)
      at org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:101)
      at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
      at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:67)
      at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:31)
      at org.apache.spark.scheduler.LiveListenerBus.post(LiveListenerBus.scala:78)
      at org.apache.spark.SparkContext.postApplicationEnd(SparkContext.scala:1081)
      at org.apache.spark.SparkContext.stop(SparkContext.scala:828)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:460)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tgraves Thomas Graves
                Reporter:
                tgraves Thomas Graves
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: