[SPARK-1407] EventLogging to HDFS doesn't work properly on yarn

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Not A Problem
• Affects Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 03/Apr/14 14:06
• Updated: 23/Jan/15 18:32
• Resolved: 08/Apr/14 21:16

Description

When running on spark on yarn and accessing an HDFS file (like in the SparkHdfsLR example) while using the event logging configured to write logs to HDFS, it throws an exception at the end of the application.

SPARK_JAVA_OPTS=-Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs:///history/spark/

14/04/03 13:41:31 INFO yarn.ApplicationMaster$$anon$1: Invoking sc stop from shutdown hook
Exception in thread "Thread-41" java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:398)
at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1465)
at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1450)
at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:116)
at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
at org.apache.spark.util.FileLogger$$anonfun$flush$2.apply(FileLogger.scala:137)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.util.FileLogger.flush(FileLogger.scala:137)
at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:69)
at org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:101)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$13.apply(SparkListenerBus.scala:67)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:67)
at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:31)
at org.apache.spark.scheduler.LiveListenerBus.post(LiveListenerBus.scala:78)
at org.apache.spark.SparkContext.postApplicationEnd(SparkContext.scala:1081)
at org.apache.spark.SparkContext.stop(SparkContext.scala:828)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:460)

Issue Links

• is duplicated by: SPARK-2906 FileLogger throws a invocation target exception.
• is related to: SPARK-1475 Drain event logging queue before stopping event logger
• relates to: SPARK-1132 Persisting Web UI through refactoring the SparkListener interface