[SPARK-25695] Spark history server event log store problem

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: Web UI
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Si Chen
• Created: 10/Oct/18 05:54
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:44

Description

Envirment:spark 2.0.0 hadoop 2.7.3

spark-default.conf

spark.driver.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.eventLog.dir file:///home/hdfs/event
spark.eventLog.enabled true
spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.history.fs.logDirectory file:///home/hdfs/event
spark.history.kerberos.keytab none
spark.history.kerberos.principal none
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
spark.history.ui.port 18081
spark.yarn.historyServer.address slave6.htdata.com:18081
spark.yarn.queue default

 I want to save eventLog to local disk.
When I submit spark job use client deploy mode, event log can write to local disk.
But when I use cluster mode,The following problems arise.I am sure all servers have this path.

18/10/10 13:10:13 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1538963194112_0033 and attemptId Some(appattempt_1538963194112_0033_000001)
18/10/10 13:10:13 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 63016.
18/10/10 13:10:13 INFO netty.NettyBlockTransferService: Server created on 192.168.0.78:63016
18/10/10 13:10:13 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.78, 63016)
18/10/10 13:10:13 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.0.78:63016 with 366.3 MB RAM, BlockManagerId(driver, 192.168.0.78, 63016)
18/10/10 13:10:13 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.78, 63016)
18/10/10 13:10:13 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@32844dd3{/metrics/json,null,AVAILABLE}
18/10/10 13:10:13 ERROR spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File file:/home/hdfs/event does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:624)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:850)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:614)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422)
	at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:93)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
	at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:836)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84)
	at com.iiot.stream.spark.HTMonitorContext$.main(HTMonitorContext.scala:23)
	at com.iiot.stream.spark.HTMonitorContext.main(HTMonitorContext.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627)
18/10/10 13:10:13 INFO server.ServerConnector: Stopped ServerConnector@4fd401cf{HTTP/1.1}{0.0.0.0:0}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@73fe17d4{/stages/stage/kill,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@11e5f9d4{/api,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5d6fa266{/,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5920363{/static,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4494dda9{/executors/threadDump/json,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5a32de89{/executors/threadDump,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7c448c23{/executors/json,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@b0949d9{/executors,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6a3e90c6{/environment/json,null,UNAVAILABLE}
18/10/10 13:10:13 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6979e53f{/environment,null,UNAVAILABLE}