[SPARK-26094] Streaming WAL should create parent dirs

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: DStreams
• Target Version/s: 3.0.0
• Assignee: Imran Rashid
• Reporter: Imran Rashid
• Created: 16/Nov/18 21:40
• Updated: 04/Dec/18 22:35
• Resolved: 04/Dec/18 22:35

Description

SPARK-25871 introduced a regression in the streaming WAL – it no longer makes all the parent dirs, so you may see an exception like this in cases that used to work:

18/11/09 03:31:48 ERROR util.FileBasedWriteAheadLog_ReceiverSupervisorImpl: Failed to write to write ahead log after 3 failures
...
org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
        at org.apache.spark.streaming.receiver.WriteAheadLogBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:210)
...
Caused by: java.io.FileNotFoundException: Parent directory doesn't exist: /tmp/__spark__1e8ba184-d323-47eb-b857-0e6285409424/88992/checkpoints/receivedData/0
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyParentDir(FSDirectory.java:1923)

Issue Links

• is caused by: SPARK-25871 Streaming WAL should not use hdfs erasure coding, regardless of FS defaults
• links to: [Github] Pull Request #23092 (squito)
• links to: [Github] Pull Request #23092 (squito)