[SPARK-22429] Streaming checkpointing code does not retry after failure due to NullPointerException

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.3, 2.2.0
• Fix Version/s: 2.1.3, 2.2.1, 2.3.0
• Component/s: DStreams
• Assignee: Tristan Stevens
• Reporter: Tristan Stevens
• Created: 02/Nov/17 16:47
• Updated: 05/Nov/17 09:12
• Resolved: 05/Nov/17 09:11

Description

CheckpointWriteHandler has a built in retry mechanism. However SPARK-14930/SPARK-13693 put in a fix to de-allocate the fs object, yet initialises it in the wrong place for the while loop, and therefore on attempt 2 it fails with a NPE.

Issue Links

• links to: [Github] Pull Request #19645 (tmgstevens)