[SPARK-23438] DStreams could lose blocks with WAL enabled when driver crashes

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.0.3, 2.1.3, 2.2.2, 2.3.1, 2.4.0
• Component/s: DStreams
• Assignee: Gabor Somogyi
• Reporter: Gabor Somogyi
• Created: 15/Feb/18 16:34
• Updated: 26/Feb/18 16:58
• Resolved: 26/Feb/18 16:58

Description

There is a race condition introduced in SPARK-11141 which could cause data loss.

This affects all versions since 1.6.0.

Problematic situation:

1. Start streaming job with 2 receivers with WAL enabled.
2. Receiver 1 receives a block and does the following

• • Writes a BlockAdditionEvent into WAL
  • Puts the block into it's received block queue with ID 1

1. Receiver 2 receives a block and does the following

• • Writes a BlockAdditionEvent into WAL

1. Spark allocates all blocks from it's received block queue and writes AllocatedBlocks(IDs=(1)) into WAL
2. Driver crashes
3. New Driver recovers from WAL
4. Realise block with ID 2 never processed

 

Issue Links

• links to: [Github] Pull Request #20620 (gaborgsomogyi)