[SPARK-11141] Batching of ReceivedBlockTrackerLogEvents for efficient WAL writes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: DStreams
• Target Version/s: 1.6.0
• Assignee: Burak Yavuz
• Reporter: Burak Yavuz
• Created: 16/Oct/15 00:35
• Updated: 09/Mar/17 05:54
• Resolved: 10/Nov/15 01:35

Description

When using S3 as a directory for WALs, the writes take too long. The driver gets very easily bottlenecked when multiple receivers send AddBlock events to the ReceiverTracker. This PR adds batching of events in the ReceivedBlockTracker so that receivers don't get blocked by the driver for too long.

Issue Links

• links to: [Github] Pull Request #9143 (brkyvz)