[SPARK-24160] ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Shuffle, Spark Core
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 02/May/18 22:08
• Updated: 17/May/20 18:30
• Resolved: 07/May/18 06:34

Description

In the shuffle layer, we guarantee that zero-size blocks will never be requested (a block containing zero records is always 0 bytes in size and is marked as empty such that it will never be legitimately requested by executors). However, we failed to take advantage of this in the shuffle-read path: the existing code did not explicitly check whether blocks are non-zero-size.

 

We should add `buf.size != 0` checks to ShuffleBlockFetcherIterator to take advantage of this invariant and prevent potential data loss / corruption issues. 

Issue Links

• is related to: SPARK-27216 Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
• is related to: SPARK-27530 FetchFailedException: Received a zero-size buffer for block shuffle
• links to: [Github] Pull Request #21219 (JoshRosen)
• links to: [Github] Pull Request #21256 (mgaido91)