[SPARK-4107] Incorrect handling of Channel.read()'s return value may lead to data truncation

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.0, 1.1.1, 1.2.0
• Fix Version/s: 1.1.1, 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.1.1, 1.2.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 28/Oct/14 03:31
• Updated: 29/Oct/14 20:11
• Resolved: 28/Oct/14 19:30

Description

When using Channel.read(), we need to properly handle the return value and account for the case where we've read fewer bytes than expected. There are a few places where we don't do this properly, which may lead to incorrect data truncation in rare circumstances. I've opened a PR to fix this.

Issue Links

• is related to: SPARK-4105 FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle
• links to: [Github] Pull Request #2969 (JoshRosen)
• links to: [Github] Pull Request #2974 (JoshRosen)