[SPARK-22982] Remove unsafe asynchronous close() call from FileDownloadChannel

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.6.0, 2.0.0, 2.1.0, 2.2.0
• Fix Version/s: 2.2.2, 2.3.0
• Component/s: Spark Core
• Labels: correctness
• Target Version/s: 2.3.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 07/Jan/18 23:09
• Updated: 18/Jan/18 18:28
• Resolved: 10/Jan/18 07:10

Description

Spark's Netty-based file transfer code contains an asynchronous IO bug which may lead to incorrect query results.

At a high-level, the problem is that an unsafe asynchronous `close()` of a pipe's source channel creates a race condition where file transfer code closes a file descriptor then attempts to read from it. If the closed file descriptor's number has been reused by an `open()` call then this invalid read may cause unrelated file operations to return incorrect results due to reading different data than intended.

I have a small, surgical fix for this bug and will submit a PR with more description on the specific race condition / underlying bug.

Issue Links

• links to: [Github] Pull Request #20179 (JoshRosen)