[SPARK-4307] Initialize FileDescriptor lazily in FileRegion

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 09/Nov/14 05:05
• Updated: 17/May/20 18:31
• Resolved: 11/Nov/14 08:26

Description

We use Netty's DefaultFileRegion to do zero copy send. However, DefaultFileRegion requires a FileDescriptor, which results in a large number of opened files in larger workloads.

Issue Links

• links to: [Github] Pull Request #3172 (rxin)