[HDFS-7291] Persist in-memory replicas with appropriate unbuffered copy API on POSIX and Windows

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Xiaoyu Yao
• Reporter: Xiaoyu Yao
• Created: 27/Oct/14 00:09
• Updated: 01/Dec/14 03:09
• Resolved: 28/Oct/14 21:28

Description

HDFS-7090 changes to persist in-memory replicas using unbuffered IO on Linux and Windows. On Linux distribution, it relies on the sendfile() API between two file descriptors to achieve unbuffered IO copy. According to Linux document at http://man7.org/linux/man-pages/man2/sendfile.2.html, this is only supported on Linux kernel 2.6.33+.

As pointed by Haowei in the discussion below, FileChannel#transferTo already has support for native unbuffered IO on POSIX platform. On Windows, JDK 6/7/8 has not implemented native unbuffered IO yet. We change to use FileChannel#transfer for POSIX and our own native wrapper of CopyFileEx on Windows for unbuffered copy.

Issue Links

• relates to: HDFS-7090 Use unbuffered writes when persisting in-memory replicas