[HDFS-4677] Editlog should support synchronous writes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1-win, 3.0.0-alpha1
• Fix Version/s: 1-win, 2.1.0-beta
• Target Version/s: 1-win
• Hadoop Flags: Reviewed
• Assignee: Ivan Mitic
• Reporter: Ivan Mitic
• Created: 09/Apr/13 06:57
• Updated: 12/May/16 18:15
• Resolved: 09/Jun/13 00:22

Description

In the current implementation, NameNode editlog performs syncs to the persistent storage using the FileChannel#force Java APIs. This API is documented to be slower compared to an alternative where RandomAccessFile is opened with "rws" flags (synchronous writes).

We instrumented FileChannel#force on Windows and it some software/hardware configurations it can perform significantly slower than the “rws” alternative.

In terms of the Windows APIs, FileChannel#force internally calls FlushFileBuffers while RandomAccessFile (“rws”) opens the file with the FILE_FLAG_WRITE_THROUGH flag.

With this Jira I'd like to introduce a flag that provide means to configure NameNode to use synchronous writes. There is a catch though, the behavior of the "rws" flags is platform and hardware specific and might not provide the same level of guarantees as FileChannel#force w.r.t. flushing the on-disk cache. This is an expert level setting, and it should be documented as such.

Issue Links

• is related to: HDFS-3602 Enhancements to HDFS for Windows Server and Windows Azure development and runtime environments
• is related to: HADOOP-8645 Stabilize branch-1-win