[HDFS-3510] Improve FSEditLog pre-allocation

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 2.0.0-alpha
• Fix Version/s: 2.0.3-alpha
• Hadoop Flags: Reviewed
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 05/Jun/12 22:15
• Updated: 03/Sep/14 23:09
• Resolved: 28/Jun/12 23:01

Description

It is good to avoid running out of space in the middle of writing a batch of edits, because when it happens, we often get partial edits at the end of the log.
Edit log preallocation can solve this problem (see HADOOP-2330 for a full description of edit log preallocation).

The current pre-allocation code was introduced for performance reasons, not for preventing partial edits. As a consequence, we sometimes do a write without using pre-allocation. We should change the pre-allocation code so that it always preallocates at least enough space before writing out the edits.

Issue Links

• is related to: HDFS-3531 EditLogFileOutputStream#preallocate should check for incomplete writes
• is related to: HADOOP-2330 Preallocate transaction log to improve namenode transaction logging performance
• is related to: HDFS-1846 Don't fill preallocated portion of edits log with 0x00
• is related to: HDFS-3596 Improve FSEditLog pre-allocation in branch-1