[HDFS-7941] hsync() not working for SequenceFile

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: hdfs-client

Environment

HDP 2.2 running on Redhat

• Assignee: Unassigned
• Reporter: Sverre Bakke
• Created: 17/Mar/15 15:18
• Updated: 10/Jun/16 08:50

Description

When using SequenceFile.Writer and appending+syncing to file repeatedly, the sync does not appear to work other than:

• once after writing headers
• when closing.

Imagine the following test case:
http://pastebin.com/Y9xysCRX

This code would append a new record every second and then immediately sync it. One would also imagine that the file would grow for every append, however, this does not happen.

After watching the behavior I have noticed that it only syncs the headers at the very beginning (providing a file of 164 bytes) and then never again until its closed. This despite it is asked to hsync() after every append.

Looking into the debug logs, this also claims the same behavior (executed the provided code example and grepped for "sync"):

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms

This was the only time the code ran fsync throughout the entire execution.

This has been tested (with similar result) for the following deployments:

• sequencefile with no compression
• sequencefile with record compression
• sequencefile with block compression
• textfile with no compression