Affects Version/s: None
Fix Version/s: 0.18.0
Hadoop Flags:Incompatible change, Reviewed
Release Note:Added sync() method to FSDataOutputStream to really, really persist data in HDFS. InterDatanodeProtocol to implement this feature.
DFSOutputStream has a method called flush() that persists block locations on the namenode and sends all outstanding data to all datanodes in the pipeline. However, this data goes to the tmp file on the datanode(s). When the block is closed, the tmp files is renamed to be the real block file. If the datanode(s) dies before the block is compete, then entire block is lost. This behaviour wil be fixed in
However, in the short term, a configuration paramater can be used to allow datanodes to write to the real block file directly, thereby avoiding writing to the tmp file. This means that data that is flushed successfully by a client does not get lost even if the datanode(s) or client dies.
The Namenode already has code to pick the largest replica (if multiple datanodes have different sizes of this block). Also, the namenode has code to not trigger replication request if the file is still being written to.
The only caveat that I can think of is that the block report periodicity should be much much smaller that the lease timeout period. A block report adds the being-written-to blocks to the blocksMap thereby avoiding any cleanup that a lease expiry processing might have otherwise done.
Not all requirements specified by
HADOOP-1700 are supported by this approach, but it could still be helpful (in the short term) for a wide range of applications.
|Component/s||dfs [ 12310710 ]|
|Status||Resolved [ 5 ]||Closed [ 6 ]|
|Hadoop Flags||[Reviewed, Incompatible change]||[Incompatible change, Reviewed]|
|Release Note||An application can invoke sync on the FSDataOutputStream to really, really persist data in HDFS! This is an incompatible change becuase it required changes to InterDatanodeProtocol.||Added sync() method to FSDataOutputStream to really, really persist data in HDFS. InterDatanodeProtocol to implement this feature.|
|Status||Patch Available [ 10002 ]||Resolved [ 5 ]|
|Release Note||An application can invoke sync on the FSDataOutputStream to really, really persist data in HDFS! This is an incompatible change becuase it required changes to InterDatanodeProtocol.|
|Resolution||Fixed [ 1 ]|
|Hadoop Flags||[Reviewed]||[Incompatible change, Reviewed]|
|Fix Version/s||0.18.0 [ 12312972 ]|
|Status||Patch Available [ 10002 ]||Open [ 1 ]|
|Summary||Provide a configurable way for DFSOututStream.flush() to flush data to real block file on DataNode.||DFSOututStream.flush() should flush data to real block file on DataNode.|
|Field||Original Value||New Value|
|Assignee||dhruba borthakur [ dhruba ]|