[HDFS-3689] Add support for variable length block - ASF JIRA

XML

Word

Printable

JSON

Hadoop Flags:

Reviewed
Release Note:

Hide
1. HDFS now can choose to append data to a new block instead of end of the last partial block. Users can pass {{CreateFlag.APPEND}} and {{CreateFlag.NEW_BLOCK}} to the {{append}} API to indicate this requirement.
2. HDFS now allows users to pass {{SyncFlag.END_BLOCK}} to the {{hsync}} API to finish the current block and write remaining data to a new block.

Show
1. HDFS now can choose to append data to a new block instead of end of the last partial block. Users can pass {{CreateFlag.APPEND}} and {{CreateFlag.NEW_BLOCK}} to the {{append}} API to indicate this requirement. 2. HDFS now allows users to pass {{SyncFlag.END_BLOCK}} to the {{hsync}} API to finish the current block and write remaining data to a new block.

Currently HDFS supports fixed length blocks. Supporting variable length block will allow new use cases and features to be built on top of HDFS.

breaks

HDFS-13120 Snapshot diff could be corrupted after concat

HDFS-8891 HDFS concat should keep srcs order

is depended upon by

HDFS-7703 Support favouredNodes for the append for new blocks

is required by

YARN-2942 Aggregated Log Files should be combined

relates to

HDFS-7943 Append cannot handle the last block with length greater than the preferred block size

HDFS-6867 For DFSOutputStream, do pipeline recovery for a single block in the background

(1 relates to)