[HDFS-3107] HDFS truncate

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7.0
• Component/s: datanode, namenode
• Hadoop Flags: Reviewed
• Assignee: Plamen Jeliazkov
• Reporter: Lei Chang
• Created: 16/Mar/12 04:53
• Updated: 26/May/16 17:03
• Resolved: 13/Jan/15 08:57

Description

Systems with transaction support often need to undo changes made to the underlying storage when a transaction is aborted. Currently HDFS does not support truncate (a standard Posix operation) which is a reverse operation of append, which makes upper layer applications use ugly workarounds (such as keeping track of the discarded byte range per file in a separate metadata store, and periodically running a vacuum process to rewrite compacted files) to overcome this limitation of HDFS.

Issue Links

• is duplicated by: HDFS-7341 Add initial snapshot support based on pipeline recovery
• is duplicated by: HADOOP-4109 FileSystem support for posix truncate method
• is related to: HDFS-9164 hdfs-nfs connector fails on O_TRUNC
• relates to: HDFS-9516 truncate file fails with data dirs on multiple disks
• relates to: HDFS-7606 Missing null check in INodeFile#getBlocks()
• relates to: HADOOP-11490 Expose truncate API via FileSystem and shell command
• relates to: HADOOP-11510 Expose truncate API via FileContext
• relates to: MAPREDUCE-6227 DFSIO for truncate
• relates to: MAPREDUCE-6228 Add truncate operation to SLive
• relates to: HADOOP-11579 Documentation for truncate

Sub-Tasks

• Add a truncate test with cached data: Open