[HDFS-3107] HDFS truncate

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7.0
• Component/s: datanode, namenode
• Hadoop Flags: Reviewed
• Assignee: Plamen Jeliazkov
• Reporter: Lei Chang
• Created: 16/Mar/12 04:53
• Updated: 26/May/16 17:03
• Resolved: 13/Jan/15 08:57

Description

Systems with transaction support often need to undo changes made to the underlying storage when a transaction is aborted. Currently HDFS does not support truncate (a standard Posix operation) which is a reverse operation of append, which makes upper layer applications use ugly workarounds (such as keeping track of the discarded byte range per file in a separate metadata store, and periodically running a vacuum process to rewrite compacted files) to overcome this limitation of HDFS.

Issue Links

• is duplicated by: HDFS-7341 Add initial snapshot support based on pipeline recovery
• is duplicated by: HADOOP-4109 FileSystem support for posix truncate method
• is related to: HDFS-9164 hdfs-nfs connector fails on O_TRUNC
• relates to: HDFS-9516 truncate file fails with data dirs on multiple disks
• relates to: HDFS-7606 Missing null check in INodeFile#getBlocks()
• relates to: HADOOP-11490 Expose truncate API via FileSystem and shell command
• relates to: HADOOP-11510 Expose truncate API via FileContext
• relates to: MAPREDUCE-6227 DFSIO for truncate
• relates to: MAPREDUCE-6228 Add truncate operation to SLive
• relates to: HADOOP-11579 Documentation for truncate

Sub-Tasks

• Snapshot support for truncate: Closed
• Tests for truncate CLI: Closed
• Disallow truncation of Lazy persist files: Closed
• Small fix and few refinements for FSN#truncate: Closed
• Test case to ensure lazy persist files cannot be truncated: Closed
• Expose truncate API for Web HDFS: Closed
• Expose truncate API for HDFS httpfs: Closed
• We should check the new length of truncate can't be a negative value.: Closed
• Fix TestFileTruncate to avoid bug of HDFS-7611: Closed
• DistributedFileSystem#truncate should resolve symlinks: Closed
• Add more tests for truncate: Closed
• Test truncate with DataNodes restarting: Closed
• Add a test randomly mixing append, truncate and snapshot: Closed
• Add a truncate test with cached data: Open
• Document truncate for WebHDFS.: Closed
• Expose truncate API for libhdfs: Closed
• A truncated file is corrupted after rollback from a rolling upgrade: Closed
• Cannot recover block after truncate and delete snapshot: Closed