[HDFS-6929] NN periodically unlinks lazy persist files with missing replicas from namespace

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Arpit Agarwal
• Reporter: Arpit Agarwal
• Created: 23/Aug/14 21:07
• Updated: 28/Apr/15 10:12
• Resolved: 27/Aug/14 22:59

Description

Occasional data loss is expected when using the lazy persist flag due to node restarts. The NN will optionally unlink lazy persist files from the namespace to avoid them from showing up as corrupt files.

This behavior can be turned off with a global option. In the future this may be made a per-file option controllable by the client.

Issue Links

• is related to: HDFS-8229 LAZY_PERSIST file gets deleted after NameNode restart.
• is related to: HDFS-8276 LazyPersistFileScrubber should be disabled if scrubber interval configured zero