[HDFS-17045] File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: namenode, snapshots
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 12/Jun/23 15:58
• Updated: 13/Feb/24 16:35
• Resolved: 14/Jun/23 23:13

Description

HDFS-16972 added a shouldDestroy method which caused the following bug.

Background:

• When FileSystem.rename(src, dst) from a snapshottable dir (src) to a snapshottable dir (dst), dstSnapshotId is set to the latest snapshot at dst. As a result, dst is NOT in dstSnapshotId because dstSnapshotId was already taken before rename.
• snapshotToBeDeleted is the snapshot id of the current operation if the operation is FileSystem.deleteSnapshot. Otherwise, snapshotToBeDeleted is set to CURRENT_STATE_ID.
• If (snapshotToBeDeleted > dstSnapshotId), dst is in snapshotToBeDeleted. The shouldDestroy method returns true to continue deletion.
• If (snapshotToBeDeleted <= dstSnapshotId), dst must not be in snapshotToBeDeleted. The shouldDestroy method returns false to stop deletion.

All the above are correct for renaming within snapshottable directories.

Bug:

• If rename(src, dst) from a snapshottable dir (src) to a non-snapshottable dir (dst), dstSnapshotId becomes CURRENT_STATE_ID.
• When FileSystem.delete(dst) happens, snapshotToBeDeleted is also set to CURRENT_STATE_ID.
• In this case, snapshotToBeDeleted == dstSnapshotId, the shouldDestroy method will return false and it incorrectly stops the deletion.

Not that this bug may cause fsimage corruption and quota miscalculation since some files can be partially deleted. Fortunately, this bug won't cause data loss.

Issue Links

• is broken by: HDFS-16972 Delete a snapshot may deleteCurrentFile
• is related to: HDFS-17380 FsImageValidation: remove inaccessible nodes
• links to: GitHub Pull Request #5738