[HDDS-9930] Deleted file reappears after HSync

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.0.0, HDDS-7593
• Component/s: Ozone Manager
• Labels: pull-request-available
• Assignee: Siyao Meng
• Reporter: Pratyush Bhatt
• Created: 15/Dec/23 09:19
• Updated: 10/Apr/24 07:11
• Resolved: 26/Jan/24 22:42

Description

Scenario:
Open multiple FSO files in Ozone, write data and do Hsync. In middle Remove some file with -skipTrash.

Before deleting File_98.txt and File_99.txt

[root@ccycloud-1 ~]# ozone fs -du -s -h ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/* | grep File_9
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_9.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_90.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_91.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_92.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_93.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_94.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_95.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_96.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_97.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_98.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_99.txt
[root@ccycloud-1 ~]# 

Removing both 98 and 99th file

[root@ccycloud-1 ~]# ozone fs -rm -r -skipTrash ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_99.txt
Deleted ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_99.txt
[root@ccycloud-1 ~]# ozone fs -rm -r -skipTrash ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_98.txt
Deleted ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_98.txt
[root@ccycloud-1 ~]# 

File gets removed:

[root@ccycloud-1 ~]# ozone fs -du -s -h ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/* | grep File_9
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_9.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_90.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_91.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_92.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_93.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_94.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_95.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_96.txt
766 K  2.2 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_97.txt
[root@ccycloud-1 ~]#
[root@ccycloud-1 ~]# 

After the next Hsync call in test, Both file reappears with the exact length as if they were not deleted previously.

[root@ccycloud-1 ~]# ozone fs -du -s -h ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/* | grep File_9
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_9.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_90.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_91.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_92.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_93.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_94.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_95.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_96.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_97.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_98.txt
771 K  2.3 M  ofs://ozone1702444879/testsyncvol1702619029/testsyncbuck1702619029/File_99.txt 

Issue Links

• fixes: HDDS-9176 two client parallel perform commit with Hsync can cause dataloss
• is related to: HDDS-10650 Deleting Volume/Bucket/Directory doesn't remove key from OpenKeyTable
• links to: GitHub Pull Request #6079
• links to: GitHub Pull Request #6472