[HDDS-10650] Deleting Volume/Bucket/Directory doesn't remove key from OpenKeyTable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDDS-7593
• Component/s: OM
• Labels: pull-request-available
• Target Version/s: 1.5.0
• Assignee: Ashish Kumar
• Reporter: Pratyush Bhatt
• Created: 04/Apr/24 17:53
• Updated: 29/Apr/24 15:23
• Resolved: 24/Apr/24 05:02

Description

Iteration 1: Volume Level Delete: Doesnt remove entry from OpenKeyTable
Create a file /hsyncvol/hsyncbuck/hsync/File_1.txt write some data, do hsync, and keep it open.
File is visible in OpenKeyTable

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214179318792293    1712252492047    Yes   /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt

Delete the volume containing the file

ozone sh volume delete -r -y hsyncvol
24/04/04 17:42:11 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/04 17:42:11 INFO client.ClientTrustManager: Loading certificates for client.
24/04/04 17:42:12 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/04 17:42:12 INFO client.ClientTrustManager: Loading certificates for client.
Volume hsyncvol is deleted

Still the file is present in OpenKeyTable

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214179318792293    1712252492047    Yes       /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt 

Iteration 2: Bucket Level Delete: Doesnt remove entry from OpenKeyTable{}
Again open the same file /hsyncvol/hsyncbuck/hsync/File_1.txt , write some data, do hsync.
File is visible in OpenKeyTable with a new ClientID

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214183435108455    1712252554857    Yes        /hsyncvol/hsyncbuck/-9223372036851769599/File_1.txt
112214179318792293    1712252492047    Yes        /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt 

Delete the Bucket containing the file

ozone fs -rm -r -skipTrash ofs://ozone1712159573//hsyncvol/hsyncbuck/
24/04/04 17:43:26 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/04 17:43:27 INFO client.ClientTrustManager: Loading certificates for client.
Deleted ofs://ozone1712159573/hsyncvol/hsyncbuck/ 

Still the same OpenKeyTable

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214183435108455    1712252554857    Yes        /hsyncvol/hsyncbuck/-9223372036851769599/File_1.txt
112214179318792293    1712252492047    Yes        /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt 

Iteration 3: Directory Level Delete: Doesnt remove entry from OpenKeyTable{}
Again open the same file /hsyncvol/hsyncbuck/hsync/File_1.txt , write some data, do hsync.
File is visible in OpenKeyTable with a new ClientID

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214188917260394    1712252638507    Yes        /hsyncvol/hsyncbuck/-9223372036851761407/File_1.txt
112214183435108455    1712252554857    Yes        /hsyncvol/hsyncbuck/-9223372036851769599/File_1.txt
112214179318792293    1712252492047    Yes        /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt 

Delete the directory containing the file
Still the same OpenKeyTable

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214188917260394    1712252638507    Yes        /hsyncvol/hsyncbuck/-9223372036851761407/File_1.txt
112214183435108455    1712252554857    Yes        /hsyncvol/hsyncbuck/-9223372036851769599/File_1.txt
112214179318792293    1712252492047    Yes        /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt 

Iteration 4: File Level Delete: Removes entry from OpenKeyTable
Again open the same file /hsyncvol/hsyncbuck/hsync/File_1.txt , write some data, do hsync.
File is visible in OpenKeyTable with a new ClientID

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214201150865518    1712252825177    Yes        /hsyncvol/hsyncbuck/-9223372036851751679/File_1.txt
112214188917260394    1712252638507    Yes        /hsyncvol/hsyncbuck/-9223372036851761407/File_1.txt
112214183435108455    1712252554857    Yes        /hsyncvol/hsyncbuck/-9223372036851769599/File_1.txt
112214179318792293    1712252492047    Yes        /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt 

Delete the file itself

ozone fs -rm -r -skipTrash ofs://ozone1712159573//hsyncvol/hsyncbuck/hsync/File_1.txt
24/04/04 17:47:45 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/04 17:47:46 INFO client.ClientTrustManager: Loading certificates for client.
Deleted ofs://ozone1712159573/hsyncvol/hsyncbuck/hsync/File_1.txt 

This time the entry is removed from the OpenKeyTable

ozone admin om lof --service-id=ozone1712159573  | grep File_1.txt
112214188917260394    1712252638507    Yes        /hsyncvol/hsyncbuck/-9223372036851761407/File_1.txt
112214183435108455    1712252554857    Yes        /hsyncvol/hsyncbuck/-9223372036851769599/File_1.txt
112214179318792293    1712252492047    Yes        /hsyncvol/hsyncbuck/-9223372036851775231/File_1.txt 

Summary: Only deleting the file directly is removing the entry from the OpenKeyTable, rest are keeping it intact.

Expected Behavior: Shouldn't it remove the entry in all the cases from the OpenKeyTable?

 

Issue Links

• relates to: HDDS-9930 Deleted file reappears after HSync
• links to: GitHub Pull Request #6495