[HADOOP-15880] WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.7.3
• Component/s: documentation, fs/azure
• Labels: WASB pull-request-available

Environment

Any HDInsigth cluster pointing to WASB. 

• Assignee: Unassigned
• Reporter: Sunil Kumar Chakrapani
• Created: 25/Oct/18 00:39
• Updated: 08/Mar/21 06:35

Description

when "fs.trash.interval" is set to a value,  trash for the local hdfs got cleared where as the trash folder on WASB doesn't get deleted and the files get piled up on WASB store..

WASB doesn't pick up  fs.trash.interval value and this fails to auto purge trash folder on WASB store.

 

Issue : WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder

Steps to reproduce Scenario:

Delete any file stored on HDFS

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -rm /hivestore.txt
18/10/23 06:18:05 INFO fs.TrashPolicyDefault: Moved: 'hdfs://mycluster/hivestore.txt' to trash at: hdfs://mycluster/user/sshuser/.Trash/Current/hivestore.txt

When deleted the file is moved to trash folder
hdfs dfs -rm wasb:///hivestore.txt
18/10/23 06:19:13 INFO fs.TrashPolicyDefault: Moved: 'wasb://[e-mail]/hivestore.txt' to trash at: wasb://[e-mail]/user/sshuser/.Trash/Current/hivestore.txt

Reduced the fs.trash.interval from 360 to 1 and restarted all related services.

Trash for the local hdfs gets cleared honoring the "fs.trash.interval" value.

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls hdfs://mycluster/user/sshuser/.Trash/Current/
ls: File hdfs://mycluster/user/sshuser/.Trash/Current does not exist.

Where as the trash for WASB doesn't get cleared.

hdfs dfs -ls wasb://[e-mail]/user/sshuser/.Trash/Current/
Found 1 items
rw-rr- 1 sshuser supergroup 1084 2018-10-23 06:19 wasb://[e-mail]/user/sshuser/.Trash/Current/hivestore.txt

 

Issue Links

• links to: GitHub Pull Request #2750