Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15880

WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.7.3
    • Fix Version/s: None
    • Component/s: documentation, fs/azure
    • Labels:
    • Environment:

      Any HDInsigth cluster pointing to WASB. 

      Description

      when "fs.trash.interval" is set to a value,  trash for the local hdfs got cleared where as the trash folder on WASB doesn't get deleted and the files get piled up on WASB store..

      WASB doesn't pick up  fs.trash.interval value and this fails to auto purge trash folder on WASB store.

       

      Issue : WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder

      Steps to reproduce Scenario:

      Delete any file stored on HDFS

      hdfs dfs -D "fs.default.name=hdfs://mycluster/" -rm /hivestore.txt
      18/10/23 06:18:05 INFO fs.TrashPolicyDefault: Moved: 'hdfs://mycluster/hivestore.txt' to trash at: hdfs://mycluster/user/sshuser/.Trash/Current/hivestore.txt

      When deleted the file is moved to trash folder
      hdfs dfs -rm wasb:///hivestore.txt
      18/10/23 06:19:13 INFO fs.TrashPolicyDefault: Moved: 'wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/hivestore.txt' to trash at: wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt

      Reduced the fs.trash.interval from 360 to 1 and restarted all related services.

      Trash for the local hdfs gets cleared honoring the "fs.trash.interval" value.

      hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls hdfs://mycluster/user/sshuser/.Trash/Current/
      ls: File hdfs://mycluster/user/sshuser/.Trash/Current does not exist.

      Where as the trash for WASB doesn't get cleared.

      hdfs dfs -ls wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/
      Found 1 items
      rw-rr- 1 sshuser supergroup 1084 2018-10-23 06:19 wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Sunilkc Sunil Kumar Chakrapani
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: