Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13529

Fix default trash policy emptier trigger time correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0, 2.7.6, 3.2.0, 2.9.2, 2.8.5
    • Fix Version/s: 3.3.0, 3.2.1, 3.1.3
    • Component/s: namenode
    • Labels:
      None

      Description

      Trash emptier is designed to auto trigger at UTC 00:00, however I am confused all the time that it usually triggers at a few minutes even half a hour after UTC 00:00 actually in our production cluster.
      The main reason is default policy emptier thread sleep more time than as expect, since it does not consider the delete operation time cost itself. especially for a large cluster, auto trash cleaner may cost dozens of minutes.
      The right way is that gets current time now before calculate end time.

            long now = Time.now();
            while (true) {
              end = ceiling(now, emptierInterval);
              try {                                     // sleep for interval
                Thread.sleep(end - now);
              } catch (InterruptedException e) {
                break;                                  // exit on interrupt
              }
      
              try {
                now = Time.now();
                ...... // delete trash checkpoint
              } catch (Exception e) {
                LOG.warn("RuntimeException during Trash.Emptier.run(): ", e); 
              }
            }
      

        Attachments

        1. HDFS-13529-trunk.001.patch
          0.9 kB
          Xiaoqiao He

          Issue Links

            Activity

              People

              • Assignee:
                hexiaoqiao Xiaoqiao He
                Reporter:
                hexiaoqiao Xiaoqiao He
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: