Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7240 Scaling HDFS
  3. HDFS-12443

Ozone: Improve SCM block deletion throttling algorithm

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HDFS-7240
    • ozone, scm
    • Reviewed

    Description

      Currently SCM scans delLog to send deletion transactions to datanode periodically, the throttling algorithm is simple, it scans at most BLOCK_DELETE_TX_PER_REQUEST_LIMIT (by default 50) at a time. This is non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN will only get 1 TX to proceed in an interval, this will make the deletion slow. An improvement to this is to make this throttling by datanode, e.g 50 TXs per datanode per interval.

      Attachments

        1. HDFS-12443-HDFS-7240.001.patch
          17 kB
          Yiqun Lin
        2. HDFS-12443-HDFS-7240.002.patch
          23 kB
          Yiqun Lin
        3. HDFS-12443-HDFS-7240.002.patch
          23 kB
          Yiqun Lin
        4. HDFS-12443-HDFS-7240.003.patch
          36 kB
          Yiqun Lin
        5. HDFS-12443-HDFS-7240.004.patch
          37 kB
          Yiqun Lin
        6. HDFS-12443-HDFS-7240.005.patch
          37 kB
          Yiqun Lin
        7. HDFS-12443-SCM-blockdeletion-throttle.pdf
          271 kB
          Weiwei Yang

        Issue Links

          is part of

          Sub-task - The sub-task of the issue HDFS-11922 Ozone: KSM: Garbage collect deleted blocks

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              linyiqun Yiqun Lin
              cheersyang Weiwei Yang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: