Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7240 Scaling HDFS
  3. HDFS-12443

Ozone: Improve SCM block deletion throttling algorithm

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HDFS-7240
    • Component/s: ozone, scm
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently SCM scans delLog to send deletion transactions to datanode periodically, the throttling algorithm is simple, it scans at most BLOCK_DELETE_TX_PER_REQUEST_LIMIT (by default 50) at a time. This is non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN will only get 1 TX to proceed in an interval, this will make the deletion slow. An improvement to this is to make this throttling by datanode, e.g 50 TXs per datanode per interval.

        Attachments

        1. HDFS-12443-HDFS-7240.001.patch
          17 kB
          Yiqun Lin
        2. HDFS-12443-HDFS-7240.002.patch
          23 kB
          Yiqun Lin
        3. HDFS-12443-HDFS-7240.002.patch
          23 kB
          Yiqun Lin
        4. HDFS-12443-HDFS-7240.003.patch
          36 kB
          Yiqun Lin
        5. HDFS-12443-HDFS-7240.004.patch
          37 kB
          Yiqun Lin
        6. HDFS-12443-HDFS-7240.005.patch
          37 kB
          Yiqun Lin
        7. HDFS-12443-SCM-blockdeletion-throttle.pdf
          271 kB
          Weiwei Yang

          Issue Links

            Activity

              People

              • Assignee:
                linyiqun Yiqun Lin
                Reporter:
                cheersyang Weiwei Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: