Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7240 Scaling HDFS
  3. HDFS-12423

Ozone: TopN container choosing policy should ignore containers that has no pending deletion blocks

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-7240
    • Fix Version/s: HDFS-7240
    • Component/s: ozone
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      TopN container choosing policy should ignore unnecessary containers. Currently TopN policy selects specified count of containers but not check the number of pending deletion blocks. So there is a chance there will be some unnecessary containers being chosen. That is say we will choose some containers that don't include any pending deletion blocks. The related output:

      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending deletion blocks num: 1.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending deletion blocks num: 0.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending deletion blocks num: 0.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending deletion blocks num: 0.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending deletion blocks num: 0.
      

      We should ignore these containers which pending deletion blocks num is 0, this can reduce some unnecessary block-deletion iterations.

        Attachments

        1. HDFS-12423-HDFS-7240.001.patch
          5 kB
          Yiqun Lin
        2. HDFS-12423-HDFS-7240.002.patch
          7 kB
          Yiqun Lin

          Activity

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: