Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7240 Scaling HDFS
  3. HDFS-12423

Ozone: TopN container choosing policy should ignore containers that has no pending deletion blocks

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • HDFS-7240
    • HDFS-7240
    • ozone
    • Reviewed

    Description

      TopN container choosing policy should ignore unnecessary containers. Currently TopN policy selects specified count of containers but not check the number of pending deletion blocks. So there is a chance there will be some unnecessary containers being chosen. That is say we will choose some containers that don't include any pending deletion blocks. The related output:

      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion, pending deletion blocks num: 1.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion, pending deletion blocks num: 0.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion, pending deletion blocks num: 0.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion, pending deletion blocks num: 0.
      17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy: Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion, pending deletion blocks num: 0.
      

      We should ignore these containers which pending deletion blocks num is 0, this can reduce some unnecessary block-deletion iterations.

      Attachments

        1. HDFS-12423-HDFS-7240.001.patch
          5 kB
          Yiqun Lin
        2. HDFS-12423-HDFS-7240.002.patch
          7 kB
          Yiqun Lin

        Activity

          People

            linyiqun Yiqun Lin
            linyiqun Yiqun Lin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: