Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11634

Optimize BlockIterator when iterating starts in the middle.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.6.5
    • 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
    • None
    • None
    • Reviewed

    Description

      BlockManager.getBlocksWithLocations() needs to iterate blocks from a randomly selected startBlock index. It creates an iterator which points to the first block and then skips all blocks until startBlock. It is inefficient when DN has multiple storages. Instead of skipping blocks one by one we can skip entire storages. Should be more efficient on average.

      Attachments

        1. HDFS-11634.001.patch
          3 kB
          Konstantin Shvachko
        2. HDFS-11634.002.patch
          12 kB
          Konstantin Shvachko
        3. HDFS-11634.003.patch
          12 kB
          Konstantin Shvachko
        4. HDFS-11634.004.patch
          12 kB
          Konstantin Shvachko
        5. HDFS-11643.005.patch
          11 kB
          Konstantin Shvachko
        6. HDFS-11634.006.patch
          11 kB
          Konstantin Shvachko
        7. HDFS-11634-branch-2.7.006.patch
          12 kB
          Konstantin Shvachko
        8. Screen Shot 2017-06-21 at 1.39.21 PM.png
          124 kB
          Zhe Zhang

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-8825 Enhancements to Balancer

          • Major - Major loss of function.
          • Open

          Activity

            People

              shv Konstantin Shvachko
              shv Konstantin Shvachko
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: