Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11634

Optimize BlockIterator when iterating starts in the middle.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.5
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      BlockManager.getBlocksWithLocations() needs to iterate blocks from a randomly selected startBlock index. It creates an iterator which points to the first block and then skips all blocks until startBlock. It is inefficient when DN has multiple storages. Instead of skipping blocks one by one we can skip entire storages. Should be more efficient on average.

        Attachments

        1. Screen Shot 2017-06-21 at 1.39.21 PM.png
          124 kB
          Zhe Zhang
        2. HDFS-11634-branch-2.7.006.patch
          12 kB
          Konstantin Shvachko
        3. HDFS-11634.006.patch
          11 kB
          Konstantin Shvachko
        4. HDFS-11643.005.patch
          11 kB
          Konstantin Shvachko
        5. HDFS-11634.004.patch
          12 kB
          Konstantin Shvachko
        6. HDFS-11634.003.patch
          12 kB
          Konstantin Shvachko
        7. HDFS-11634.002.patch
          12 kB
          Konstantin Shvachko
        8. HDFS-11634.001.patch
          3 kB
          Konstantin Shvachko

          Issue Links

            Activity

              People

              • Assignee:
                shv Konstantin Shvachko
                Reporter:
                shv Konstantin Shvachko
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: