Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-529

More redundant block searches in BlockManager.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Found more redundant searches in BlockManager.addStoredBLock() and BlockManager.getNodes(Block).
      The pattern here is that we first call BlocksMap.numNodes(Block) in order to determine the size of the array of data-nodes. This is one search in blocksMap. Then we call BlocksMap.nodeIterator(Block), which also performs a search. This is all not necessary because we already have the block in the form of BlockInfo from blocksMap and do not need to search for it.

      1. BlocksMap.patch
        4 kB
        Konstantin Shvachko
      2. BlocksMap.patch
        4 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          By replacing type Block with BlockInfo for local variables and methods the patch avoids unnecessary searches.

          Show
          Konstantin Shvachko added a comment - By replacing type Block with BlockInfo for local variables and methods the patch avoids unnecessary searches.
          Hide
          Jakob Homan added a comment -

          Looks almost good. Talked offline about the TODO comment and the poor comment below one of the methods.

          Show
          Jakob Homan added a comment - Looks almost good. Talked offline about the TODO comment and the poor comment below one of the methods.
          Hide
          Konstantin Shvachko added a comment -

          Corrected the TODO comment, and provided meaningful comments for newIterator() methods.

          Show
          Konstantin Shvachko added a comment - Corrected the TODO comment, and provided meaningful comments for newIterator() methods.
          Hide
          Konstantin Shvachko added a comment -

          Both run-test-hdfs and test-patch pass:

          [exec] There appear to be 150 release audit warnings before the patch and 150 release audit warnings after applying the patch.
          [exec] -1 overall.  
          [exec]     +1 @author.  The patch does not contain any @author tags.
          [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
          [exec]                         Please justify why no new tests are needed for this patch.
          [exec]                         Also please list what manual steps were performed to verify this patch.
          [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
          [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          [exec] ======================================================================
          [exec] ======================================================================
          [exec]     Finished build.
          [exec] ======================================================================
          [exec] ======================================================================
          
          Show
          Konstantin Shvachko added a comment - Both run-test-hdfs and test-patch pass: [exec] There appear to be 150 release audit warnings before the patch and 150 release audit warnings after applying the patch. [exec] -1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #41 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/41/)
          . Use BlockInfo instead of Block to avoid redundant block searches in BlockManager. Contributed by Konstantin Shvachko.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #41 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/41/ ) . Use BlockInfo instead of Block to avoid redundant block searches in BlockManager. Contributed by Konstantin Shvachko.
          Hide
          Konstantin Shvachko added a comment -

          > -1 tests included.

          This is covered by existing tests. The patch does not introduce new functionality.

          Show
          Konstantin Shvachko added a comment - > -1 tests included. This is covered by existing tests. The patch does not introduce new functionality.

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development