Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2239

BlockPlacementPolicyRaid should call getBlockLocations only when necessary

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently BlockPlacementPolicyRaid calls getBlockLocations for every chooseTarget().
      This puts pressure on NameNode. We should avoid calling if this file is not raided or a parity file.

      1. MAPREDUCE-2239-3.txt
        15 kB
        Scott Chen
      2. MAPREDUCE-2239-2.txt
        14 kB
        Scott Chen
      3. MAPREDUCE-2239-1.txt
        16 kB
        Scott Chen
      4. MAPREDUCE-2239.txt
        16 kB
        Scott Chen

        Activity

        Hide
        Ramkumar Vadali added a comment -

        Do you need to change FSNamesystem.LOG.debug -> FSNamesystem.LOG.info?

        Show
        Ramkumar Vadali added a comment - Do you need to change FSNamesystem.LOG.debug -> FSNamesystem.LOG.info?
        Hide
        Scott Chen added a comment -

        Good catch. I have changed it to debug.

        Show
        Scott Chen added a comment - Good catch. I have changed it to debug.
        Hide
        Scott Chen added a comment -

        Rebased

        Show
        Scott Chen added a comment - Rebased
        Hide
        Scott Chen added a comment -
             [exec]
             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
             [exec]     +1 system test framework.  The patch passed system test framework compile.
             [exec]
             [exec]
             [exec]
             [exec]
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]     Finished build.
             [exec] ======================================================================
             [exec] ======================================================================
        
        Show
        Scott Chen added a comment - [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12469219/MAPREDUCE-2239-2.txt
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/41//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469219/MAPREDUCE-2239-2.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/41//console This message is automatically generated.
        Hide
        Tom White added a comment -

        This patch has fallen out of date, can you please regenerate it?

        Show
        Tom White added a comment - This patch has fallen out of date, can you please regenerate it?
        Hide
        Scott Chen added a comment -

        Thanks Tom. I will rebase this patch and upload again.

        Show
        Scott Chen added a comment - Thanks Tom. I will rebase this patch and upload again.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472269/MAPREDUCE-2239-3.txt
        against trunk revision 1075670.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/108//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472269/MAPREDUCE-2239-3.txt against trunk revision 1075670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/108//console This message is automatically generated.
        Hide
        Scott Chen added a comment -

        I forgot to use --no-prefix for my git patch. This time it should be correct.

        Show
        Scott Chen added a comment - I forgot to use --no-prefix for my git patch. This time it should be correct.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472357/MAPREDUCE-2239-3.txt
        against trunk revision 1075670.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.mapreduce.security.TestTokenCache

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472357/MAPREDUCE-2239-3.txt against trunk revision 1075670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapreduce.security.TestTokenCache -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//console This message is automatically generated.
        Hide
        Scott Chen added a comment -

        The failed tests are unrelated to the code path.
        I run TestTokenCache again on this patch. It succeeded.

        Show
        Scott Chen added a comment - The failed tests are unrelated to the code path. I run TestTokenCache again on this patch. It succeeded.
        Hide
        Ramkumar Vadali added a comment -

        +1
        Patch looks good

        Show
        Ramkumar Vadali added a comment - +1 Patch looks good
        Hide
        Scott Chen added a comment -

        I just committed this. Thanks for the review, Ram

        Show
        Scott Chen added a comment - I just committed this. Thanks for the review, Ram
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development