Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2239

BlockPlacementPolicyRaid should call getBlockLocations only when necessary

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently BlockPlacementPolicyRaid calls getBlockLocations for every chooseTarget().
      This puts pressure on NameNode. We should avoid calling if this file is not raided or a parity file.

      1. MAPREDUCE-2239.txt
        16 kB
        Scott Chen
      2. MAPREDUCE-2239-1.txt
        16 kB
        Scott Chen
      3. MAPREDUCE-2239-2.txt
        14 kB
        Scott Chen
      4. MAPREDUCE-2239-3.txt
        15 kB
        Scott Chen

        Activity

        Scott Chen created issue -
        Scott Chen made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-2239.txt [ 12468321 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ramkumar Vadali added a comment -

        Do you need to change FSNamesystem.LOG.debug -> FSNamesystem.LOG.info?

        Show
        Ramkumar Vadali added a comment - Do you need to change FSNamesystem.LOG.debug -> FSNamesystem.LOG.info?
        Scott Chen made changes -
        Attachment MAPREDUCE-2239-1.txt [ 12468340 ]
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Scott Chen added a comment -

        Good catch. I have changed it to debug.

        Show
        Scott Chen added a comment - Good catch. I have changed it to debug.
        Scott Chen made changes -
        Attachment MAPREDUCE-2239-2.txt [ 12469219 ]
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Scott Chen added a comment -

        Rebased

        Show
        Scott Chen added a comment - Rebased
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Scott Chen added a comment -
             [exec]
             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
             [exec]     +1 system test framework.  The patch passed system test framework compile.
             [exec]
             [exec]
             [exec]
             [exec]
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]     Finished build.
             [exec] ======================================================================
             [exec] ======================================================================
        
        Show
        Scott Chen added a comment - [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ======================================================================
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12469219/MAPREDUCE-2239-2.txt
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/41//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469219/MAPREDUCE-2239-2.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/41//console This message is automatically generated.
        Hide
        Tom White added a comment -

        This patch has fallen out of date, can you please regenerate it?

        Show
        Tom White added a comment - This patch has fallen out of date, can you please regenerate it?
        Tom White made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Scott Chen added a comment -

        Thanks Tom. I will rebase this patch and upload again.

        Show
        Scott Chen added a comment - Thanks Tom. I will rebase this patch and upload again.
        Scott Chen made changes -
        Attachment MAPREDUCE-2239-3.txt [ 12472269 ]
        Scott Chen made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Scott Chen made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472269/MAPREDUCE-2239-3.txt
        against trunk revision 1075670.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/108//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472269/MAPREDUCE-2239-3.txt against trunk revision 1075670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/108//console This message is automatically generated.
        Scott Chen made changes -
        Attachment MAPREDUCE-2239-3.txt [ 12472269 ]
        Scott Chen made changes -
        Attachment MAPREDUCE-2239-3.txt [ 12472357 ]
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Scott Chen made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Hide
        Scott Chen added a comment -

        I forgot to use --no-prefix for my git patch. This time it should be correct.

        Show
        Scott Chen added a comment - I forgot to use --no-prefix for my git patch. This time it should be correct.
        Scott Chen made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Hadoop Flags [Reviewed]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472357/MAPREDUCE-2239-3.txt
        against trunk revision 1075670.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.mapreduce.security.TestTokenCache

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472357/MAPREDUCE-2239-3.txt against trunk revision 1075670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapreduce.security.TestTokenCache -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/109//console This message is automatically generated.
        Hide
        Scott Chen added a comment -

        The failed tests are unrelated to the code path.
        I run TestTokenCache again on this patch. It succeeded.

        Show
        Scott Chen added a comment - The failed tests are unrelated to the code path. I run TestTokenCache again on this patch. It succeeded.
        Hide
        Ramkumar Vadali added a comment -

        +1
        Patch looks good

        Show
        Ramkumar Vadali added a comment - +1 Patch looks good
        Hide
        Scott Chen added a comment -

        I just committed this. Thanks for the review, Ram

        Show
        Scott Chen added a comment - I just committed this. Thanks for the review, Ram
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        9d 22h 35m 4 Scott Chen 31/Jan/11 23:49
        Patch Available Patch Available Open Open
        46d 55m 5 Scott Chen 01/Mar/11 21:45
        Open Open In Progress In Progress
        20h 25m 2 Scott Chen 01/Mar/11 21:45
        In Progress In Progress Patch Available Patch Available
        1m 2 Scott Chen 01/Mar/11 21:46
        Patch Available Patch Available Resolved Resolved
        2d 20h 20m 1 Scott Chen 04/Mar/11 18:07
        Resolved Resolved Closed Closed
        255d 6h 42m 1 Arun C Murthy 15/Nov/11 00:49

          People

          • Assignee:
            Scott Chen
            Reporter:
            Scott Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development