Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4337

Backport HDFS-4240 to branch-1: Make sure nodes are avoided to place replica if some replica are already under the same nodegroup.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0, 1-win
    • Component/s: namenode
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Backport HDFS-4240 to branch-1
      Show
      Backport HDFS-4240 to branch-1
    • Tags:
      Backport

      Description

      Update affects version from 1.0.0 to 1.2.0.

      1. HDFS-4337-v3.patch
        14 kB
        meng gong
      2. HDFS-4337-v2.patch
        14 kB
        meng gong

        Issue Links

          Activity

          Hide
          meng gong added a comment -

          Backport HDFS-4240

          Show
          meng gong added a comment - Backport HDFS-4240
          Hide
          Junping Du added a comment -

          Thanks Meng for doing this work. I will review your patch. Would you post the result of test patch?

          Show
          Junping Du added a comment - Thanks Meng for doing this work. I will review your patch. Would you post the result of test patch?
          Hide
          Junping Du added a comment -

          Thanks for the patch. There are some trivial issue need to be addressed here:
          1. Fix an indent issue in method of addToExcludedNodes() in BlockPlacementPolicyWithNodeGroup.java
          2. Fix indent issue in define dataNodesInBoundaryCase[] and dataNodesInMoreTargetsCase[] in TestReplicationPolicyWithNodeGroup.java to keep consistent.
          3. Replace unnecessary two blank lines below with one blank line

          +     
          +      File baseDir = new File(System.getProperty(
          +           "test.build.data", "build/test/data"), "dfs/");
          +      CONF.set(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY,
          +           new File(baseDir, "name").getPath());
          +            
          + 
          

          4. Indent and unnecessary double blank lines issue for checkTargetsOnDifferentNodeGroup() in TestReplicationPolicyWithNodeGroup.java
          5. Indent issue for following code:

          +    for(int i=0; i<NUM_OF_DATANODES_BOUNDARY; i++) {
          +        dataNodes[0].updateHeartbeat(
          +                2*MIN_BLOCKS_FOR_WRITE*BLOCK_SIZE, 0L,
          +                (MIN_BLOCKS_FOR_WRITE-1)*BLOCK_SIZE, 0);
          +        
          +      dataNodesInBoundaryCase[i].updateHeartbeat(
          +          2*MIN_BLOCKS_FOR_WRITE*BLOCK_SIZE, 0L,
          +          2*MIN_BLOCKS_FOR_WRITE*BLOCK_SIZE, 0);
          +    }
          
          

          6. Remove unnecessary blank lines in the last several lines of TestReplicationPolicyWithNodeGroup.java

          Show
          Junping Du added a comment - Thanks for the patch. There are some trivial issue need to be addressed here: 1. Fix an indent issue in method of addToExcludedNodes() in BlockPlacementPolicyWithNodeGroup.java 2. Fix indent issue in define dataNodesInBoundaryCase[] and dataNodesInMoreTargetsCase[] in TestReplicationPolicyWithNodeGroup.java to keep consistent. 3. Replace unnecessary two blank lines below with one blank line + + File baseDir = new File(System.getProperty( + "test.build.data", "build/test/data"), "dfs/"); + CONF.set(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY, + new File(baseDir, "name").getPath()); + + 4. Indent and unnecessary double blank lines issue for checkTargetsOnDifferentNodeGroup() in TestReplicationPolicyWithNodeGroup.java 5. Indent issue for following code: + for(int i=0; i<NUM_OF_DATANODES_BOUNDARY; i++) { + dataNodes[0].updateHeartbeat( + 2*MIN_BLOCKS_FOR_WRITE*BLOCK_SIZE, 0L, + (MIN_BLOCKS_FOR_WRITE-1)*BLOCK_SIZE, 0); + + dataNodesInBoundaryCase[i].updateHeartbeat( + 2*MIN_BLOCKS_FOR_WRITE*BLOCK_SIZE, 0L, + 2*MIN_BLOCKS_FOR_WRITE*BLOCK_SIZE, 0); + } 6. Remove unnecessary blank lines in the last several lines of TestReplicationPolicyWithNodeGroup.java
          Hide
          meng gong added a comment -

          Thanks For Junping, I have corrected these intents

          Show
          meng gong added a comment - Thanks For Junping, I have corrected these intents
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12562437/HDFS-4337-v2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3693//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562437/HDFS-4337-v2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3693//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks for updating in v2 patch. The patch looks good to me but a minor issue below:
          The method of checkTargetsOnDifferentNodeGroup() in TestReplicationPolicyWithNodeGroup(), you should use indent 2 instead of 3:

              if(targets.length == 0)
                 return true;
          

          Can you correct it and post the test-patch result on branch-1? Thanks!
          BTW, you don't have to remove previous patch in updating a new one. Also, for non-trunk patch, please get rid of submit patch. As current pre-commit test only for trunk branch.

          Show
          Junping Du added a comment - Thanks for updating in v2 patch. The patch looks good to me but a minor issue below: The method of checkTargetsOnDifferentNodeGroup() in TestReplicationPolicyWithNodeGroup(), you should use indent 2 instead of 3: if(targets.length == 0) return true; Can you correct it and post the test-patch result on branch-1? Thanks! BTW, you don't have to remove previous patch in updating a new one. Also, for non-trunk patch, please get rid of submit patch. As current pre-commit test only for trunk branch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12562444/HDFS-4337-v3.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3695//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562444/HDFS-4337-v3.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3695//console This message is automatically generated.
          Hide
          meng gong added a comment -

          Thanks for junping. Following is the result of test
          Ant test-commit build success

          -1 overall.  
          
              +1 @author.  The patch does not contain any @author tags.
          
              +1 tests included.  The patch appears to include 4 new or modified tests.
          
              +1 javadoc.  The javadoc tool did not generate any warning messages.
          
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          
              +1 findbugs.  The patch does not introduce any new Findbugs (version 2.0.1) warnings.
          
          
          Show
          meng gong added a comment - Thanks for junping. Following is the result of test Ant test-commit build success -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.1) warnings.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Meng, could you also run all the unit tests for your patch?

          Show
          Tsz Wo Nicholas Sze added a comment - Meng, could you also run all the unit tests for your patch?
          Hide
          meng gong added a comment -

          For Nicholas:
          I have runned all the unit tests and triggered two issue: MAPREDUCE-4906 and MAPREDUCE-4904
          These two issue can be reproduce without this patch HDFS-4337
          I will resolve them
          Thank you

          Show
          meng gong added a comment - For Nicholas: I have runned all the unit tests and triggered two issue: MAPREDUCE-4906 and MAPREDUCE-4904 These two issue can be reproduce without this patch HDFS-4337 I will resolve them Thank you
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Meng!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Meng!
          Hide
          gongsuochen added a comment -
          Show
          gongsuochen added a comment -
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

            People

            • Assignee:
              meng gong
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development