Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8073

Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We want to implement a placement policy( eg. HDFS-7891) based on default policy. It'll be easier if we could split BlockPlacementPolicyDefault.chooseTarget(..).

      1. HDFS-8073.001.patch
        5 kB
        Walter Su
      2. HDFS-8073.002.patch
        5 kB
        Walter Su
      3. HDFS-8073.003.patch
        5 kB
        Walter Su

        Issue Links

          Activity

          Hide
          walter.k.su Walter Su added a comment -

          There appear to be 0 release audit warnings after applying the patch.

          -1 overall.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          walter.k.su Walter Su added a comment - There appear to be 0 release audit warnings after applying the patch. -1 overall . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version ) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12723550/HDFS-8073.001.patch
          against trunk revision 3fb5abf.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723550/HDFS-8073.001.patch against trunk revision 3fb5abf. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//console This message is automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Changes looks good.
          Just needs to format it.
          +1 after that

          Show
          vinayrpet Vinayakumar B added a comment - Changes looks good. Just needs to format it. +1 after that
          Hide
          walter.k.su Walter Su added a comment -

          Thanks Vinayakumar B. 002 patch fixes the formats.

          Show
          walter.k.su Walter Su added a comment - Thanks Vinayakumar B . 002 patch fixes the formats.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12723598/HDFS-8073.002.patch
          against trunk revision 75c5454.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723598/HDFS-8073.002.patch against trunk revision 75c5454. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//console This message is automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -
          -      if ((numOfReplicas = requiredStorageTypes.size()) == 0) {
          +      if (requiredStorageTypes.size() == 0) {
          

          Why changing above?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - - if ((numOfReplicas = requiredStorageTypes.size()) == 0) { + if (requiredStorageTypes.size() == 0) { Why changing above?
          Hide
          walter.k.su Walter Su added a comment -

          Hi, Tsz Wo Nicholas Sze! I think let numOfReplicas=0 is useless. I removed it so the code looks clean.
          Sorry about that. I'll leave it alone. Now 003 patch only moves code.

          Show
          walter.k.su Walter Su added a comment - Hi, Tsz Wo Nicholas Sze ! I think let numOfReplicas=0 is useless. I removed it so the code looks clean. Sorry about that. I'll leave it alone. Now 003 patch only moves code.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          We does need numOfReplicas = requiredStorageTypes.size(), since it is possible that requiredStorageTypes.size() < (original) numOfReplicas.

          +1 the new patch looks good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - We does need numOfReplicas = requiredStorageTypes.size(), since it is possible that requiredStorageTypes.size() < (original) numOfReplicas. +1 the new patch looks good.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk and branch-2.
          Thanks Walter Su for the contribution.
          Thanks Tsz Wo Nicholas Sze for the review.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2. Thanks Walter Su for the contribution. Thanks Tsz Wo Nicholas Sze for the review.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7529 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7529/)
          HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7529 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7529/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12723808/HDFS-8073.003.patch
          against trunk revision 5449adc.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723808/HDFS-8073.003.patch against trunk revision 5449adc. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/)
          HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/)
          HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/)
          HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/)
          HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/)
          HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/)
          HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java

            People

            • Assignee:
              walter.k.su Walter Su
              Reporter:
              walter.k.su Walter Su
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development