Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8073

Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided.

        Details

        • Type: Improvement
        • Status: Resolved
        • Priority: Trivial
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 2.8.0, 3.0.0-alpha1
        • Component/s: namenode
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          We want to implement a placement policy( eg. HDFS-7891) based on default policy. It'll be easier if we could split BlockPlacementPolicyDefault.chooseTarget(..).

          1. Text File
            HDFS-8073.001.patch
            5 kB
            Walter Su
          2. Text File
            HDFS-8073.002.patch
            5 kB
            Walter Su
          3. Text File
            HDFS-8073.003.patch
            5 kB
            Walter Su

            Issue Links

            is required by

            New Feature - A new feature of the product, which has yet to be developed. HDFS-7891 A block placement policy with best rack failure tolerance

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              walter.k.su Walter Su added a comment -

              There appear to be 0 release audit warnings after applying the patch.

              -1 overall.

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. There were no new javadoc warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              Show
              walter.k.su Walter Su added a comment - There appear to be 0 release audit warnings after applying the patch. -1 overall . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version ) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12723550/HDFS-8073.001.patch
              against trunk revision 3fb5abf.

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. There were no new javadoc warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

              Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723550/HDFS-8073.001.patch against trunk revision 3fb5abf. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10189//console This message is automatically generated.
              Hide
              Permalink
              vinayrpet Vinayakumar B added a comment -

              Changes looks good.
              Just needs to format it.
              +1 after that

              Show
              vinayrpet Vinayakumar B added a comment - Changes looks good. Just needs to format it. +1 after that
              Hide
              Permalink
              walter.k.su Walter Su added a comment -

              Thanks Vinayakumar B. 002 patch fixes the formats.

              Show
              walter.k.su Walter Su added a comment - Thanks Vinayakumar B . 002 patch fixes the formats.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12723598/HDFS-8073.002.patch
              against trunk revision 75c5454.

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. There were no new javadoc warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

              Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723598/HDFS-8073.002.patch against trunk revision 75c5454. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10191//console This message is automatically generated.
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment - 
              -      if ((numOfReplicas = requiredStorageTypes.size()) == 0) {
+      if (requiredStorageTypes.size() == 0) {

              Why changing above?

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - - if ((numOfReplicas = requiredStorageTypes.size()) == 0) { + if (requiredStorageTypes.size() == 0) { Why changing above?
              Hide
              Permalink
              walter.k.su Walter Su added a comment -

              Hi, Tsz Wo Nicholas Sze! I think let numOfReplicas=0 is useless. I removed it so the code looks clean.
              Sorry about that. I'll leave it alone. Now 003 patch only moves code.

              Show
              walter.k.su Walter Su added a comment - Hi, Tsz Wo Nicholas Sze ! I think let numOfReplicas=0 is useless. I removed it so the code looks clean. Sorry about that. I'll leave it alone. Now 003 patch only moves code.
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              We does need numOfReplicas = requiredStorageTypes.size(), since it is possible that requiredStorageTypes.size() < (original) numOfReplicas.

              +1 the new patch looks good.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - We does need numOfReplicas = requiredStorageTypes.size(), since it is possible that requiredStorageTypes.size() < (original) numOfReplicas. +1 the new patch looks good.
              Hide
              Permalink
              vinayrpet Vinayakumar B added a comment -

              Committed to trunk and branch-2.
              Thanks Walter Su for the contribution.
              Thanks Tsz Wo Nicholas Sze for the review.

              Show
              vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2. Thanks Walter Su for the contribution. Thanks Tsz Wo Nicholas Sze for the review.
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-trunk-Commit #7529 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7529/)
              HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7529 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7529/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12723808/HDFS-8073.003.patch
              against trunk revision 5449adc.

              +1 @author. The patch does not contain any @author tags.

              -1 tests included. The patch doesn't appear to include any new or modified tests.
              Please justify why no new tests are needed for this patch.
              Also please list what manual steps were performed to verify this patch.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. There were no new javadoc warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

              Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//testReport/
              Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723808/HDFS-8073.003.patch against trunk revision 5449adc. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10203//console This message is automatically generated.
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/)
              HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/157/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/)
              HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2089/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/)
              HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/148/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/)
              HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #891 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/891/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/)
              HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/158/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/)
              HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc)

              • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
              • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2107 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2107/ ) HDFS-8073 . Split BlockPlacementPolicyDefault.chooseTarget(..) so it can be easily overrided. (Contributed by Walter Su) (vinayakumarb: rev d505c8acd30d6f40d0632fe9c93c886a4499a9fc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java

                People

                • Assignee:
                  walter.k.su Walter Su
                  Reporter:
                  walter.k.su Walter Su
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development