Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4946

Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 2.7.2, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Allow preferLocalNode in BlockPlacementPolicyDefault to be disabled in configuration to prevent clients from writing the first replica of every block (i.e. the entire file) to the local DataNode.

      1. HDFS-4946-1.patch
        2 kB
        James Kinley
      2. HDFS-4946-2.patch
        3 kB
        Nathan Roberts

        Issue Links

          Activity

          Hide
          jrkinley James Kinley added a comment -

          Added config key: dfs.namenode.replication.prefer.local

          Show
          jrkinley James Kinley added a comment - Added config key: dfs.namenode.replication.prefer.local
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12590861/HDFS-4946-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4595//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4595//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12590861/HDFS-4946-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4595//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4595//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          Allow preferLocalNode in BlockPlacementPolicyDefault to be disabled in configuration to prevent a client from writing the first replica of every block (i.e. the entire file) to the local DataNode.

          The description reads to prevent specific clients, but the config toggle would shut off all clients from writing locally, which may not be desirous. Ideally we would like a client sent hint that influences the selection.

          Show
          qwertymaniac Harsh J added a comment - Allow preferLocalNode in BlockPlacementPolicyDefault to be disabled in configuration to prevent a client from writing the first replica of every block (i.e. the entire file) to the local DataNode. The description reads to prevent specific clients, but the config toggle would shut off all clients from writing locally, which may not be desirous. Ideally we would like a client sent hint that influences the selection.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12590861/HDFS-4946-1.patch
          against trunk revision 581176c.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7999//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12590861/HDFS-4946-1.patch against trunk revision 581176c. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7999//console This message is automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Uploaded a new version of this patch for trunk. We have found this config to be extremely useful across many large clusters. It avoids hot-spots for large files that can be quite problematic during localization and/or task scheduling.

          Hopefully folks will be agreeable to this simple config option.

          Show
          nroberts Nathan Roberts added a comment - Uploaded a new version of this patch for trunk. We have found this config to be extremely useful across many large clusters. It avoids hot-spots for large files that can be quite problematic during localization and/or task scheduling. Hopefully folks will be agreeable to this simple config option.
          Hide
          nroberts Nathan Roberts added a comment -

          James Kinley, re-opening because this is a very useful patch. Let me know if you disagree or would like me to assign it to myself to close out any remaining issues.

          Show
          nroberts Nathan Roberts added a comment - James Kinley , re-opening because this is a very useful patch. Let me know if you disagree or would like me to assign it to myself to close out any remaining issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 6s trunk passed
          +1 compile 0m 42s trunk passed with JDK v1.8.0_72
          +1 compile 0m 43s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 48s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.8.0_72
          +1 javac 0m 40s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.7.0_95
          +1 javac 0m 41s the patch passed
          -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new + 428 unchanged - 0 fixed = 431 total (was 428)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 16s the patch passed
          +1 javadoc 1m 8s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95
          -1 unit 59m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 54m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          140m 30s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.balancer.TestBalancer
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.tools.TestDFSAdminWithHA
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788466/HDFS-4946-2.patch
          JIRA Issue HDFS-4946
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 37648fee9d9e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2440671
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14526/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14526/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 6s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_72 +1 compile 0m 43s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 47s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_72 +1 javac 0m 40s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_95 +1 javac 0m 41s the patch passed -1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: patch generated 3 new + 428 unchanged - 0 fixed = 431 total (was 428) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 16s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95 -1 unit 59m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 54m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 140m 30s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.balancer.TestBalancer JDK v1.7.0_95 Failed junit tests hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788466/HDFS-4946-2.patch JIRA Issue HDFS-4946 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 37648fee9d9e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2440671 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14526/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14526/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14526/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          Assigned it to Nathan Roberts for now, the credit goes to the both gentlemen.
          +1 the patch looks good.

          Show
          kihwal Kihwal Lee added a comment - Assigned it to Nathan Roberts for now, the credit goes to the both gentlemen. +1 the patch looks good.
          Hide
          kihwal Kihwal Lee added a comment -

          Ideally we would like a client sent hint that influences the selection.

          Yes, I would like that too, but I believe that is beyond the scope of this jira. There are so many things that we can allow clients to drive in block placement.

          Show
          kihwal Kihwal Lee added a comment - Ideally we would like a client sent hint that influences the selection. Yes, I would like that too, but I believe that is beyond the scope of this jira. There are so many things that we can allow clients to drive in block placement.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9323/)
          HDFS-4946. Allow preferLocalNode in BlockPlacementPolicyDefault to be (kihwal: rev ba6d5ed955aa34cdcb7aaa4b1025da6ee5e91355)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9323 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9323/ ) HDFS-4946 . Allow preferLocalNode in BlockPlacementPolicyDefault to be (kihwal: rev ba6d5ed955aa34cdcb7aaa4b1025da6ee5e91355) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          kihwal Kihwal Lee added a comment -

          Committed it to trunk through branch-2.7. Thanks for working on this James Kinley and Nathan Roberts.

          Show
          kihwal Kihwal Lee added a comment - Committed it to trunk through branch-2.7. Thanks for working on this James Kinley and Nathan Roberts .
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              nroberts Nathan Roberts
              Reporter:
              jrkinley James Kinley
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development