Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2825

Add test hook to turn off the writer preferring its local DN

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.24.0, 0.23.1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Currently, the default block placement policy always places the first replica in the pipeline on the local node if there is a valid DN running there. In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN.
      [per discussion below, this was changed to not add a config, but only to add a hook for testing]

      1. hdfs-2825.txt
        4 kB
        Todd Lipcon
      2. hdfs-2825.txt
        4 kB
        Todd Lipcon

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Todd Lipcon added a comment -

          Yes, for create/append

          Show
          Todd Lipcon added a comment - Yes, for create/append
          Hide
          Arun C Murthy added a comment -

          Maybe we should open a new JIRA to do this as a client-side config, where the client would pass a new flag when asking the NN for a pipeline?

          It should just be an option for the create call?

          Show
          Arun C Murthy added a comment - Maybe we should open a new JIRA to do this as a client-side config, where the client would pass a new flag when asking the NN for a pipeline? It should just be an option for the create call?
          Hide
          Todd Lipcon added a comment -

          Maybe we should open a new JIRA to do this as a client-side config, where the client would pass a new flag when asking the NN for a pipeline?

          Show
          Todd Lipcon added a comment - Maybe we should open a new JIRA to do this as a client-side config, where the client would pass a new flag when asking the NN for a pipeline?
          Hide
          Milind Bhandarkar added a comment -

          Actually, there is a valid use case for turning off local datanode preferrence (not only for testing), especially with Yarn being used to deploy long running services. One such service is a kafka-like daemon which is listening to event producers, and loads batches into HDFS. This would fill up the local datanode.

          Show
          Milind Bhandarkar added a comment - Actually, there is a valid use case for turning off local datanode preferrence (not only for testing), especially with Yarn being used to deploy long running services. One such service is a kafka-like daemon which is listening to event producers, and loads batches into HDFS. This would fill up the local datanode.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #968 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/968/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #968 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/968/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #170 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/170/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #170 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/170/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #148 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/148/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #148 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/148/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #935 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/935/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #935 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/935/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Todd Lipcon added a comment -

          Allen - I agree it would be useful to set on the writer side. But that's not quite the original patch, which operates DFS-wide.

          Show
          Todd Lipcon added a comment - Allen - I agree it would be useful to set on the writer side. But that's not quite the original patch, which operates DFS-wide.
          Hide
          Allen Wittenauer added a comment -

          This would have actually been useful for ETL jobs. Oh well. Maybe I'll throw the original patch into our build anyway.

          Show
          Allen Wittenauer added a comment - This would have actually been useful for ETL jobs. Oh well. Maybe I'll throw the original patch into our build anyway.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1591 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1591/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1591 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1591/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #407 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/407/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #407 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/407/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #422 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/422/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #422 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/422/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1646 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1646/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1646 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1646/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1573 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1573/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1573 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1573/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #397 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/397/)
          HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #397 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/397/ ) HDFS-2825 . Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Fix Version/s 0.23.1 [ 12318885 ]
          Resolution Fixed [ 1 ]
          Hide
          Todd Lipcon added a comment -

          Committed to 23 and trunk. Thanks for reviews, all.

          Show
          Todd Lipcon added a comment - Committed to 23 and trunk. Thanks for reviews, all.
          Hide
          Arun C Murthy added a comment -

          +1 for the setter.

          Show
          Arun C Murthy added a comment - +1 for the setter.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12511547/hdfs-2825.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511547/hdfs-2825.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//console This message is automatically generated.
          Todd Lipcon made changes -
          Summary Add config option to turn off the writer preferring its local DN Add test hook to turn off the writer preferring its local DN
          Description Currently, the default block placement policy always places the first replica in the pipeline on the local node if there is a valid DN running there. In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN. Currently, the default block placement policy always places the first replica in the pipeline on the local node if there is a valid DN running there. In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN.
          _[per discussion below, this was changed to not add a config, but only to add a hook for testing]_
          Todd Lipcon made changes -
          Attachment hdfs-2825.txt [ 12511547 ]
          Hide
          Todd Lipcon added a comment -

          Attached patch gets rid of the config in favor of a setter method for use in tests.

          Show
          Todd Lipcon added a comment - Attached patch gets rid of the config in favor of a setter method for use in tests.
          Hide
          Eli Collins added a comment -

          If this is for testing only, then one could just do it by setting BlockPlacementPolicyDefault#preferLocal to false from the test without any new configuration.

          Seems reasonable. Can always add the config later if we want to enable it on a real cluster.

          Show
          Eli Collins added a comment - If this is for testing only, then one could just do it by setting BlockPlacementPolicyDefault#preferLocal to false from the test without any new configuration. Seems reasonable. Can always add the config later if we want to enable it on a real cluster.
          Hide
          Suresh Srinivas added a comment -

          It is not meant for general purpose use by users

          Description: In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN.

          Since the description says "It would also be useful", I assumed this is a generic capability that is being added.

          If this is for testing only, then one could just do it by setting BlockPlacementPolicyDefault#preferLocal to false from the test without any new configuration.

          Show
          Suresh Srinivas added a comment - It is not meant for general purpose use by users Description: In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN. Since the description says "It would also be useful", I assumed this is a generic capability that is being added. If this is for testing only, then one could just do it by setting BlockPlacementPolicyDefault#preferLocal to false from the test without any new configuration.
          Hide
          Eli Collins added a comment -

          Agree with Todd, writing a new block placement policy for this case doesn't make sense. This options is in the same spirit as eg considerLoad, something that you'd like to configure but we don't expect to become the default.

          Show
          Eli Collins added a comment - Agree with Todd, writing a new block placement policy for this case doesn't make sense. This options is in the same spirit as eg considerLoad, something that you'd like to configure but we don't expect to become the default.
          Todd Lipcon made changes -
          Link This issue blocks HDFS-2826 [ HDFS-2826 ]
          Hide
          Todd Lipcon added a comment -

          (do you have another way of writing the test in HDFS-2826? it's impossible to get replicas spread evenly around a minicluster without this patch, as far as I know)

          Show
          Todd Lipcon added a comment - (do you have another way of writing the test in HDFS-2826 ? it's impossible to get replicas spread evenly around a minicluster without this patch, as far as I know)
          Hide
          Todd Lipcon added a comment -

          This still reuses 99% of the code from the existing policy – it places the first replica rack-local and the other two on a separate (but same) rack. It still takes into account load as well as disk space on all the nodes.

          The primary purpose is for unit tests - see HDFS-2826 for example. It is not meant for general purpose use by users, though as noted in the description there is potentially a use case.

          Adding an entirely new block placement implementation for this purpose is overkill, since it will duplicate all of the code from the existing one.

          Show
          Todd Lipcon added a comment - This still reuses 99% of the code from the existing policy – it places the first replica rack-local and the other two on a separate (but same) rack. It still takes into account load as well as disk space on all the nodes. The primary purpose is for unit tests - see HDFS-2826 for example. It is not meant for general purpose use by users, though as noted in the description there is potentially a use case. Adding an entirely new block placement implementation for this purpose is overkill, since it will duplicate all of the code from the existing one.
          Hide
          Suresh Srinivas added a comment -

          Why are we adding another configuration to change the default behavior?

          If this is not required, using a different block placement policy, which is already pluggable is the right way to go.

          Show
          Suresh Srinivas added a comment - Why are we adding another configuration to change the default behavior? If this is not required, using a different block placement policy, which is already pluggable is the right way to go.
          Hide
          Arun C Murthy added a comment -

          I'm a little leery of adding more such arcane configs, wouldn't it be better to add a notion of a block-placement policy? Maybe HDFS-1094, maybe some other jira along these lines?

          Show
          Arun C Murthy added a comment - I'm a little leery of adding more such arcane configs, wouldn't it be better to add a notion of a block-placement policy? Maybe HDFS-1094 , maybe some other jira along these lines?
          Hide
          Eli Collins added a comment -

          +1 lgtm

          Show
          Eli Collins added a comment - +1 lgtm
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12511471/hdfs-2825.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511471/hdfs-2825.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//console This message is automatically generated.
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment hdfs-2825.txt [ 12511471 ]
          Hide
          Todd Lipcon added a comment -

          Simple patch - no direct tests included, but the test in HDFS-2826 uses this new feature, and will fail if the config doesn't work correctly. The old code path/behavior remains the same.

          Show
          Todd Lipcon added a comment - Simple patch - no direct tests included, but the test in HDFS-2826 uses this new feature, and will fail if the config doesn't work correctly. The old code path/behavior remains the same.
          Todd Lipcon created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development