Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3258

Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node)

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 1.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For updating TestNetworkTopology to cover HADOOP-8144.

      1. HDFS-3258.patch
        1 kB
        Junping Du
      2. hdfs-3258.txt
        1 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #344 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/344/)
          svn merge -c 1330613 FIXES: HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1372956)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372956
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #344 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/344/ ) svn merge -c 1330613 FIXES: HDFS-3258 . Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1372956) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372956 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Hide
          Daryn Sharp added a comment -

          Merged to branch 23.

          Show
          Daryn Sharp added a comment - Merged to branch 23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1061 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/)
          HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1061 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/ ) HDFS-3258 . Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1026 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/)
          HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1026 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/ ) HDFS-3258 . Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2.

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2152 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2152/)
          HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2152 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2152/ ) HDFS-3258 . Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2135 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2135/)
          HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2135 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2135/ ) HDFS-3258 . Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2209 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2209/)
          HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2209 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2209/ ) HDFS-3258 . Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node). Contributed by Junping Du (Revision 1330613) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330613 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          Hide
          Eli Collins added a comment -

          +1 looks good, thanks Junping

          Show
          Eli Collins added a comment - +1 looks good, thanks Junping
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523613/HDFS-3258.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2311//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2311//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523613/HDFS-3258.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2311//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2311//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Update the test as Eli's suggestion.

          Show
          Junping Du added a comment - Update the test as Eli's suggestion.
          Hide
          Eli Collins added a comment -

          I've committed HADOOP-8144, when you update the patch you can click "submit patch".

          Show
          Eli Collins added a comment - I've committed HADOOP-8144 , when you update the patch you can click "submit patch".
          Hide
          Eli Collins added a comment -

          Ah, that explains it, so the order here in the test just needs to be updated to 1,5,3. I think we can leave the current behavior as is for this jira. Please also put a comment to this effect (that peudoSortByDistance does not take the "data center" layer into consideration and that it doesn't sort by getDistance) next to this test case so it's clear why 1,5,3 is a valid order.

          Show
          Eli Collins added a comment - Ah, that explains it, so the order here in the test just needs to be updated to 1,5,3. I think we can leave the current behavior as is for this jira. Please also put a comment to this effect (that peudoSortByDistance does not take the "data center" layer into consideration and that it doesn't sort by getDistance) next to this test case so it's clear why 1,5,3 is a valid order.
          Hide
          Junping Du added a comment -

          Hi Eli,
          pseudoSortByDistance() doesn't use getDistance() to calculate distance between nodes to sort in a precisely sequence but just a "pseudo" sort and figure out what's the first (node-local in general case) and the second (rack-local). The previous one to sort "5,3,1" just works well occasionally as it happen to swap 1 with 5 (5,3,1 -> 1,3,5) then keep order there. The test of "3,5,1" will also failed.
          Do we need a precisely sort here?

          Thanks,

          Junping

          Show
          Junping Du added a comment - Hi Eli, pseudoSortByDistance() doesn't use getDistance() to calculate distance between nodes to sort in a precisely sequence but just a "pseudo" sort and figure out what's the first (node-local in general case) and the second (rack-local). The previous one to sort "5,3,1" just works well occasionally as it happen to swap 1 with 5 (5,3,1 -> 1,3,5) then keep order there. The test of "3,5,1" will also failed. Do we need a precisely sort here? Thanks, Junping
          Hide
          Eli Collins added a comment -

          Hey Junping,
          That was my initial thought but testGetDistance checks that "5" should have distance 6 and "3" has distance 4 right?
          Also, the previous test stably sorts to order 1,3,5 as well. What am I missing?

          Show
          Eli Collins added a comment - Hey Junping, That was my initial thought but testGetDistance checks that "5" should have distance 6 and "3" has distance 4 right? Also, the previous test stably sorts to order 1,3,5 as well. What am I missing?
          Hide
          Junping Du added a comment -

          This bug is because pseudoSortByDistance is not taking the topology of "data center" layer into consideration in sorting the distance, so that "5" and "3" are treated equally because they are both off-rack. There are also some places in hadoop (like BlockPlacePolicyDefault.java, FifoScheduler.java, etc) considering rack/host only but not aware of datacenter layer. Will add dc layer awareness in pseudoSortByDistance soon if required.

          Thanks,

          Junping

          Show
          Junping Du added a comment - This bug is because pseudoSortByDistance is not taking the topology of "data center" layer into consideration in sorting the distance, so that "5" and "3" are treated equally because they are both off-rack. There are also some places in hadoop (like BlockPlacePolicyDefault.java, FifoScheduler.java, etc) considering rack/host only but not aware of datacenter layer. Will add dc layer awareness in pseudoSortByDistance soon if required. Thanks, Junping
          Hide
          Eli Collins added a comment -

          Attaching Junping's test from HADOOP-8144.

          Show
          Eli Collins added a comment - Attaching Junping's test from HADOOP-8144 .

            People

            • Assignee:
              Junping Du
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development