Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8780

Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Current implementation:
      ======================

      DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be removed from dead/live node list only if below conditions are met
      I . If the Include list is not empty.
      II. If include and exclude list does not have decommissioned node and node state is decommissioned.

            if (!hostFileManager.hasIncludes()) {
                    return;
             }
      
            if ((!hostFileManager.isIncluded(node)) && (!hostFileManager.isExcluded(node))
                && node.isDecommissioned()) {
              // Include list is not empty, an existing datanode does not appear
              // in both include or exclude lists and it has been decommissioned.
              // Remove it from the node list.
              it.remove();
            }
      

      As mentioned in javadoc a datanode cannot be in "already decommissioned datanode state".
      Following the steps mentioned in javadoc datanode state is "dead" and not decommissioned.

      Can we avoid the unnecessary checks and have check for the node is in decommissioned state then remove from node list. ?
      Please provide your feedback.

      1. HDFS-8780.1.patch
        8 kB
        J.Andreina
      2. HDFS-8780.2.patch
        8 kB
        J.Andreina
      3. HDFS-8780.3.patch
        8 kB
        J.Andreina
      4. HDFS-8780-branch-2.7.patch
        7 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          ebadger Eric Badger added a comment -

          Thanks, Konstantin Shvachko! I just confirmed that this fixes the test failure on branch-2.7

          Show
          ebadger Eric Badger added a comment - Thanks, Konstantin Shvachko ! I just confirmed that this fixes the test failure on branch-2.7
          Hide
          shv Konstantin Shvachko added a comment -

          Yup, that was the porting problem. Fixed.

          Show
          shv Konstantin Shvachko added a comment - Yup, that was the porting problem. Fixed.
          Hide
          ebadger Eric Badger added a comment -

          Please investigate 2.7 unit test failure

          Show
          ebadger Eric Badger added a comment - Please investigate 2.7 unit test failure
          Hide
          ebadger Eric Badger added a comment -

          Konstantin Shvachko, the 2.7 patch that you committed here breaks TestHostsFiles.testHostsExcludeInUI. The failure is consistently reproducible and the associated stack trace is shown below.

          java.lang.AssertionError: Live nodes should contain the decommissioned node
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.apache.hadoop.hdfs.server.namenode.TestHostsFiles.testHostsExcludeInUI(TestHostsFiles.java:126)
          
          Show
          ebadger Eric Badger added a comment - Konstantin Shvachko , the 2.7 patch that you committed here breaks TestHostsFiles.testHostsExcludeInUI. The failure is consistently reproducible and the associated stack trace is shown below. java.lang.AssertionError: Live nodes should contain the decommissioned node at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hdfs.server.namenode.TestHostsFiles.testHostsExcludeInUI(TestHostsFiles.java:126)
          Hide
          shv Konstantin Shvachko added a comment -

          Committed to branch-2.7.

          Show
          shv Konstantin Shvachko added a comment - Committed to branch-2.7.
          Hide
          shv Konstantin Shvachko added a comment -

          Patch for branch-2.7

          Show
          shv Konstantin Shvachko added a comment - Patch for branch-2.7
          Hide
          shv Konstantin Shvachko added a comment -

          Would be good to port this to branch-2.7.

          Show
          shv Konstantin Shvachko added a comment - Would be good to port this to branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2341 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2341/)
          HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2341 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2341/ ) HDFS-8780 . Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #403 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/403/)
          HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #403 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/403/ ) HDFS-8780 . Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2368 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2368/)
          HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2368 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2368/ ) HDFS-8780 . Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1162 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1162/)
          HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1162 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1162/ ) HDFS-8780 . Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #430 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/430/)
          HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #430 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/430/ ) HDFS-8780 . Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #422 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/422/)
          HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #422 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/422/ ) HDFS-8780 . Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8498 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8498/)
          HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8498 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8498/ ) HDFS-8780 . Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) (vinayakumarb: rev 10ab7d595ece59f2d00b406ba8812c6295a4187f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk and branch-2.

          Thanks J.Andreina for the contribution.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2. Thanks J.Andreina for the contribution.
          Hide
          andreina J.Andreina added a comment -

          TestCase failures are unrelated.
          Please review.

          Show
          andreina J.Andreina added a comment - TestCase failures are unrelated. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 5s There were no new javac warning messages.
          +1 javadoc 10m 12s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 22s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 164m 28s Tests failed in hadoop-hdfs.
              210m 20s  



          Reason Tests
          Failed unit tests hadoop.tools.TestJMXGet
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755550/HDFS-8780.3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d845547
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12411/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12411/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12411/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 5s There were no new javac warning messages. +1 javadoc 10m 12s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 22s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 164m 28s Tests failed in hadoop-hdfs.     210m 20s   Reason Tests Failed unit tests hadoop.tools.TestJMXGet   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755550/HDFS-8780.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d845547 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12411/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12411/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12411/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Latest patch looks good to me.
          +1. Pending jenkins

          Show
          vinayrpet Vinayakumar B added a comment - Latest patch looks good to me. +1. Pending jenkins
          Hide
          andreina J.Andreina added a comment -

          Updated the patch fixing checkstyle comments.
          Testcase failures are unrelated
          Please review.

          Show
          andreina J.Andreina added a comment - Updated the patch fixing checkstyle comments. Testcase failures are unrelated Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 3s There were no new javac warning messages.
          +1 javadoc 10m 2s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 23s The applied patch generated 1 new checkstyle issues (total was 323, now 322).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 164m 8s Tests failed in hadoop-hdfs.
              209m 47s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
            hadoop.tools.TestJMXGet
            hadoop.hdfs.web.TestWebHDFSOAuth2



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755530/HDFS-8780.2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9538af0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12409/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12409/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12409/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12409/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 3s There were no new javac warning messages. +1 javadoc 10m 2s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 23s The applied patch generated 1 new checkstyle issues (total was 323, now 322). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 164m 8s Tests failed in hadoop-hdfs.     209m 47s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.tools.TestJMXGet   hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755530/HDFS-8780.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9538af0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12409/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12409/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12409/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12409/console This message was automatically generated.
          Hide
          andreina J.Andreina added a comment -

          Updated the patch fixing checkstyle comments.
          Testcase failures are unrelated.
          Please review.

          Show
          andreina J.Andreina added a comment - Updated the patch fixing checkstyle comments. Testcase failures are unrelated. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 37s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 9m 59s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 19s The applied patch generated 2 new checkstyle issues (total was 323, now 323).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 9s Pre-build of native portion
          -1 hdfs tests 114m 27s Tests failed in hadoop-hdfs.
              159m 13s  



          Reason Tests
          Timed out tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755383/HDFS-8780.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 486d5cb
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12396/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12396/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12396/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12396/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 19s The applied patch generated 2 new checkstyle issues (total was 323, now 323). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 9s Pre-build of native portion -1 hdfs tests 114m 27s Tests failed in hadoop-hdfs.     159m 13s   Reason Tests Timed out tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755383/HDFS-8780.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 486d5cb checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12396/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12396/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12396/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12396/console This message was automatically generated.
          Hide
          andreina J.Andreina added a comment -

          Updated an initial patch.
          Please review.

          Show
          andreina J.Andreina added a comment - Updated an initial patch. Please review.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Following are the possible configurations for include/exclude lists as mentioned in the javadoc.

          1. Host must have been in the include hosts list and the include hosts list must not be empty.
          2. Host is decommissioned by remaining in the include hosts list and added into the exclude hosts list. Name node is updated with the new information by issuing dfsadmin -refreshNodes command.
          3. Host is removed from both include hosts and exclude hosts lists. Name node is updated with the new informationby issuing dfsamin -refreshNodes command

          In this list one more important and default possible condition is not considered.
          Which is Include List is not configured ( i.e. *, allow everyone) and only exclude list is configured, where decommissioned node is present.

          In the condition checks, #2 is missed, where host is present in both lists.

          I think following changes will do

            private void removeDecomNodeFromList(final List<DatanodeDescriptor> nodeList) {
              // If the include list is empty, any nodes are welcomed and it does not
              // make sense to exclude any nodes from the cluster. Therefore, no remove.
              if (!hostFileManager.hasIncludes() && !hostFileManager.hasExcludes()) {
                return;
              }
          
              for (Iterator<DatanodeDescriptor> it = nodeList.iterator(); it.hasNext();) {
                DatanodeDescriptor node = it.next();
                if ((hostFileManager.isExcluded(node) || !hostFileManager
                    .isIncluded(node)) && node.isDecommissioned()) {
                  // Include list is not empty, an existing datanode does not appear
                  // in both include or exclude lists and it has been decommissioned.
                  // Remove it from the node list.
                  it.remove();
                }
              }
            }

          BUT, After the above fix, Since the node decommission done based these checks already, and if node is included again, node would not be decommissioned state.
          So I think its Okay to remove entire hostmanager checks and just remove decommissioned nodes directly from list

          Andrew Wang, Do you think this is fine?

          Show
          vinayrpet Vinayakumar B added a comment - Following are the possible configurations for include/exclude lists as mentioned in the javadoc. Host must have been in the include hosts list and the include hosts list must not be empty. Host is decommissioned by remaining in the include hosts list and added into the exclude hosts list. Name node is updated with the new information by issuing dfsadmin -refreshNodes command. Host is removed from both include hosts and exclude hosts lists. Name node is updated with the new informationby issuing dfsamin -refreshNodes command In this list one more important and default possible condition is not considered. Which is Include List is not configured ( i.e. *, allow everyone) and only exclude list is configured, where decommissioned node is present. In the condition checks, #2 is missed, where host is present in both lists. I think following changes will do private void removeDecomNodeFromList( final List<DatanodeDescriptor> nodeList) { // If the include list is empty, any nodes are welcomed and it does not // make sense to exclude any nodes from the cluster. Therefore, no remove. if (!hostFileManager.hasIncludes() && !hostFileManager.hasExcludes()) { return ; } for (Iterator<DatanodeDescriptor> it = nodeList.iterator(); it.hasNext();) { DatanodeDescriptor node = it.next(); if ((hostFileManager.isExcluded(node) || !hostFileManager .isIncluded(node)) && node.isDecommissioned()) { // Include list is not empty, an existing datanode does not appear // in both include or exclude lists and it has been decommissioned. // Remove it from the node list. it.remove(); } } } BUT, After the above fix, Since the node decommission done based these checks already, and if node is included again, node would not be decommissioned state. So I think its Okay to remove entire hostmanager checks and just remove decommissioned nodes directly from list Andrew Wang , Do you think this is fine?

            People

            • Assignee:
              andreina J.Andreina
              Reporter:
              andreina J.Andreina
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development