Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9279

Decomissioned capacity should not be considered for configured/used capacity

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Capacity of a decommissioned node is being accounted as configured and used capacity metrics. This gives incorrect perception of cluster usage.
      Once a node is decommissioned, its capacity should be considered similar to a dead node.

      1. HDFS-9279-v1.patch
        5 kB
        Kuhu Shukla
      2. HDFS-9279-v2.patch
        5 kB
        Kuhu Shukla
      3. HDFS-9279-v3.patch
        7 kB
        Kuhu Shukla
      4. HDFS-9279-v4.patch
        6 kB
        Kuhu Shukla

        Activity

        Hide
        kshukla Kuhu Shukla added a comment -

        The patch in addition to dfsUsed, also updates XceiverCount , and blockPoolUsed only when a node is not decommissioning or decommissioned. cacheCapacity and cacheUsed are updated for all nodes that are not decommissioned.

        Show
        kshukla Kuhu Shukla added a comment - The patch in addition to dfsUsed, also updates XceiverCount , and blockPoolUsed only when a node is not decommissioning or decommissioned. cacheCapacity and cacheUsed are updated for all nodes that are not decommissioned.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768373/HDFS-9279-v1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 15eb84b
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13169/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768373/HDFS-9279-v1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 15eb84b Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13169/console This message was automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        Correcting patch apply error.

        Show
        kshukla Kuhu Shukla added a comment - Correcting patch apply error.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 16m 29s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 4s There were no new javac warning messages.
        +1 javadoc 10m 35s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 31s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 40s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 15s Pre-build of native portion
        -1 hdfs tests 64m 26s Tests failed in hadoop-hdfs.
            108m 34s  



        Reason Tests
        FindBugs module:hadoop-hdfs
        Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
          hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.server.namenode.TestNameNodeMXBean
          hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          hadoop.hdfs.TestDecommission
          hadoop.hdfs.server.namenode.TestCacheDirectives
          hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768433/HDFS-9279-v2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5679e46
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13176/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13176/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13176/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13176/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 29s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 35s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 31s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 40s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 2m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 64m 26s Tests failed in hadoop-hdfs.     108m 34s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768433/HDFS-9279-v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5679e46 Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13176/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13176/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13176/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13176/console This message was automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        Fixed the code and tests pertinent to the code change. The test failures caused by this patch are TestDecommission, TestCacheDirectives, TestNameNodeCapacityReport and TestNameNodeMXBean.

        TestDataNodeHotSwapVolumes is showing intermittent failures , with and without the patch.

        Show
        kshukla Kuhu Shukla added a comment - Fixed the code and tests pertinent to the code change. The test failures caused by this patch are TestDecommission, TestCacheDirectives, TestNameNodeCapacityReport and TestNameNodeMXBean. TestDataNodeHotSwapVolumes is showing intermittent failures , with and without the patch.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 23m 4s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 10m 23s There were no new javac warning messages.
        +1 javadoc 14m 1s There were no new javadoc warning messages.
        +1 release audit 0m 38s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 53s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 53s mvn install still works.
        +1 eclipse:eclipse 0m 43s The patch built with eclipse:eclipse.
        +1 findbugs 2m 59s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 4m 15s Pre-build of native portion
        -1 hdfs tests 64m 57s Tests failed in hadoop-hdfs.
            124m 50s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestDFSOutputStream
          hadoop.hdfs.TestWriteReadStripedFile
          hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          hadoop.fs.TestSymlinkHdfsFileContext



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768762/HDFS-9279-v3.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 67e3d75
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13196/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13196/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13196/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13196/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 23m 4s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 10m 23s There were no new javac warning messages. +1 javadoc 14m 1s There were no new javadoc warning messages. +1 release audit 0m 38s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 53s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 53s mvn install still works. +1 eclipse:eclipse 0m 43s The patch built with eclipse:eclipse. +1 findbugs 2m 59s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 4m 15s Pre-build of native portion -1 hdfs tests 64m 57s Tests failed in hadoop-hdfs.     124m 50s   Reason Tests Failed unit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSOutputStream   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.fs.TestSymlinkHdfsFileContext Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768762/HDFS-9279-v3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 67e3d75 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13196/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13196/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13196/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13196/console This message was automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        Updated patch that keeps the XceiverCount update as-is. Changed the Unit test accordingly. Only TestNamenodeCapacityReport test failure was related to my patch, rest of the test failures are not seen locally.

        Show
        kshukla Kuhu Shukla added a comment - Updated patch that keeps the XceiverCount update as-is. Changed the Unit test accordingly. Only TestNamenodeCapacityReport test failure was related to my patch, rest of the test failures are not seen locally.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 24m 18s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 10m 49s There were no new javac warning messages.
        +1 javadoc 13m 43s There were no new javadoc warning messages.
        +1 release audit 0m 32s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 54s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 59s mvn install still works.
        +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse.
        +1 findbugs 3m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 4m 13s Pre-build of native portion
        -1 hdfs tests 82m 24s Tests failed in hadoop-hdfs.
            144m 0s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate
          hadoop.hdfs.web.TestWebHdfsUrl
          hadoop.hdfs.web.TestWebHdfsTokens
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12769047/HDFS-9279-v4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5c24fe7
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13230/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13230/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13230/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13230/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 24m 18s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 10m 49s There were no new javac warning messages. +1 javadoc 13m 43s There were no new javadoc warning messages. +1 release audit 0m 32s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 54s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 59s mvn install still works. +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse. +1 findbugs 3m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 4m 13s Pre-build of native portion -1 hdfs tests 82m 24s Tests failed in hadoop-hdfs.     144m 0s   Reason Tests Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.web.TestWebHdfsUrl   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12769047/HDFS-9279-v4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5c24fe7 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13230/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13230/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13230/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13230/console This message was automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        The latest test failures are unrelated. From a previous failure, TestDataNodeHotSwapVolumes is failing intermittently on trunk(HDFS-9310). Requesting Kihwal Lee to review and share comments. Thanks.

        Show
        kshukla Kuhu Shukla added a comment - The latest test failures are unrelated. From a previous failure, TestDataNodeHotSwapVolumes is failing intermittently on trunk( HDFS-9310 ). Requesting Kihwal Lee to review and share comments. Thanks.
        Hide
        kihwal Kihwal Lee added a comment -

        The jira was originally to exclude "live" decommissioned nodes from stat, but it seems decommissioning nodes are excluded as well.

        Show
        kihwal Kihwal Lee added a comment - The jira was originally to exclude "live" decommissioned nodes from stat, but it seems decommissioning nodes are excluded as well.
        Hide
        kihwal Kihwal Lee added a comment -

        Actually, I think this makes more sense. Assuming roughly even distribution of data, excluding everything from dicomm'ing nodes makes the used/total unchanged initially. As blocks are replicated used/total will increase. I think this tracks the reality better.

        Show
        kihwal Kihwal Lee added a comment - Actually, I think this makes more sense. Assuming roughly even distribution of data, excluding everything from dicomm'ing nodes makes the used/total unchanged initially. As blocks are replicated used/total will increase. I think this tracks the reality better.
        Hide
        kihwal Kihwal Lee added a comment -

        +1 the patch looks good.

        Show
        kihwal Kihwal Lee added a comment - +1 the patch looks good.
        Hide
        kihwal Kihwal Lee added a comment -

        I've committed this to trunk and branch-2. Thank you for working on the issue, Kuhu.

        Show
        kihwal Kihwal Lee added a comment - I've committed this to trunk and branch-2. Thank you for working on the issue, Kuhu.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8720 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8720/)
        HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8720 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8720/ ) HDFS-9279 . Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #596 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/596/)
        HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #596 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/596/ ) HDFS-9279 . Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #609 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/609/)
        HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #609 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/609/ ) HDFS-9279 . Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1332 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1332/)
        HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1332 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1332/ ) HDFS-9279 . Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2539 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2539/)
        HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2539 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2539/ ) HDFS-9279 . Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2485 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2485/)
        HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2485 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2485/ ) HDFS-9279 . Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #547 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/547/)
        HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #547 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/547/ ) HDFS-9279 . Decomissioned capacity should not be considered for (kihwal: rev 19a77f546657b086af8f41fa631099bdde7e010c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        rjainqb Rajat Jain added a comment -

        While it makes sense to not include decommissioning nodes in configured capacity, but they should still be used for calculating used capacity. Because the data present in the decommissioning nodes would eventually be transferred over to the live nodes. Is this understanding correct?

        Show
        rjainqb Rajat Jain added a comment - While it makes sense to not include decommissioning nodes in configured capacity, but they should still be used for calculating used capacity. Because the data present in the decommissioning nodes would eventually be transferred over to the live nodes. Is this understanding correct?
        Hide
        kihwal Kihwal Lee added a comment -

        Because the data present in the decommissioning nodes would eventually be transferred over to the live nodes. Is this understanding correct?

        The replicas are not invalidated on decommissioning nodes even after replicating, so the capacity tracking was not accurate either. It ended up double counting the used space toward the end, at which the process seems to stall more frequently nowadays (this is another topic). If a significant portion of a cluster is decommissioned, the stat will look very strange and confuse people. That actually happened to us multiple times. The free/total ratio will look considerably smaller than the actual value. Monitoring tools cannot easily dismiss it as 'Nah.. it's a temporary discrepancy caused by decommissioning.'

        With this change, the storage capacity stat has become more like regular under-replication scenario caused by node/disk outages. Additional space will be used for re-replicating those blocks, but it is not yet allocated to those blocks. That's the actual state of used/usable storage and the stat reflects that now. If we want the stat to reflect what would be used in the future, we are talking space reservation feature.

        Show
        kihwal Kihwal Lee added a comment - Because the data present in the decommissioning nodes would eventually be transferred over to the live nodes. Is this understanding correct? The replicas are not invalidated on decommissioning nodes even after replicating, so the capacity tracking was not accurate either. It ended up double counting the used space toward the end, at which the process seems to stall more frequently nowadays (this is another topic). If a significant portion of a cluster is decommissioned, the stat will look very strange and confuse people. That actually happened to us multiple times. The free/total ratio will look considerably smaller than the actual value. Monitoring tools cannot easily dismiss it as 'Nah.. it's a temporary discrepancy caused by decommissioning.' With this change, the storage capacity stat has become more like regular under-replication scenario caused by node/disk outages. Additional space will be used for re-replicating those blocks, but it is not yet allocated to those blocks. That's the actual state of used/usable storage and the stat reflects that now. If we want the stat to reflect what would be used in the future, we are talking space reservation feature.

          People

          • Assignee:
            kshukla Kuhu Shukla
            Reporter:
            kshukla Kuhu Shukla
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development