Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9902

Support different values of dfs.datanode.du.reserved per storage type

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.
      Show
      Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.

      Description

      Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, but they share one configuration dfs.datanode.du.reserved.
      The DISK size may be several TB and the RAM_DISK size may be only several tens of GB.
      The problem is that when I configure DISK and RAM_DISK (tmpfs) in the same DN, and I set dfs.datanode.du.reserved values 10GB, this will waste a lot of RAM_DISK size.
      Since the usage of RAM_DISK can be 100%, so I don't want dfs.datanode.du.reserved configured for DISK impacts the usage of tmpfs.
      So can we make a new configuration for RAM_DISK or just skip this configuration for RAM_DISK?

      1. HDFS-9902.patch
        2 kB
        Brahma Reddy Battula
      2. HDFS-9902-02.patch
        5 kB
        Brahma Reddy Battula
      3. HDFS-9902-03.patch
        6 kB
        Brahma Reddy Battula
      4. HDFS-9902-04.patch
        6 kB
        Brahma Reddy Battula
      5. HDFS-9902-05.patch
        6 kB
        Brahma Reddy Battula

        Activity

        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Pan Yuxuan thanks for reporting this issue..

        I feel, we can introduce a general config, which can be used for all storagetypes individually.

        Example:

        • Configuration can be like dfs.datanode.du.reserved.<storage-type> ,storage-type can be lowercase
          • Conf for RAM_DISK is dfs.datanode.du.reserved.ram_disk and
          • for SSD is dfs.datanode.du.reserved.ssd

        and consider the default value as value of dfs.datanode.du.reserved.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Pan Yuxuan thanks for reporting this issue.. I feel, we can introduce a general config, which can be used for all storagetypes individually. Example: Configuration can be like dfs.datanode.du.reserved.<storage-type> ,storage-type can be lowercase Conf for RAM_DISK is dfs.datanode.du.reserved.ram_disk and for SSD is dfs.datanode.du.reserved.ssd and consider the default value as value of dfs.datanode.du.reserved .
        Hide
        panyuxuan Pan Yuxuan added a comment -

        Brahma Reddy Battula thanks for your quick reply and fix this issue.
        I think the general config make sense for me.

        Show
        panyuxuan Pan Yuxuan added a comment - Brahma Reddy Battula thanks for your quick reply and fix this issue. I think the general config make sense for me.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Pan Yuxuan thanks for feedback..

        Uploaded the patch with testcase..Kindly review..

        Show
        brahmareddy Brahma Reddy Battula added a comment - Pan Yuxuan thanks for feedback.. Uploaded the patch with testcase..Kindly review..
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 11m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 34s trunk passed
        +1 compile 0m 38s trunk passed with JDK v1.8.0_77
        +1 compile 0m 40s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 57s trunk passed
        +1 javadoc 1m 4s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 42s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 36s the patch passed with JDK v1.8.0_77
        +1 javac 0m 36s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_95
        +1 javac 0m 39s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 7s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95
        -1 unit 54m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
        -1 unit 53m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        144m 1s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestRollingUpgrade
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAMetrics
          hadoop.hdfs.server.namenode.TestNameNodeResourceChecker



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796727/HDFS-9902-02.patch
        JIRA Issue HDFS-9902
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 47883e3846e1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1e6f929
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15054/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15054/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 34s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_77 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 42s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_77 +1 javac 0m 36s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95 -1 unit 54m 45s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 53m 23s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 144m 1s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.TestRollingUpgrade JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12796727/HDFS-9902-02.patch JIRA Issue HDFS-9902 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 47883e3846e1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1e6f929 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15054/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15054/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Arpit Agarwal if you've chance,,can you please review..?.. thanks.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal if you've chance,,can you please review..?.. thanks.
        Hide
        arpitagarwal Arpit Agarwal added a comment - - edited

        Hi Brahma Reddy Battula, thank you for reporting this. The fix lgtm.

        The unit test can be done more simply without MiniDFSCluster. Just instantiate "FsVolumeImpl" objects with different storage types and check #reserved is initialized correctly. Also could you please update the documentation of dfs.datanode.du.reserved?

        Show
        arpitagarwal Arpit Agarwal added a comment - - edited Hi Brahma Reddy Battula , thank you for reporting this. The fix lgtm. The unit test can be done more simply without MiniDFSCluster. Just instantiate "FsVolumeImpl" objects with different storage types and check #reserved is initialized correctly. Also could you please update the documentation of dfs.datanode.du.reserved ?
        Hide
        xyao Xiaoyu Yao added a comment -

        Agree with Arpit Agarwal, we need to document the new keys dfs.datanode.du.#storagetype#.reserved.

        Show
        xyao Xiaoyu Yao added a comment - Agree with Arpit Agarwal , we need to document the new keys dfs.datanode.du.#storagetype#.reserved .
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Arpit Agarwal and Xiaoyu Yao thanks a lot for looking into this issue...Uploaded the patch to address your comments..

        Show
        brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal and Xiaoyu Yao thanks a lot for looking into this issue...Uploaded the patch to address your comments..
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 20s trunk passed
        +1 compile 0m 38s trunk passed with JDK v1.8.0_91
        +1 compile 0m 41s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_91
        +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 44s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.8.0_91
        +1 javac 0m 37s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_95
        +1 javac 0m 39s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 8s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_91
        +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95
        -1 unit 56m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
        -1 unit 53m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        136m 5s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.balancer.TestBalancer
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.qjournal.client.TestQJMWithFaults
          hadoop.hdfs.TestRenameWhileOpen



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801697/HDFS-9902-03.patch
        JIRA Issue HDFS-9902
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 91d14a5adc40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 971af60
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15334/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15334/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 20s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_91 +1 compile 0m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 44s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_91 +1 javac 0m 37s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95 -1 unit 56m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 53m 27s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 136m 5s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.server.balancer.TestBalancer JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801697/HDFS-9902-03.patch JIRA Issue HDFS-9902 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 91d14a5adc40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 971af60 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15334/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15334/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment - - edited

        Thanks for addressing the feedback Brahma Reddy Battula. Looks like there is a typo in the documentation (should read If specific storage type reservation is not configured). Also suggest replacing considered with used:

        327	      'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is configured
        328	      then dfs.datanode.du.reserved will be considered.
        

        +1 with this addressed.

        Show
        arpitagarwal Arpit Agarwal added a comment - - edited Thanks for addressing the feedback Brahma Reddy Battula . Looks like there is a typo in the documentation (should read If specific storage type reservation is not configured ). Also suggest replacing considered with used : 327 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is configured 328 then dfs.datanode.du.reserved will be considered. +1 with this addressed.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded the patch to address the above comments..

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch to address the above comments..
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 pending Jenkins. Will hold off committing today to let Xiaoyu Yao take a look.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 pending Jenkins. Will hold off committing today to let Xiaoyu Yao take a look.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 46s trunk passed
        +1 compile 1m 1s trunk passed with JDK v1.8.0_92
        +1 compile 0m 46s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 2m 5s trunk passed
        +1 javadoc 1m 17s trunk passed with JDK v1.8.0_92
        +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 56s the patch passed
        +1 compile 0m 55s the patch passed with JDK v1.8.0_92
        +1 javac 0m 55s the patch passed
        +1 compile 0m 46s the patch passed with JDK v1.7.0_95
        +1 javac 0m 46s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 0m 58s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 18s the patch passed
        +1 javadoc 1m 18s the patch passed with JDK v1.8.0_92
        +1 javadoc 2m 0s the patch passed with JDK v1.7.0_95
        -1 unit 115m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_92.
        -1 unit 107m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 32s Patch does not generate ASF License warnings.
        252m 45s



        Reason Tests
        JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.8.0_92 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
          hadoop.hdfs.server.namenode.TestEditLog
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801773/HDFS-9902-04.patch
        JIRA Issue HDFS-9902
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 9454124b22c1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2beedea
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15337/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15337/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 46s trunk passed +1 compile 1m 1s trunk passed with JDK v1.8.0_92 +1 compile 0m 46s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 5s trunk passed +1 javadoc 1m 17s trunk passed with JDK v1.8.0_92 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 56s the patch passed +1 compile 0m 55s the patch passed with JDK v1.8.0_92 +1 javac 0m 55s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_95 +1 javac 0m 46s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 18s the patch passed +1 javadoc 1m 18s the patch passed with JDK v1.8.0_92 +1 javadoc 2m 0s the patch passed with JDK v1.7.0_95 -1 unit 115m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_92. -1 unit 107m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 32s Patch does not generate ASF License warnings. 252m 45s Reason Tests JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.8.0_92 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801773/HDFS-9902-04.patch JIRA Issue HDFS-9902 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 9454124b22c1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2beedea Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15337/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15337/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15337/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Looks good to me. Some NITS and suggestion:

        Specific Storage type based reservation also supported. The property should be followed with 
        corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for HDFS storage policies.
        

        ==>

        Specific storage type based reservation is also supported. The property can be followed with
        corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.
        
        Show
        xyao Xiaoyu Yao added a comment - Looks good to me. Some NITS and suggestion: Specific Storage type based reservation also supported. The property should be followed with corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for HDFS storage policies. ==> Specific storage type based reservation is also supported. The property can be followed with corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploading the patch to address the above comments..

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploading the patch to address the above comments..
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 53s trunk passed
        +1 compile 0m 57s trunk passed with JDK v1.8.0_92
        +1 compile 0m 48s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 7s trunk passed
        +1 javadoc 1m 18s trunk passed with JDK v1.8.0_92
        +1 javadoc 2m 2s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 56s the patch passed
        +1 compile 0m 55s the patch passed with JDK v1.8.0_92
        +1 javac 0m 55s the patch passed
        +1 compile 0m 46s the patch passed with JDK v1.7.0_95
        +1 javac 0m 46s the patch passed
        +1 checkstyle 0m 20s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 22s the patch passed
        +1 javadoc 1m 14s the patch passed with JDK v1.8.0_92
        +1 javadoc 2m 6s the patch passed with JDK v1.7.0_95
        -1 unit 89m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_92.
        -1 unit 82m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        201m 43s



        Reason Tests
        JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.TestAsyncDFSRename
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.TestAsyncDFSRename



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801864/HDFS-9902-05.patch
        JIRA Issue HDFS-9902
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 23fe3f7babb4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9e8411d
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15342/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15342/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 53s trunk passed +1 compile 0m 57s trunk passed with JDK v1.8.0_92 +1 compile 0m 48s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 18s trunk passed with JDK v1.8.0_92 +1 javadoc 2m 2s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 56s the patch passed +1 compile 0m 55s the patch passed with JDK v1.8.0_92 +1 javac 0m 55s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_95 +1 javac 0m 46s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 22s the patch passed +1 javadoc 1m 14s the patch passed with JDK v1.8.0_92 +1 javadoc 2m 6s the patch passed with JDK v1.7.0_95 -1 unit 89m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_92. -1 unit 82m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 201m 43s Reason Tests JDK v1.8.0_92 Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestAsyncDFSRename JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestAsyncDFSRename Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801864/HDFS-9902-05.patch JIRA Issue HDFS-9902 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 23fe3f7babb4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9e8411d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15342/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15342/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Brahma Reddy Battula for updating the patch. +1 for patch v05.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Brahma Reddy Battula for updating the patch. +1 for patch v05.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 from me too for the v5 patch.

        I committed it for 2.8.0. Thanks for the contribution Brahma Reddy Battula and thanks for the reviews Xiaoyu Yao and Pan Yuxuan.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 from me too for the v5 patch. I committed it for 2.8.0. Thanks for the contribution Brahma Reddy Battula and thanks for the reviews Xiaoyu Yao and Pan Yuxuan .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9709 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9709/)
        HDFS-9902. Support different values of dfs.datanode.du.reserved per (arp: rev 6d77d6eab7790ed7ae2cad5b327ba5d1deb485db)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9709 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9709/ ) HDFS-9902 . Support different values of dfs.datanode.du.reserved per (arp: rev 6d77d6eab7790ed7ae2cad5b327ba5d1deb485db) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java

          People

          • Assignee:
            brahmareddy Brahma Reddy Battula
            Reporter:
            panyuxuan Pan Yuxuan
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development