Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9229

Expose size of NameNode directory as a metric

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Useful for admins in reserving / managing NN local file system space. Also useful when transferring NN backups.

      1. HDFS-9229.001.patch
        7 kB
        Surendra Singh Lilhore
      2. HDFS-9229.002.patch
        7 kB
        Surendra Singh Lilhore
      3. HDFS-9229.003.patch
        7 kB
        Surendra Singh Lilhore
      4. HDFS-9229.004.patch
        12 kB
        Surendra Singh Lilhore
      5. HDFS-9229.005.patch
        13 kB
        Surendra Singh Lilhore
      6. HDFS-9229.006.patch
        12 kB
        Surendra Singh Lilhore

        Activity

        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Please feel free to reassign if you started working on this

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Please feel free to reassign if you started working on this
        Hide
        zhz Zhe Zhang added a comment -

        Surendra Singh Lilhore Thanks for the interest in this issue! I haven't started any work.

        Show
        zhz Zhe Zhang added a comment - Surendra Singh Lilhore Thanks for the interest in this issue! I haven't started any work.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Attached Patch.
        Please review...

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Attached Patch. Please review...
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 27m 8s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 44s There were no new javac warning messages.
        +1 javadoc 11m 3s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 17s Site still builds.
        -1 checkstyle 2m 51s The applied patch generated 4 new checkstyle issues (total was 421, now 424).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 4m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 8m 28s Tests passed in hadoop-common.
        -1 hdfs tests 65m 31s Tests failed in hadoop-hdfs.
            134m 26s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestRecoverStripedFile
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
        Timed out tests org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12767211/HDFS-9229.001.patch
        Optional Tests site javadoc javac unit findbugs checkstyle
        git revision trunk / 58590fe
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13038/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13038/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 27m 8s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 44s There were no new javac warning messages. +1 javadoc 11m 3s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 site 3m 17s Site still builds. -1 checkstyle 2m 51s The applied patch generated 4 new checkstyle issues (total was 421, now 424). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 4m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 28s Tests passed in hadoop-common. -1 hdfs tests 65m 31s Tests failed in hadoop-hdfs.     134m 26s   Reason Tests Failed unit tests hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.namenode.TestRecoverStripedBlocks Timed out tests org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767211/HDFS-9229.001.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 58590fe Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13038/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13038/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13038/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Thank you Surendra Singh Lilhore for the contribution, patch looks almost good. Just few comments, please take a look at it.

        1. Unit is missing, please add it. (for example, in bytes)
          +| `NameDirSize` | NameNode name directories size|
          
        2. In tests, it would be good to assert the the size of the nnDirMap.size(). Presently it will miss the assertion and won't fail the test if the map wrongly returns the content with zero length, right?
          +      Map<String, Long> nnDirMap =
          +          (Map<String, Long>) JSON.parse(
          +              (String) mbs.getAttribute(mxbeanName, "NameDirSize"));
          
        3. Also, there are couple of checkstyle warnings related to the patch, please take care. Thanks!
        Show
        rakeshr Rakesh R added a comment - Thank you Surendra Singh Lilhore for the contribution, patch looks almost good. Just few comments, please take a look at it. Unit is missing, please add it. (for example, in bytes) +| `NameDirSize` | NameNode name directories size| In tests, it would be good to assert the the size of the nnDirMap.size() . Presently it will miss the assertion and won't fail the test if the map wrongly returns the content with zero length, right? + Map< String , Long > nnDirMap = + (Map< String , Long >) JSON.parse( + ( String ) mbs.getAttribute(mxbeanName, "NameDirSize" )); Also, there are couple of checkstyle warnings related to the patch, please take care. Thanks!
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Rakesh R for review.
        Attached updated patch. Please review...

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Rakesh R for review. Attached updated patch. Please review...
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 26m 6s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 9m 26s There were no new javac warning messages.
        +1 javadoc 11m 40s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 26s Site still builds.
        -1 checkstyle 2m 55s The applied patch generated 1 new checkstyle issues (total was 421, now 421).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 41s mvn install still works.
        +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
        +1 findbugs 5m 7s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 8m 39s Tests passed in hadoop-common.
        -1 hdfs tests 65m 48s Tests failed in hadoop-hdfs.
            135m 53s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestRecoverStripedFile
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.util.TestByteArrayManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12767257/HDFS-9229.002.patch
        Optional Tests site javadoc javac unit findbugs checkstyle
        git revision trunk / 0ab3f9d
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13042/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13042/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 26m 6s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 26s There were no new javac warning messages. +1 javadoc 11m 40s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 site 3m 26s Site still builds. -1 checkstyle 2m 55s The applied patch generated 1 new checkstyle issues (total was 421, now 421). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. +1 findbugs 5m 7s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 39s Tests passed in hadoop-common. -1 hdfs tests 65m 48s Tests failed in hadoop-hdfs.     135m 53s   Reason Tests Failed unit tests hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.util.TestByteArrayManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767257/HDFS-9229.002.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 0ab3f9d Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13042/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13042/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13042/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for the work Surendra! A few quick comments:

        +  public String getNNDirectorySize() {
        +    Map<String, Long> storageTypeMap = new HashMap<String, Long>();
        

        With Java 7 we don't need to specify type here.

        +    public long getDirecorySize() {
        +      if (!isShared && root != null) {
        +        return org.apache.commons.io.FileUtils.sizeOfDirectory(root);
        

        Maybe we should import org.apache.commons.io.FileUtils?

        Another suggestion is to enhance the unit test to actually verify the size matches, instead of just verifying it's positive.

        Show
        zhz Zhe Zhang added a comment - Thanks for the work Surendra! A few quick comments: + public String getNNDirectorySize() { + Map< String , Long > storageTypeMap = new HashMap< String , Long >(); With Java 7 we don't need to specify type here. + public long getDirecorySize() { + if (!isShared && root != null ) { + return org.apache.commons.io.FileUtils.sizeOfDirectory(root); Maybe we should import org.apache.commons.io.FileUtils ? Another suggestion is to enhance the unit test to actually verify the size matches, instead of just verifying it's positive.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Zhe Zhang for review...
        Attached updated patch, please review..

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Zhe Zhang for review... Attached updated patch, please review..
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 26m 43s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 43s There were no new javac warning messages.
        +1 javadoc 11m 1s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 14s Site still builds.
        -1 checkstyle 2m 56s The applied patch generated 1 new checkstyle issues (total was 421, now 421).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 45s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 4m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 8m 33s Tests passed in hadoop-common.
        -1 hdfs tests 63m 17s Tests failed in hadoop-hdfs.
            132m 8s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.server.blockmanagement.TestNodeCount



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12767769/HDFS-9229.003.patch
        Optional Tests site javadoc javac unit findbugs checkstyle
        git revision trunk / 0c4af0f
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13105/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13105/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 26m 43s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 43s There were no new javac warning messages. +1 javadoc 11m 1s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 site 3m 14s Site still builds. -1 checkstyle 2m 56s The applied patch generated 1 new checkstyle issues (total was 421, now 421). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 45s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 4m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 8m 33s Tests passed in hadoop-common. -1 hdfs tests 63m 17s Tests failed in hadoop-hdfs.     132m 8s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.blockmanagement.TestNodeCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767769/HDFS-9229.003.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / 0c4af0f Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13105/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13105/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13105/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Surendra Singh Lilhore, latest patch looks good to me.

        Show
        rakeshr Rakesh R added a comment - Thanks Surendra Singh Lilhore , latest patch looks good to me.
        Hide
        wheat9 Haohui Mai added a comment -

        The metric needs to live in a different place instead of NameNodeMXBean. All other metrics in the interface do not involve I/O and are polled frequently.

        This metric requires an unbound I/O time which can break many monitoring applications.

        Show
        wheat9 Haohui Mai added a comment - The metric needs to live in a different place instead of NameNodeMXBean . All other metrics in the interface do not involve I/O and are polled frequently. This metric requires an unbound I/O time which can break many monitoring applications.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Haohui Mai for suggestion..
        Can I move this metric in NameNodeStatusMXBean ??

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Haohui Mai for suggestion.. Can I move this metric in NameNodeStatusMXBean ??
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Haohui for the suggestion.

        Surendra Singh Lilhore I agree NameNodeStatusMXBean does get pulled less frequently. But intuitively the directory size is a metric instead of NN status.

        My alternative suggestion is to limit the frequency of checking the directory size to e.g. every 10 mins. If a request for the metric is made within the 10 mins window, the result from the last measurement is returned.

        Show
        zhz Zhe Zhang added a comment - Thanks Haohui for the suggestion. Surendra Singh Lilhore I agree NameNodeStatusMXBean does get pulled less frequently. But intuitively the directory size is a metric instead of NN status. My alternative suggestion is to limit the frequency of checking the directory size to e.g. every 10 mins. If a request for the metric is made within the 10 mins window, the result from the last measurement is returned.
        Hide
        jingzhao Jing Zhao added a comment -

        How about updating the number only when a checkpoint is done or edits are rolled?

        Show
        jingzhao Jing Zhao added a comment - How about updating the number only when a checkpoint is done or edits are rolled?
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        How about updating the number only when a checkpoint is done or edits are rolled?

        I think its good idea. We can update metric when edits are rolled.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - How about updating the number only when a checkpoint is done or edits are rolled? I think its good idea. We can update metric when edits are rolled.
        Hide
        zhz Zhe Zhang added a comment -

        Good idea! Thanks Jing and Surendra.

        Show
        zhz Zhe Zhang added a comment - Good idea! Thanks Jing and Surendra.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Zhe Zhang and Jing Zhao

        Attached updated patch..

        Changes in 004 :

        1. After NN start metric count will update in NNStorage constructor.
        2. ANN will update count after edit log roll.
        3. SNN will update metric count after tailing edit from ANN.
        4. Added new test based on new changes.

        Please review...

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Zhe Zhang and Jing Zhao Attached updated patch.. Changes in 004 : 1. After NN start metric count will update in NNStorage constructor. 2. ANN will update count after edit log roll. 3. SNN will update metric count after tailing edit from ANN. 4. Added new test based on new changes. Please review...
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 24m 50s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        -1 javac 8m 34s The applied patch generated 1 additional warning messages.
        +1 javadoc 11m 5s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 site 3m 36s Site still builds.
        -1 checkstyle 2m 44s The applied patch generated 2 new checkstyle issues (total was 490, now 491).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 4m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 common tests 12m 54s Tests passed in hadoop-common.
        -1 hdfs tests 62m 53s Tests failed in hadoop-hdfs.
            133m 57s  



        Reason Tests
        Failed unit tests hadoop.hdfs.util.TestByteArrayManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768621/HDFS-9229.004.patch
        Optional Tests site javadoc javac unit findbugs checkstyle
        git revision trunk / ab8eb87
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/diffJavacWarnings.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/testrun_hadoop-common.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13186/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13186/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 24m 50s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 8m 34s The applied patch generated 1 additional warning messages. +1 javadoc 11m 5s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 site 3m 36s Site still builds. -1 checkstyle 2m 44s The applied patch generated 2 new checkstyle issues (total was 490, now 491). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 12m 54s Tests passed in hadoop-common. -1 hdfs tests 62m 53s Tests failed in hadoop-hdfs.     133m 57s   Reason Tests Failed unit tests hadoop.hdfs.util.TestByteArrayManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768621/HDFS-9229.004.patch Optional Tests site javadoc javac unit findbugs checkstyle git revision trunk / ab8eb87 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13186/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13186/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13186/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Thank you Haohui Mai, Jing Zhao, Zhe Zhang for the useful discussions.

        Thank you Surendra Singh Lilhore for taking care this. I've few comments:

        1. Do we need to update size inside lock, how about moving it outside?
          +            //Update NameDirSize Metric
          +            namesystem.getFSImage().getStorage().updateNameDirSize();
          
        2. I could see name dir will be updated during FSImage#saveNamespace call FSImage.java#L1061, probably we could cover this logic also, right?
        3. In tests, can we try avoids constant sleeping. One alternate approach I can think is to call #rollEditLog() function explicitly. For example, HATestUtil#waitForStandbyToCatchUp()
          +      Thread.sleep(3*1000);
          +      checkNNDirSize(cluster.getNameDirs(0), nn0.getNameDirSize());
          +      checkNNDirSize(cluster.getNameDirs(1), nn1.getNameDirSize());
          
        4. There is one minor checkstyle warning, please fix it.
          NNStorage.java:1090:23: 'nnDirSizeMap' hides a field.
          
        Show
        rakeshr Rakesh R added a comment - Thank you Haohui Mai , Jing Zhao , Zhe Zhang for the useful discussions. Thank you Surendra Singh Lilhore for taking care this. I've few comments: Do we need to update size inside lock, how about moving it outside? + //Update NameDirSize Metric + namesystem.getFSImage().getStorage().updateNameDirSize(); I could see name dir will be updated during FSImage#saveNamespace call FSImage.java#L1061 , probably we could cover this logic also, right? In tests, can we try avoids constant sleeping. One alternate approach I can think is to call #rollEditLog() function explicitly. For example, HATestUtil#waitForStandbyToCatchUp() + Thread .sleep(3*1000); + checkNNDirSize(cluster.getNameDirs(0), nn0.getNameDirSize()); + checkNNDirSize(cluster.getNameDirs(1), nn1.getNameDirSize()); There is one minor checkstyle warning, please fix it. NNStorage.java:1090:23: 'nnDirSizeMap' hides a field.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Rakesh R for review and suggestions.

        Attached updated patch. Addressed above review comments and accordingly changed test code..

        Please review ...

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Rakesh R for review and suggestions. Attached updated patch. Addressed above review comments and accordingly changed test code.. Please review ...
        Hide
        zhz Zhe Zhang added a comment -

        Great work Surendra! The patch looks good overall. A few comments:

        1. We should add Javadoc for nameDirSizeMap
        2. The behavior of shared directories is worth more discussions. The current patch returns 0 if the directory is shared. Since the purpose of this new metric is for local storage planning / provisioning, shall we report size of shared dirs as well?

        Nits:

        1. Instead of directly using isShared variable, we should call isShared()
        2. The temporary nnDirSizeMap is not necessary. I think we can directly clear and add to nameDirSizeMap.
        Show
        zhz Zhe Zhang added a comment - Great work Surendra! The patch looks good overall. A few comments: We should add Javadoc for nameDirSizeMap The behavior of shared directories is worth more discussions. The current patch returns 0 if the directory is shared. Since the purpose of this new metric is for local storage planning / provisioning, shall we report size of shared dirs as well? Nits: Instead of directly using isShared variable, we should call isShared() The temporary nnDirSizeMap is not necessary. I think we can directly clear and add to nameDirSizeMap .
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Zhe Zhang for review..

        The behavior of shared directories is worth more discussions. The current patch returns 0 if the directory is shared. Since the purpose of this new metric is for local storage planning / provisioning, shall we report size of shared dirs as well?

        Do we have any interface to get size of shared dirs in namenode ?.

        The temporary nnDirSizeMap is not necessary. I think we can directly clear and add to nameDirSizeMap.

        This I did purposefully. Getting size for all the dir and updating in map will take some time because it is I/O call. Suppose we updated size for one dir and in-between some one read that map then he will get incomplete info.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Zhe Zhang for review.. The behavior of shared directories is worth more discussions. The current patch returns 0 if the directory is shared. Since the purpose of this new metric is for local storage planning / provisioning, shall we report size of shared dirs as well? Do we have any interface to get size of shared dirs in namenode ?. The temporary nnDirSizeMap is not necessary. I think we can directly clear and add to nameDirSizeMap. This I did purposefully. Getting size for all the dir and updating in map will take some time because it is I/O call. Suppose we updated size for one dir and in-between some one read that map then he will get incomplete info.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Surendra! I agree with you on both points above. I understood shared dir wrong, thought it meant shared among different block pools.

        So +1 pending a Javadoc on nameDirSizeMap and changing isShared to isShared() (assuming you agree).

        To be extra safe I will file a follow-on JIRA to bound the time of sizeOfDirectory call (if timeout, just don't update the value).

        Show
        zhz Zhe Zhang added a comment - Thanks Surendra! I agree with you on both points above. I understood shared dir wrong, thought it meant shared among different block pools. So +1 pending a Javadoc on nameDirSizeMap and changing isShared to isShared() (assuming you agree). To be extra safe I will file a follow-on JIRA to bound the time of sizeOfDirectory call (if timeout, just don't update the value).
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Zhe Zhang! Fixed remaining two comments.
        Attached updated patch. Please review..

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Zhe Zhang ! Fixed remaining two comments. Attached updated patch. Please review..
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 5s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 4s trunk passed
        +1 compile 4m 19s trunk passed with JDK v1.8.0_60
        +1 compile 4m 8s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 56s trunk passed
        +1 mvnsite 1m 35s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
        +1 javadoc 2m 0s trunk passed with JDK v1.8.0_60
        +1 javadoc 2m 47s trunk passed with JDK v1.7.0_79
        +1 mvninstall 2m 11s the patch passed
        +1 compile 4m 19s the patch passed with JDK v1.8.0_60
        +1 javac 4m 19s the patch passed
        +1 compile 4m 8s the patch passed with JDK v1.7.0_79
        +1 javac 4m 8s the patch passed
        -1 checkstyle 0m 57s Patch generated 1 new checkstyle issues in root (total was 491, now 491).
        +1 mvnsite 1m 33s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 45s the patch passed
        +1 javadoc 1m 56s the patch passed with JDK v1.8.0_60
        +1 javadoc 2m 50s the patch passed with JDK v1.7.0_79
        -1 unit 6m 13s hadoop-common in the patch failed with JDK v1.8.0_60.
        -1 unit 61m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        +1 unit 7m 53s hadoop-common in the patch passed with JDK v1.7.0_79.
        -1 unit 53m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        -1 asflicense 0m 23s Patch generated 58 ASF License warnings.
        175m 39s



        Reason Tests
        JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestIPC
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-29
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769473/HDFS-9229.006.patch
        JIRA Issue HDFS-9229
        Optional Tests asflicense site mvnsite javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux 94b3d6c64bab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh
        git revision trunk / b0c818b
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13270/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Max memory used 226MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13270/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 4s trunk passed +1 compile 4m 19s trunk passed with JDK v1.8.0_60 +1 compile 4m 8s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 56s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 2m 0s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 47s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 11s the patch passed +1 compile 4m 19s the patch passed with JDK v1.8.0_60 +1 javac 4m 19s the patch passed +1 compile 4m 8s the patch passed with JDK v1.7.0_79 +1 javac 4m 8s the patch passed -1 checkstyle 0m 57s Patch generated 1 new checkstyle issues in root (total was 491, now 491). +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 45s the patch passed +1 javadoc 1m 56s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 50s the patch passed with JDK v1.7.0_79 -1 unit 6m 13s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 61m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 7m 53s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 53m 31s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 23s Patch generated 58 ASF License warnings. 175m 39s Reason Tests JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestIPC   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-29 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769473/HDFS-9229.006.patch JIRA Issue HDFS-9229 Optional Tests asflicense site mvnsite javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 94b3d6c64bab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh git revision trunk / b0c818b Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13270/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13270/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13270/console This message was automatically generated.
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Findbug and checkstyle warnings unrelated to this patch.
        Failed tests are passing locally...

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Findbug and checkstyle warnings unrelated to this patch. Failed tests are passing locally...
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Surendra! +1 on the latest patch. I just committed it to trunk and branch-2. Also thanks Rakesh, Haohui, and Jing for the helpful reviews.

        Show
        zhz Zhe Zhang added a comment - Thanks Surendra! +1 on the latest patch. I just committed it to trunk and branch-2. Also thanks Rakesh, Haohui, and Jing for the helpful reviews.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8725 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8725/)
        HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8725 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8725/ ) HDFS-9229 . Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #602 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/602/)
        HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #602 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/602/ ) HDFS-9229 . Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2544 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2544/)
        HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2544 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2544/ ) HDFS-9229 . Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1337 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1337/)
        HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1337 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1337/ ) HDFS-9229 . Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #614 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/614/)
        HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #614 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/614/ ) HDFS-9229 . Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #552 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/552/)
        HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78)

        • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #552 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/552/ ) HDFS-9229 . Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2490 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2490/)
        HDFS-9229. Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2490 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2490/ ) HDFS-9229 . Expose size of NameNode directory as a metric. Contributed by (zhz: rev 8def51a708e5de8a57689b8c9b3fd034cfd2cd78) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
        Hide
        surendrasingh Surendra Singh Lilhore added a comment -

        Thanks Zhe Zhang for reviews and commit.

        Show
        surendrasingh Surendra Singh Lilhore added a comment - Thanks Zhe Zhang for reviews and commit.

          People

          • Assignee:
            surendrasingh Surendra Singh Lilhore
            Reporter:
            zhz Zhe Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development