Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4185

Add a metric for number of active leases

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.4, 2.0.2-alpha
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:

      Description

      We have seen cases of systematic open file leaks, which could have been detected if we have a metric that shows number of active leases.

      1. HDFS-4185-001.patch
        4 kB
        Rakesh R
      2. HDFS-4185-002.patch
        4 kB
        Rakesh R
      3. HDFS-4185-003.patch
        4 kB
        Rakesh R
      4. HDFS-4185-004.patch
        4 kB
        Rakesh R
      5. HDFS-4185-005.patch
        7 kB
        Rakesh R
      6. HDFS-4185-006.patch
        8 kB
        Rakesh R
      7. HDFS-4185-007.patch
        8 kB
        Rakesh R
      8. HDFS-4185-008.patch
        8 kB
        Rakesh R
      9. HDFS-4185-009.patch
        7 kB
        Rakesh R

        Activity

        Hide
        rakeshr Rakesh R added a comment -

        I've tried to expose FSNamesystemMBean#getNumActiveLeases, please review the attached patch. Thanks!

        Show
        rakeshr Rakesh R added a comment - I've tried to expose FSNamesystemMBean#getNumActiveLeases , please review the attached patch. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 3s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 44s There were no new javac warning messages.
        +1 javadoc 9m 50s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 14s The applied patch generated 2 new checkstyle issues (total was 319, now 320).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 19s Pre-build of native portion
        +1 hdfs tests 165m 31s Tests passed in hadoop-hdfs.
            209m 34s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729969/HDFS-4185-001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a319771
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10743/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10743/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10743/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10743/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10743/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 3s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 50s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 2 new checkstyle issues (total was 319, now 320). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 13s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 19s Pre-build of native portion +1 hdfs tests 165m 31s Tests passed in hadoop-hdfs.     209m 34s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729969/HDFS-4185-001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a319771 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10743/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10743/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10743/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10743/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10743/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Attaching another patch fixing whitespace issue.

        Show
        rakeshr Rakesh R added a comment - Attaching another patch fixing whitespace issue.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 34s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 11s The applied patch generated 2 new checkstyle issues (total was 315, now 316).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 1s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        +1 hdfs tests 165m 45s Tests passed in hadoop-hdfs.
            208m 33s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730038/HDFS-4185-002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a319771
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10764/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10764/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10764/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10764/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 11s The applied patch generated 2 new checkstyle issues (total was 315, now 316). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 1s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 165m 45s Tests passed in hadoop-hdfs.     208m 33s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730038/HDFS-4185-002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a319771 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10764/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10764/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10764/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10764/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Attaching another patch addressing the checkstyle comment. Kihwal Lee Please have a look at the patch, thanks!

        FYI: There is one more checkstyle comment reported but this is not related to my patch.

        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 8,290 lines (max allowed is 2,000).
        
        Show
        rakeshr Rakesh R added a comment - Attaching another patch addressing the checkstyle comment. Kihwal Lee Please have a look at the patch, thanks! FYI: There is one more checkstyle comment reported but this is not related to my patch. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 8,290 lines (max allowed is 2,000).
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 32s There were no new javac warning messages.
        +1 javadoc 9m 41s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 14s The applied patch generated 1 new checkstyle issues (total was 315, now 315).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        -1 hdfs tests 165m 2s Tests failed in hadoop-hdfs.
            208m 3s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730125/HDFS-4185-003.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 3ba1836
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10775/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10775/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10775/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10775/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 1 new checkstyle issues (total was 315, now 315). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 165m 2s Tests failed in hadoop-hdfs.     208m 3s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730125/HDFS-4185-003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3ba1836 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10775/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10775/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10775/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10775/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Test failure and checkstyle issue are not related to the patch.

        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 8,290 lines (max allowed is 2,000).
        
        Show
        rakeshr Rakesh R added a comment - Test failure and checkstyle issue are not related to the patch. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 8,290 lines (max allowed is 2,000).
        Hide
        vinayrpet Vinayakumar B added a comment -

        Thanks Rakesh R for taking up this issue.

        Source changes looks good.
        I think current test will just check the API, not the metric via metric system.
        It would be better to have a test to verify the metric via metric system. Check TestNameNodeMetrics FYR.

        Show
        vinayrpet Vinayakumar B added a comment - Thanks Rakesh R for taking up this issue. Source changes looks good. I think current test will just check the API, not the metric via metric system. It would be better to have a test to verify the metric via metric system. Check TestNameNodeMetrics FYR.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Vinayakumar B for the review comments. It helped to identify wrong API leaseManager#countLease to get the count of active leases. Now in the latest patch I used leaseManager#countPath(). Also, I've moved the test case to the TestNameNodeMetrics. Could you please take another look at the patch.

        Show
        rakeshr Rakesh R added a comment - Thanks Vinayakumar B for the review comments. It helped to identify wrong API leaseManager#countLease to get the count of active leases. Now in the latest patch I used leaseManager#countPath() . Also, I've moved the test case to the TestNameNodeMetrics . Could you please take another look at the patch.
        Hide
        kihwal Kihwal Lee added a comment -

        Don't we want both?

        • countLease() gives the number of clients
        • countPath() gives the number of files under construction
        Show
        kihwal Kihwal Lee added a comment - Don't we want both? countLease() gives the number of clients countPath() gives the number of files under construction
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Kihwal Lee for pointing out this. Do you agree to publish the metrics names like:

        NumActiveLeases --> will get the value using leaseMgr#countPath()

        NumActiveClients --> will get the value using leaseMgr#countLease()

        Show
        rakeshr Rakesh R added a comment - Thanks Kihwal Lee for pointing out this. Do you agree to publish the metrics names like: NumActiveLeases --> will get the value using leaseMgr#countPath() NumActiveClients --> will get the value using leaseMgr#countLease()
        Hide
        rakeshr Rakesh R added a comment -

        Attached patch by exposing the metrics -> NumActiveLeases and NumActiveClients. Could you please take a look at the patch. Thanks!

        Show
        rakeshr Rakesh R added a comment - Attached patch by exposing the metrics -> NumActiveLeases and NumActiveClients . Could you please take a look at the patch. Thanks!
        Hide
        raviprak Ravi Prakash added a comment -

        Hi Rakesh! Thanks for your work!
        There's an issue though. The way leases are granted is as follows

        1. A client requests for a lease
        2. This lease could be for several files. (Yes! One lease to write multiple files)
        3. There could be multiple clients (Obviously they have to be writing different files)

        So your getNumActiveLeases() is actually returning number of files open for write (which is >= number of active leases)

        Show
        raviprak Ravi Prakash added a comment - Hi Rakesh! Thanks for your work! There's an issue though. The way leases are granted is as follows A client requests for a lease This lease could be for several files. (Yes! One lease to write multiple files) There could be multiple clients (Obviously they have to be writing different files) So your getNumActiveLeases() is actually returning number of files open for write (which is >= number of active leases)
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 52s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 44s There were no new javac warning messages.
        +1 javadoc 9m 57s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 16s The applied patch generated 11 new checkstyle issues (total was 1243, now 397).
        -1 whitespace 0m 4s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 20s Pre-build of native portion
        -1 hdfs tests 165m 22s Tests failed in hadoop-hdfs.
            209m 16s  



        Reason Tests
        Failed unit tests hadoop.tracing.TestTraceAdmin
          hadoop.hdfs.server.namenode.TestAddBlock
          hadoop.hdfs.TestClientBlockVerification
          hadoop.hdfs.tools.TestHdfsConfigFields



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731554/HDFS-4185-005.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 2edcf93
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10883/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10883/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10883/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10883/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10883/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 11 new checkstyle issues (total was 1243, now 397). -1 whitespace 0m 4s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 20s Pre-build of native portion -1 hdfs tests 165m 22s Tests failed in hadoop-hdfs.     209m 16s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.TestClientBlockVerification   hadoop.hdfs.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731554/HDFS-4185-005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 2edcf93 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10883/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10883/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10883/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10883/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10883/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        1. Whitespace warnings are unrelated to this patch.

        ./hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md:140:The DataNode stores HDFS data in files in its local file system. The DataNode has no knowledge about HDFS files. It stores each block of HDFS data in a separate file in its local file system. The DataNode does not create all files in the same directory. Instead, it uses a heuristic to determine the optimal number of files per directory and creates subdirectories appropriately.  It is not optimal to create all local files in the same directory because the local file system might not be able to efficiently support a huge number of files in a single directory. When a DataNode starts up, it scans through its local file system, generates a list of all HDFS data blocks that correspond to each of these local files, and sends this report to the NameNode. The report is called the _Blockreport_. 
        

        2. Checkstyle warnings also unrelated

        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:225:32: 'out' hides a field.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:242: First sentence should end with a period.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:250:28: 'out' hides a field.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:89: First sentence should end with a period.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:523:18: ',' is not followed by whitespace.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:524:30: ',' is not followed by whitespace.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:250:28: 'out' hides a field.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:89: First sentence should end with a period.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:523:18: ',' is not followed by whitespace.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:524:30: ',' is not followed by whitespace.
        ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 8,302 lines (max allowed is 2,000).
        

        3. testcase failure also unrelated.

        Show
        rakeshr Rakesh R added a comment - 1. Whitespace warnings are unrelated to this patch. ./hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md:140:The DataNode stores HDFS data in files in its local file system. The DataNode has no knowledge about HDFS files. It stores each block of HDFS data in a separate file in its local file system. The DataNode does not create all files in the same directory. Instead, it uses a heuristic to determine the optimal number of files per directory and creates subdirectories appropriately. It is not optimal to create all local files in the same directory because the local file system might not be able to efficiently support a huge number of files in a single directory. When a DataNode starts up, it scans through its local file system, generates a list of all HDFS data blocks that correspond to each of these local files, and sends this report to the NameNode. The report is called the _Blockreport_. 2. Checkstyle warnings also unrelated ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:225:32: 'out' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:242: First sentence should end with a period. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:250:28: 'out' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:89: First sentence should end with a period. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:523:18: ',' is not followed by whitespace. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:524:30: ',' is not followed by whitespace. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:250:28: 'out' hides a field. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:89: First sentence should end with a period. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:523:18: ',' is not followed by whitespace. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:524:30: ',' is not followed by whitespace. ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1: File length is 8,302 lines (max allowed is 2,000). 3. testcase failure also unrelated.
        Hide
        rakeshr Rakesh R added a comment -

        Thank you Ravi Prakash for your time and review comments. Could you please give some more insight on the following statement:

        So your getNumActiveLeases() is actually returning number of files open for write (which is >= number of active leases)

        LeaseManager has the data structure private final Collection<String> paths = new TreeSet<String>();. IIUC during the write/append operation the client who owns the file will get a unique lease from NN and maintains by adding a path entry in this list. Could you please tell me in which case files open for write (which is >= number of active leases), am I missing any case?

        Here, the metric NumActiveLeases count shows the cumulative of all the unique file paths which will be the number of leases granted by the NN at that point of time.

        Show
        rakeshr Rakesh R added a comment - Thank you Ravi Prakash for your time and review comments. Could you please give some more insight on the following statement: So your getNumActiveLeases() is actually returning number of files open for write (which is >= number of active leases) LeaseManager has the data structure private final Collection<String> paths = new TreeSet<String>(); . IIUC during the write/append operation the client who owns the file will get a unique lease from NN and maintains by adding a path entry in this list. Could you please tell me in which case files open for write (which is >= number of active leases) , am I missing any case? Here, the metric NumActiveLeases count shows the cumulative of all the unique file paths which will be the number of leases granted by the NN at that point of time.
        Hide
        raviprak Ravi Prakash added a comment -

        Hi Rakesh!

        Where do you see this data structure private final Collection<String> paths = new TreeSet<String>(); ? I'm not able to find it anywhere in the trunk source code Are you sure you are on the right branch? A commit has gone in lately, but I can't find this particular string anywhere

        Please check this member of Lease . In any case we'll have to bring back some methods after the commit that has gone in

        Show
        raviprak Ravi Prakash added a comment - Hi Rakesh! Where do you see this data structure private final Collection<String> paths = new TreeSet<String>(); ? I'm not able to find it anywhere in the trunk source code Are you sure you are on the right branch? A commit has gone in lately, but I can't find this particular string anywhere Please check this member of Lease . In any case we'll have to bring back some methods after the commit that has gone in
        Hide
        rakeshr Rakesh R added a comment -

        Oh, HDFS-6757 has completely re-factored the LeaseManager and removed the datastructure Collection<String> paths. Now I need to change the logic as follows:

        1. IMO following functions can be used to count the NumActiveClients
          LeaseManager.java
          
            /** @return the number of leases currently in the system */
            @VisibleForTesting
            public synchronized int countLease() {return sortedLeases.size();}
          
        2. Secondly, we need to introduce new API to count the NumActiveLeases
            /** @return the number of paths contained in all leases */
            synchronized int countPath() {
              int count = 0;
              for (Lease lease : sortedLeases) {
                count += lease.getFiles().size();
              }
              return count;
            }
          

        Ravi Prakash Does this sounds good to you ?

        Show
        rakeshr Rakesh R added a comment - Oh, HDFS-6757 has completely re-factored the LeaseManager and removed the datastructure Collection<String> paths . Now I need to change the logic as follows: IMO following functions can be used to count the NumActiveClients LeaseManager.java /** @ return the number of leases currently in the system */ @VisibleForTesting public synchronized int countLease() { return sortedLeases.size();} Secondly, we need to introduce new API to count the NumActiveLeases /** @ return the number of paths contained in all leases */ synchronized int countPath() { int count = 0; for (Lease lease : sortedLeases) { count += lease.getFiles().size(); } return count; } Ravi Prakash Does this sounds good to you ?
        Hide
        rakeshr Rakesh R added a comment -

        Attached patch where I've included the above mentioned changes. I hope this would give more details about the idea.

        Show
        rakeshr Rakesh R added a comment - Attached patch where I've included the above mentioned changes. I hope this would give more details about the idea.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 9s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 40s There were no new javac warning messages.
        +1 javadoc 9m 47s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 51s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 20s Pre-build of native portion
        +1 hdfs tests 168m 57s Tests passed in hadoop-hdfs.
            211m 21s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733133/HDFS-4185-006.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 1ea9014
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11004/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11004/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11004/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 47s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 20s Pre-build of native portion +1 hdfs tests 168m 57s Tests passed in hadoop-hdfs.     211m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733133/HDFS-4185-006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1ea9014 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11004/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11004/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11004/console This message was automatically generated.
        Hide
        raviprak Ravi Prakash added a comment -

        Rakesh!

        Thanks for your work. Please read what Kihwal and I wrote earlier.

        1. In FSNamesystemMBean.java, please rename getNumActiveLeases to getNumFilesUnderConstruction .
        2.   /**
             * Get the total number of active leases in the system.
             */
            @Override // FSNamesystemMBean
            @Metric({ "NumActiveLeases", "Number of active leases" })
            public long getNumActiveLeases() {
              return leaseManager.countPath();
            }

          Please change this to

            /**
             * Get the number of files under construction.
             */
            @Override // FSNamesystemMBean
            @Metric({ "NumFilesUnderConstruction", "Number of files under construction" })
            public long getNumFilesUnderConstruction() {
              return leaseManager.countPath();
            } 
        3. Please also make the changes necessary in the Test code for these changes above
        Show
        raviprak Ravi Prakash added a comment - Rakesh! Thanks for your work. Please read what Kihwal and I wrote earlier. In FSNamesystemMBean.java, please rename getNumActiveLeases to getNumFilesUnderConstruction . /** * Get the total number of active leases in the system. */ @Override // FSNamesystemMBean @Metric({ "NumActiveLeases" , " Number of active leases" }) public long getNumActiveLeases() { return leaseManager.countPath(); } Please change this to /** * Get the number of files under construction. */ @Override // FSNamesystemMBean @Metric({ "NumFilesUnderConstruction" , " Number of files under construction" }) public long getNumFilesUnderConstruction() { return leaseManager.countPath(); } Please also make the changes necessary in the Test code for these changes above
        Hide
        rakeshr Rakesh R added a comment -

        Thanks again Ravi Prakash for the advice. IIUC, your suggestion to use NumFilesUnderConstruction would makes things fine. Attached patch with the necessary changes.

        Show
        rakeshr Rakesh R added a comment - Thanks again Ravi Prakash for the advice. IIUC, your suggestion to use NumFilesUnderConstruction would makes things fine. Attached patch with the necessary changes.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 31s There were no new javac warning messages.
        +1 javadoc 9m 32s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 16s The applied patch generated 1 new checkstyle issues (total was 332, now 332).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        +1 hdfs tests 162m 51s Tests passed in hadoop-hdfs.
            205m 31s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733371/HDFS-4185-007.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / cab0dad
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11018/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11018/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11018/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11018/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11018/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 1 new checkstyle issues (total was 332, now 332). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 2s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 162m 51s Tests passed in hadoop-hdfs.     205m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733371/HDFS-4185-007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cab0dad checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11018/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11018/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11018/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11018/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11018/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 30s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 13s The applied patch generated 1 new checkstyle issues (total was 332, now 332).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        +1 hdfs tests 163m 32s Tests passed in hadoop-hdfs.
            206m 12s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733389/HDFS-4185-008.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / cab0dad
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11019/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11019/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11019/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11019/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 30s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 13s The applied patch generated 1 new checkstyle issues (total was 332, now 332). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 163m 32s Tests passed in hadoop-hdfs.     206m 12s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733389/HDFS-4185-008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cab0dad checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11019/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11019/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11019/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11019/console This message was automatically generated.
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks for all your work Rakesh! I noticed that the metrics are showing up in 2 places on the JMX page (FSNamesystemState and FSNamesystem). Do we need that? If not, could you please remove the changes in FSNamesystemMBean.java, remove the @Override annotations from the two methods in FSNameSystem.java and submit a new patch? I'm a +1 after that.

        Show
        raviprak Ravi Prakash added a comment - Thanks for all your work Rakesh! I noticed that the metrics are showing up in 2 places on the JMX page (FSNamesystemState and FSNamesystem). Do we need that? If not, could you please remove the changes in FSNamesystemMBean.java, remove the @Override annotations from the two methods in FSNameSystem.java and submit a new patch? I'm a +1 after that.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Ravi Prakash for the comments. Attached another patch by removing the methods from FSNamesystemMBean.java

        Show
        rakeshr Rakesh R added a comment - Thanks Ravi Prakash for the comments. Attached another patch by removing the methods from FSNamesystemMBean.java
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks a lot Rakesh! LGTM. +1. Will commit shortly.

        Show
        raviprak Ravi Prakash added a comment - Thanks a lot Rakesh! LGTM. +1. Will commit shortly.
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks a lot Rakesh and Kihwal! I've committed this to trunk and branch-2

        Show
        raviprak Ravi Prakash added a comment - Thanks a lot Rakesh and Kihwal! I've committed this to trunk and branch-2
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7855 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7855/)
        HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7855 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7855/ ) HDFS-4185 . Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 43s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 35s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 12s The applied patch generated 1 new checkstyle issues (total was 291, now 291).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 13s Pre-build of native portion
        -1 hdfs tests 164m 7s Tests failed in hadoop-hdfs.
            207m 4s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733582/HDFS-4185-009.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 060c84e
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11033/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11033/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11033/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11033/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 12s The applied patch generated 1 new checkstyle issues (total was 291, now 291). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 13s Pre-build of native portion -1 hdfs tests 164m 7s Tests failed in hadoop-hdfs.     207m 4s   Reason Tests Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733582/HDFS-4185-009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 060c84e checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11033/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11033/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11033/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11033/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Ravi Prakash, Kihwal Lee, Vinayakumar B for the help.

        Show
        rakeshr Rakesh R added a comment - Thanks Ravi Prakash , Kihwal Lee , Vinayakumar B for the help.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #932 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/932/)
        HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #932 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/932/ ) HDFS-4185 . Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/201/)
        HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/201/ ) HDFS-4185 . Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2130/)
        HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2130 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2130/ ) HDFS-4185 . Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/190/)
        HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/190/ ) HDFS-4185 . Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/200/)
        HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/200/ ) HDFS-4185 . Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2148/)
        HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2148 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2148/ ) HDFS-4185 . Add a metric for number of active leases (Rakesh R via raviprak) (raviprak: rev cdfae446ad285db979a79bf55665363fd943702c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            rakeshr Rakesh R
            Reporter:
            kihwal Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development